Skip to content

Add monitoring to hydra demo #2161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 20 commits into
base: master
Choose a base branch
from
Open

Add monitoring to hydra demo #2161

wants to merge 20 commits into from
+1,393 −2

Conversation

ffakenz
Copy link
Contributor

@ffakenz ffakenz commented Jul 31, 2025 •
edited
Loading

Adding a minimal Prometheus + Grafana monitoring setup for Hydra nodes, including panels for insight into the node's activity during demo.

Acknowledgment: This dashboard is largely based on the work originally done by @jmagan. Thanks for laying the groundwork! 🙏

  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch from 09edacc to 579e455 Compare July 31, 2025 02:44
@ffakenz ffakenz self-assigned this Jul 31, 2025
@ffakenz ffakenz moved this from Triage 🏥 to In review 👀 in ☕ Hydra Team Work Jul 31, 2025
@ffakenz ffakenz requested a review from a team July 31, 2025 02:47
@ffakenz ffakenz changed the title add monitoring to hydra demo Add monitoring to hydra demo Jul 31, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 31, 2025 •
edited
Loading

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2025-08-25 15:41:17.570100912 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial c8a101a5c8ac4816b0dceb59ce31fc2258e387de828f02961d2f2045 2652
νCommit 61458bc2f297fff3cc5df6ac7ab57cefd87763b0b7bd722146a1035c 685
νHead be6ebc744208c660bf0fdc1cfbb5157477cd305de5b1777e575cbb4c 14665
μHead 1f47a42d1d6edc32ccd834acb19d5db3b2a5232f0bd7eaa8908dc519* 5284
νDeposit ae01dade3a9c346d5c93ae3ce339412b90a0b8f83f94ec6baa24e30c 1102
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 5837 10.57 3.36 0.52
2 6038 12.23 3.86 0.54
3 6238 14.31 4.52 0.57
5 6641 18.98 6.00 0.64
10 7646 29.14 9.19 0.79
43 14279 98.78 30.87 1.80

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 561 2.44 1.16 0.20
2 739 3.38 1.73 0.22
3 920 4.36 2.33 0.24
5 1282 6.41 3.60 0.28
10 2172 12.13 7.25 0.40
54 10057 98.61 68.52 1.88

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 525 24.46 7.13 0.42
2 113 640 32.24 9.37 0.51
3 170 747 41.51 12.02 0.61
4 225 858 48.11 13.97 0.68
5 282 969 62.69 17.85 0.83
6 339 1081 75.66 21.39 0.96
7 394 1192 72.34 20.99 0.94
8 449 1307 98.23 27.59 1.20
9 505 1414 90.51 26.08 1.13

Cost of Increment Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 1797 23.87 7.96 0.48
2 1936 25.24 9.07 0.50
3 2060 27.14 10.30 0.53
5 2436 32.23 13.37 0.61
10 3109 40.34 19.35 0.75
40 7714 97.37 58.19 1.70

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 634 22.57 7.33 0.41
2 787 24.33 8.46 0.44
3 939 27.07 9.88 0.48
5 1282 30.12 12.06 0.54
10 2088 41.22 18.51 0.71
42 6421 94.08 54.54 1.59

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 635 26.74 8.51 0.46
2 766 28.43 9.71 0.48
3 872 31.91 11.38 0.53
5 1274 37.48 14.72 0.62
10 2117 45.13 20.75 0.76
39 6153 99.66 57.43 1.64

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 628 33.06 10.20 0.52
2 827 35.73 11.79 0.56
3 974 37.62 13.04 0.59
5 1273 42.47 15.94 0.66
10 1985 53.10 22.61 0.83
29 5051 98.90 50.17 1.53

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 5815 26.97 9.06 0.69
2 6002 36.74 12.39 0.80
3 6162 46.04 15.52 0.91
4 6167 52.90 17.71 0.98
5 6398 60.21 20.25 1.06
6 6631 75.12 25.39 1.23
7 6709 80.22 27.00 1.29
8 6800 87.99 29.55 1.37

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
10 0 0 5834 18.75 6.26 0.60
10 1 57 5868 22.30 7.58 0.64
10 5 284 6004 29.99 10.65 0.73
10 20 1138 6512 60.01 22.53 1.08
10 39 2219 7158 99.88 38.20 1.55

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2025-08-25 15:44:04.18311937 UTC

Baseline Scenario

Number of nodes 1
Number of txs 300
Avg. Confirmation Time (ms) 9.928240390
P99 12.711228219999255ms
P95 7.435975950000001ms
P50 5.063249000000001ms
Number of Invalid txs 0

Memory data

Time Used Free
2025-08-25 15:42:49.641872912 UTC 1181M 7086M
2025-08-25 15:42:50.641760664 UTC 1186M 7081M
2025-08-25 15:42:51.641808553 UTC 1226M 7040M
2025-08-25 15:42:52.641797642 UTC 1277M 6963M
2025-08-25 15:42:53.641819272 UTC 1274M 6965M
2025-08-25 15:42:54.642260139 UTC 1263M 6976M
2025-08-25 15:42:55.64179956 UTC 1280M 6955M
2025-08-25 15:42:56.641812976 UTC 1294M 6938M
2025-08-25 15:42:57.641990589 UTC 1298M 6934M
2025-08-25 15:42:58.641750842 UTC 1302M 6930M
2025-08-25 15:42:59.641751504 UTC 1305M 6926M
2025-08-25 15:43:00.641789058 UTC 1306M 6926M
2025-08-25 15:43:01.641691175 UTC 1306M 6925M
2025-08-25 15:43:02.641715801 UTC 1308M 6924M
2025-08-25 15:43:03.641751581 UTC 1308M 6923M
2025-08-25 15:43:04.64171379 UTC 1308M 6923M
2025-08-25 15:43:05.641772782 UTC 1308M 6923M
2025-08-25 15:43:06.641756933 UTC 1292M 6939M
2025-08-25 15:43:07.641745362 UTC 1292M 6939M
2025-08-25 15:43:08.641753469 UTC 1292M 6939M
2025-08-25 15:43:09.64176977 UTC 1292M 6939M
2025-08-25 15:43:10.641839372 UTC 1292M 6938M
2025-08-25 15:43:11.641812451 UTC 1292M 6938M
2025-08-25 15:43:12.641799248 UTC 1292M 6938M
2025-08-25 15:43:13.641781968 UTC 1292M 6938M
2025-08-25 15:43:14.641782935 UTC 1296M 6934M
2025-08-25 15:43:15.641787581 UTC 1296M 6934M
2025-08-25 15:43:16.641735148 UTC 1296M 6934M

Three local nodes

Number of nodes 3
Number of txs 900
Avg. Confirmation Time (ms) 32.030130517
P99 47.86910254ms
P95 41.58663239999999ms
P50 31.041207ms
Number of Invalid txs 0

Memory data

Time Used Free
2025-08-25 15:43:28.504427057 UTC 1204M 7063M
2025-08-25 15:43:29.50470649 UTC 1216M 7050M
2025-08-25 15:43:30.504672985 UTC 1356M 6857M
2025-08-25 15:43:31.504673205 UTC 1382M 6804M
2025-08-25 15:43:32.506883956 UTC 1438M 6747M
2025-08-25 15:43:33.505690579 UTC 1450M 6733M
2025-08-25 15:43:34.50698324 UTC 1470M 6702M
2025-08-25 15:43:35.504630559 UTC 1493M 6668M
2025-08-25 15:43:36.506385126 UTC 1502M 6649M
2025-08-25 15:43:37.504626629 UTC 1504M 6636M
2025-08-25 15:43:38.507039039 UTC 1521M 6607M
2025-08-25 15:43:39.504846169 UTC 1524M 6593M
2025-08-25 15:43:40.507862767 UTC 1526M 6580M
2025-08-25 15:43:41.504987021 UTC 1529M 6566M
2025-08-25 15:43:42.504632296 UTC 1535M 6550M
2025-08-25 15:43:43.504548756 UTC 1539M 6541M
2025-08-25 15:43:44.507089751 UTC 1539M 6540M
2025-08-25 15:43:45.504889585 UTC 1540M 6540M
2025-08-25 15:43:46.504517346 UTC 1540M 6539M
2025-08-25 15:43:47.50446951 UTC 1540M 6539M
2025-08-25 15:43:48.504666702 UTC 1545M 6534M
2025-08-25 15:43:49.504647439 UTC 1545M 6534M
2025-08-25 15:43:50.504579763 UTC 1546M 6533M
2025-08-25 15:43:51.504647606 UTC 1546M 6532M
2025-08-25 15:43:52.504637364 UTC 1546M 6532M
2025-08-25 15:43:53.504713989 UTC 1546M 6531M
2025-08-25 15:43:54.504603494 UTC 1547M 6531M
2025-08-25 15:43:55.504854588 UTC 1546M 6531M
2025-08-25 15:43:56.504890015 UTC 1546M 6531M
2025-08-25 15:43:57.504666022 UTC 1546M 6531M
2025-08-25 15:43:58.504616037 UTC 1546M 6531M
2025-08-25 15:43:59.504599603 UTC 1546M 6531M
2025-08-25 15:44:00.504692586 UTC 1545M 6531M
2025-08-25 15:44:01.504684403 UTC 1546M 6531M
2025-08-25 15:44:02.504567431 UTC 1545M 6531M
2025-08-25 15:44:03.504525069 UTC 1545M 6531M

@github-actions GitHub Actions
Copy link

Transaction cost differences

No cost or size differences found

@noonio
Copy link
Contributor

noonio commented Jul 31, 2025 •
edited
Loading

I'm not seeing any data coming through.

I suspect it's because of docker networking differences between linux and mac; do you see data?

I recommend just using nixpkgs to get grafana and prometheus, and using them directly in the process-compose setup.

@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch 2 times, most recently from d1a8754 to 35db741 Compare August 5, 2025 03:09
@noonio noonio moved this from In review 👀 to In progress 🕐 in ☕ Hydra Team Work Aug 5, 2025
@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch 2 times, most recently from 66638b6 to 6e51fdb Compare August 6, 2025 10:27
@ffakenz ffakenz moved this from In progress 🕐 to In review 👀 in ☕ Hydra Team Work Aug 6, 2025
@ffakenz ffakenz requested a review from a team August 6, 2025 10:27
@noonio
Copy link
Contributor

noonio commented Aug 6, 2025

Looking good!

image

There's a bunch of panels that are empty -
image

Are you planning to get those working? If not maybe should just remove them, or leave a note somewhere?

noonio
noonio reviewed Aug 6, 2025
View reviewed changes
@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch 6 times, most recently from d56e8a5 to cee562c Compare August 19, 2025 11:02
@noonio noonio moved this from In review 👀 to In progress 🕐 in ☕ Hydra Team Work Aug 19, 2025
@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch 2 times, most recently from 4e011ac to 41caea1 Compare August 21, 2025 09:20
ffakenz added 15 commits August 25, 2025 14:19
@ffakenz
add grafana dashboard ~ first draft
85a3847
@ffakenz
add monitoring command to nix demo
9a9c190 
* this starts grafana and prometheus from docker-compose
@ffakenz
update prometheus config to scrap node metrics from all demos
5908395 
* from docker-compose demo containers
* from nix process-compose demo commands
@ffakenz
split demo monitoring into nix and docker
a7dfb8e 
* as previous solution was not compatible with all systems
@ffakenz
fix grafana data paths from nix process
53b7447
@ffakenz
fix dashboard provisioning
7fcb6ee 
- by adding default.yaml
> also removed grafana.ini to use defaults
as we are overriding confs using nix cmd to access working demo path
@ffakenz
add default provisioning config files
371d10c
@ffakenz
fmt
f103fc5
@ffakenz
update demo docker-compose file
a9c5e0b
@ffakenz
update CHANGELOG
e0a8785
@ffakenz
configure node-exporter to run alongside prometheus
68cd702 
* to gather system metrics
@ffakenz
revert upgrading container versions in demo compose file
f9ab524 
> otherwise, master will be in a little bit of an odd state (with the .cabal file still being the older version)

These versions can be fixed in the generic catchup of #2170
@ffakenz
enable node-exporter in docker/demo
2ec2f96
@ffakenz
update node exporter metrics on dashboard
bd7ac9a
@ffakenz
fix running nodes panel
6d75d29 
also change docker image versions to match master
@ffakenz ffakenz force-pushed the hydra-monitoring-grafana branch 2 times, most recently from 8e82556 to 6d75d29 Compare August 25, 2025 12:21
@ffakenz
remove system dependant metrics from grafana dashboards
bfe19ed 
> because they are not cross platform compatible
and depend on the underlying host system
@ffakenz ffakenz moved this from In progress 🕐 to In review 👀 in ☕ Hydra Team Work Aug 25, 2025
noonio
noonio reviewed Aug 25, 2025
View reviewed changes
noonio
noonio requested changes Aug 25, 2025
View reviewed changes
Copy link
Contributor

@noonio noonio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great; happy to merge with just some minor feedback:

  1. I think this is based on the dashboards @jmagan made? If so, just a note to acknowledge would be nice :)
  2. Can you add some docs somewhere to show this off? Perhaps under the docker section; with some screenshots to show what people can see and perhaps what they might want to keep an eye on? I.e. number of peers connected; a rise in confirmed Txs, etc.

Otherwise; looking awesome, thanks @ffakenz !

@github-project-automation github-project-automation bot moved this from In review 👀 to In progress 🕐 in ☕ Hydra Team Work Aug 25, 2025
@jmagan
Copy link
Contributor

jmagan commented Aug 25, 2025

Looks great; happy to merge with just some minor feedback:

  1. I think this is based on the dashboards @jmagan made? If so, just a note to acknowledge would be nice :)
  2. Can you add some docs somewhere to show this off? Perhaps under the docker section; with some screenshots to show what people can see and perhaps what they might want to keep an eye on? I.e. number of peers connected; a rise in confirmed Txs, etc.

Otherwise; looking awesome, thanks @ffakenz !

Those are actually some kind of old. Take a look to the newest Dashboards at https://github.com/no-witness-labs/hydra-deployment-poc/tree/main/participant-node-monitoring/grafana we also have a new Dashboard for ETCD ;)

@ffakenz ffakenz requested a review from noonio August 25, 2025 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@v0d1ch v0d1ch v0d1ch approved these changes

@noonio noonio Awaiting requested review from noonio

Requested changes must be addressed to merge this pull request.

Assignees

@ffakenz ffakenz

Labels
None yet
Projects
☕ Hydra Team Work
Status: In progress 🕐
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ffakenz @noonio @jmagan @v0d1ch