Skip to content

feat(load): Add metrics + gun reliability#758

Merged
rodrigombsoares merged 5 commits into
mainfrom
pr1-load-metrics
Jul 8, 2026
Merged

feat(load): Add metrics + gun reliability#758
rodrigombsoares merged 5 commits into
mainfrom
pr1-load-metrics

Conversation

@rodrigombsoares

@rodrigombsoares rodrigombsoares commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

First stacked PR for devenv load-test observability and gun reliability. Extracted from the e2e-load-prod-enablement monolith; still scoped to local devenv only.

Running existing load tests now prints per-phase timing breakdowns, failure counts, and a message ID summary at the end of each run. CCIPLoadGun uses sync.Cond single-flight concurrency so overlapping WASP calls wait instead of failing.

This PR also moves token minting on e2e tests to top level. To avoid holdings conflicts between tests leading to flakyness. Holding picking function is still sub-optimal, we need to harden that in follow up PRs.

Features

  • LoadMetricsCollector (metrics.go) — records Send, Confirm Send, Confirm Exec, and Total durations per successful message; tracks phase failure counts (send / confirm send / confirm exec)
  • Phase metrics outputPrintPhaseMetricsSummary logs Min/P90/P95/P99 per phase; bridges successful records to CCV MessageMetrics summary
  • CCIPLoadGun reliabilitysync.Cond-based single-flight (peak concurrency asserted ≤ 1); collects messageIDs after confirm-send (canton needs to run sequentially due to CIDs changing on every call)
  • New gun APILoadGunOptions with injectable ConfirmSendFunc and ConfirmExecTimeout (prod-testnet workaround wired in a later PR); devenv tests use cantonSourceConfirmSend / evmSourceConfirmSend
  • runWASP enhancements — configurable CallTimeout via CANTON_LOAD_CALL_TIMEOUT; schedule logging; deferred message summary; metrics printed at end of run
  • Canton send loggingimpl.go logs messageID + seqNo after CCIP send parse

Testing

Requires a running local devenv and ccip/devenv/env-canton-evm-out.toml.

  1. make start-devenv
  2. make run-canton2evm-load
  3. make run-evm2canton-load
  4. make run-canton2evm-token-load
  5. make run-evm2canton-token-load

@rodrigombsoares
rodrigombsoares requested a review from a team as a code owner July 6, 2026 14:09
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

👋 rodrigombsoares, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

)

// ConfirmSendFunc confirms a CCIP send on the source chain after SendMessage returns.
// TODO: this is needed because EVM impls method has a bug on prod-testnet checks.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this TODO?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will make more sense in followup PRs, but EVM has a confirmSend method that is basically uncallable on prod-testnet because it doesn't set a starting block on query.
Since this PR is still devenv only it doesn't make too much sense to have this wrapper around, but it will in next PRs

@rodrigombsoares
rodrigombsoares merged commit b4c5f1e into main Jul 8, 2026
39 checks passed
@rodrigombsoares
rodrigombsoares deleted the pr1-load-metrics branch July 8, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants