Skip to content

feat(metrics): Prometheus histogram for per-event processing latency across all stages #146

Description

@Depo-dev

Summary

Issue #57 adds the /metrics endpoint to the Rust indexer. This issue defines the specific metrics that endpoint must expose — particularly the per-event latency histogram that breaks processing time down by stage: XDR parse, database insert, and Redis publish. Without this histogram, operators cannot tell whether a slowdown in indexing is caused by CPU-bound XDR decoding, a slow Postgres insert, or a Redis backpressure issue.

Context

The indexer's hot loop processes one event at a time: parse the XDR, insert to Postgres, publish to Redis. The total latency of this loop determines how quickly events become queryable via the API. If this loop slows down — say due to a Postgres vacuum running — the indexer starts falling behind chain tip. The histogram makes this diagnosable without having to grep through logs.

What needs to be built

Using the prometheus crate:

Histogram: trident_event_processing_duration_seconds

  • Labels: stage — one of parse, db_insert, redis_publish
  • Buckets: 0.001, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500, 1.0, +Inf
  • Measured with Instant::now() before and after each stage in the streamer loop

Counter: trident_events_processed_total

  • Labels: statussuccess, parse_error, db_error, redis_error
  • Incremented on every event attempt

Gauge: trident_indexer_lag_ledgers

  • Current chain tip ledger minus the indexed cursor ledger
  • Updated after every poll cycle

All three metrics are exposed on the /metrics endpoint added in issue #57. They should be registered in a shared Prometheus registry passed through from main to the streamer.

A sample Grafana dashboard JSON should be committed to docs/grafana/indexer_dashboard.json so operators have a ready-made starting point.

Acceptance Criteria

  • Histogram observes parse, db_insert, and redis_publish latency separately for every event
  • Counter increments with the correct status label on every event
  • Lag gauge updated after every poll cycle
  • Metrics are visible on /metrics endpoint
  • Grafana dashboard JSON committed to docs/grafana/
  • Unit test: process a synthetic event, assert all three metrics have non-zero values

Dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stellar WaveIssues in the Stellar wave program

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions