SoroTrail A contract event indexer for the Stellar/Soroban network.

Stellar RPC's getEvents method only retains contract events for roughly 24 hours to 7 days. Anyone who needs historical Soroban event data — dapp dashboards, analytics, audits, notification services — must ingest and store events themselves before the RPC drops them.

SoroTrail does exactly that: it polls a Stellar RPC endpoint, stores contract events durably in Postgres or SQLite, and serves them back through a queryable HTTP API long after the RPC has forgotten them.

 Stellar RPC ──getEvents──▶ ingester ──▶ Postgres/SQLite ◀── HTTP API ◀── you

Quickstart

text

Published image (fastest)

Tagged releases publish a multi-arch (amd64/arm64) image to GHCR. Point it at a Postgres you already have:

docker run --rm -p 8080:8080 \
  -e DATABASE_URL='postgres://user:pass@host:5432/sorotrail?sslmode=disable' \
  -e RPC_URL='https://soroban-testnet.stellar.org' \
  ghcr.io/sorotrail/sorotrail:latest

Pin a specific release with a version tag instead of latest, e.g. ghcr.io/sorotrail/sorotrail:v1.2.0. See Configuration for the full list of environment variables.

Docker Compose (full stack)

Brings up Postgres and the indexer together — no external database required:

docker compose up --build This starts Postgres and the indexer against the public Stellar testnet RPC. The API is on http://localhost:8080; watch the logs to see events flow in.

To watch specific contracts instead of everything:

Shell

Container health. The published image ships with a HEALTHCHECK that probes /health via the in-binary sorotrail healthcheck subcommand (alpine has no curl/wget — installing curl or shipping a second binary would just grow the image; routing the probe through the existing binary reuses the net/http client that's already linked in for the server, so the cost is a few hundred bytes of flag-parsing and a probe function). Compose mirrors the same probe so docker ps shows an honest health status, and combined with depends_on: condition: service_healthy on Postgres, a fresh docker compose up --build brings the stack up in the right order instead of hoping the indexer wins a race against a half-up database.

WATCHED_CONTRACTS=CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC docker compose up --build Bare metal Shell

docker compose up -d postgres # or bring your own Postgres cp .env.example .env # adjust as needed set -a; source .env; set +a make run

No Postgres? Use SQLite instead for a zero-dependency run:

```sh
DATABASE_URL=sqlite:./sorotrail.db make run

Migrations run automatically on startup.

Configuration All configuration comes from environment variables (see .env.example ):

Variable Default Description RPC_URL https://soroban-testnet.stellar.org Stellar RPC endpoint (JSON-RPC 2.0). Point at a provider URL for mainnet. DATABASE_URL — (required) Postgres connection string. POLL_INTERVAL 5s Sleep between polls once caught up. HTTP_ADDR :8080 API listen address. WATCHED_CONTRACTS empty Comma-separated contract IDs (C...). Empty = ingest all contract events. START_LEDGER unset Force cold-start ingestion from this ledger. RETENTION_LEDGERS 17280 Cold-start reach-back in ledgers (~24h at 5s/ledger). LOG_LEVEL info debug | info | warn | error. AUDIT_ENABLED false Enable the background auditor. When unset/false the binary behaves exactly like the pre-audit build. AUDIT_POLL_INTERVAL 30s Sleep between audit passes. AUDIT_BATCH_LEDGERS 100 Ledger range covered by one audit pass. AUDIT_LAG_THRESHOLD 200 Auditor sleeps until ingest is at least this many ledgers past the verified mark. AUDIT_BUDGET_SHARE 0.10 Fraction of the request budget the audit pool gets (rest goes to ingest). AUDIT_MAX_RPS 10 Total request budget (split between ingest and audit). AUDIT_MAX_REPAIR_ATTEMPTS 3 Repair iterations before a finding is kept open as unrecoverable. AUDIT_FINDING_MAX_LEDGERS 100 Largest range a single finding is allowed to span. RATE_LIMIT_RPS unset Per-client HTTP request rate limit (requests/second). Both RATE_LIMIT_RPS and RATE_LIMIT_BURST must be set together; otherwise no rate limiting is applied. RATE_LIMIT_BURST unset Maximum instantaneous burst size for the rate limiter. Pairs with RATE_LIMIT_RPS. RATE_LIMIT_TRUSTED_PROXY false Honor X-Forwarded-For for client IP detection. Must only be enabled behind a proxy you trust to strip/rewrite the header — clients control X-Forwarded-For themselves, so enabling it on an Internet-facing surface lets any caller pick their own rate-limit key. CACHE_PRIVATE false Flip cacheable responses from Cache-Control: public to private. Set this when the deployment serves per-user data behind an auth layer (see Caching). Ingestion behavior Cold start (empty database): begins at latest ledger − RETENTION_LEDGERS (clamped to what the RPC still retains) so it captures as much recent history as possible, then follows the chain head. START_LEDGER overrides this. Warm start: resumes from the persisted cursor / last ingested ledger. Events are upserted idempotently by ID, so re-scans and restarts never duplicate rows. If the indexer is down long enough that its resume point falls out of the RPC's retention window, it logs a warning and skips ahead to the oldest retained ledger (the gap is unrecoverable from RPC — that's the problem this project exists to prevent). Requests are rate-limited (~10/s, matching public endpoint limits) and errors are retried with jittered exponential backoff. Topics/values are stored as JSON. When the RPC supports xdrFormat: "json" its decoding is used verbatim; otherwise the base64 XDR is decoded locally into shapes like {"symbol":"transfer"}, {"u64":42}, {"i128":"-1000"}, {"address":"C..."}. The raw base64 XDR is stored alongside the decoded JSON, so an improved decoder can be applied to already-indexed events — see decoder replay. This intentionally duplicates payload data in events.topics_xdr and events.value_xdr; budget extra event-table storage for deployments that retain large event histories. Decoder replay Decoders improve over time. sorotrail replay re-runs the current decoder

Configuration

All configuration comes from environment variables (see .env.example):

Variable	Default	Description
RPC_URL	https://soroban-testnet.stellar.org	Stellar RPC endpoint (JSON-RPC 2.0). Point at a provider URL for mainnet.
RPC_URLS	unset	Comma-separated, priority-ordered list of Stellar RPC endpoints. When set, RPC_URL is ignored and the multi-provider failover client is used. List order is priority: index 0 is tried first.
RPC_RATE_LIMIT_RPS	10	Per-provider request rate limit (requests/second) applied to each RPC endpoint independently. Only used when RPC_URLS is set.
DATABASE_URL	— (required)	Postgres connection string.
POLL_INTERVAL	5s	Sleep between polls once caught up.
HTTP_ADDR	:8080	API listen address.
WATCHED_CONTRACTS	empty	Comma-separated contract IDs (C...). Empty = ingest all contract events. Each watched contract tracks its own resume cursor; adding a contract automatically triggers a backfill from latest − RETENTION_LEDGERS (clamped to RPC retention), independent of other contracts.
START_LEDGER	unset	Force cold-start ingestion from this ledger.
RETENTION_LEDGERS	17280	Cold-start reach-back in ledgers (~24h at 5s/ledger).
LOG_LEVEL	info	debug | info | warn | error.
LOG_FORMAT	text	text | json. JSON emits one JSON object per line, compatible with Loki, CloudWatch, and ELK.
API_QUERY_TIMEOUT	25s	Per-request database timeout for API-originated store reads. The timeout is enforced in-process and mirrored to Postgres via statement_timeout.
API_SLOW_QUERY_THRESHOLD	2s	Warn when an API-originated store query takes longer than this threshold; logs include the query name and elapsed duration.
AUDIT_ENABLED	false	Enable the background auditor. When unset/false the binary behaves exactly like the pre-audit build.
AUDIT_POLL_INTERVAL	30s	Sleep between audit passes.
AUDIT_BATCH_LEDGERS	100	Ledger range covered by one audit pass.
AUDIT_LAG_THRESHOLD	200	Auditor sleeps until ingest is at least this many ledgers past the verified mark.
AUDIT_BUDGET_SHARE	0.10	Fraction of the request budget the audit pool gets (rest goes to ingest).
AUDIT_MAX_RPS	10	Total request budget (split between ingest and audit).
AUDIT_MAX_REPAIR_ATTEMPTS	3	Repair iterations before a finding is kept open as unrecoverable.
AUDIT_FINDING_MAX_LEDGERS	100	Largest range a single finding is allowed to span.
API_MAX_LIMIT	500	Maximum page size accepted for list endpoints (/events, /subscriptions/{id}/deliveries). Values above this are rejected with 400.
RATE_LIMIT_RPS	unset	Per-client HTTP request rate limit (requests/second). Both RATE_LIMIT_RPS and RATE_LIMIT_BURST must be set together; otherwise no rate limiting is applied.
RATE_LIMIT_BURST	unset	Maximum instantaneous burst size for the rate limiter. Pairs with RATE_LIMIT_RPS.
RATE_LIMIT_TRUSTED_PROXY	false	Honor X-Forwarded-For for client IP detection. Must only be enabled behind a proxy you trust to strip/rewrite the header — clients control X-Forwarded-For themselves, so enabling it on an Internet-facing surface lets any caller pick their own rate-limit key.
CACHE_PRIVATE	false	Flip cacheable responses from Cache-Control: public to private. Set this when the deployment serves per-user data behind an auth layer (see Caching).
CORS_ALLOWED_ORIGINS	unset	Comma-separated browser origins allowed to call the API. * allows any origin; otherwise each entry is an explicit origin (scheme://host). Unset = CORS disabled, no CORS headers emitted.
RATE_LIMIT_RPS	unset	Per-client HTTP request rate limit (requests/second). Both RATE_LIMIT_RPS and RATE_LIMIT_BURST must be set together; otherwise no rate limiting is applied.
RATE_LIMIT_BURST	unset	Maximum instantaneous burst size for the rate limiter. Pairs with RATE_LIMIT_RPS.
RATE_LIMIT_TRUSTED_PROXY	false	Honor X-Forwarded-For for client IP detection. Must only be enabled behind a proxy you trust to strip/rewrite the header — clients control X-Forwarded-For themselves, so enabling it on an Internet-facing surface lets any caller pick their own rate-limit key.
COMPRESS_MIN_SIZE	1400	Response body size (bytes) at or above which responses are gzip/deflate encoded for clients advertising support. Set negative to disable compression. See Compression.
SHUTDOWN_TIMEOUT	15s	Time the graceful shutdown may wait for in-flight requests and the current ingest cycle to wind down before the process is killed. Zero means wait indefinitely.
SWEEP_CONCURRENCY	1	Number of filter batches fanned out in parallel during a windowSweep pass. The per-request RPC interval limiter still caps total request rate at ~10 req/s, so raising this helps only against private RPCs with more headroom. The single-batch path (<=25 watched contracts) is unchanged.
REORG_CONFIRMATION_WINDOW	64	Number of ledgers behind the ingest frontier re-scanned on a schedule for RPC-side reorgs. Once a ledger is further behind the frontier than this, it is considered finalized and never rewritten. Zero disables reorg detection.
REORG_RESCAN_INTERVAL	1m	Cadence of the periodic reorg re-scan over the recent finalized window. The re-scan shares the live RPC budget and runs after a successful ingest cycle.
EXPORT_MAX_RANGE	17280	Maximum ledger span a single /contracts/{id}/export call may request (~24h at 5s/ledger). Returns 400 with the bound if exceeded. Raise for dedicated analytical deployments; lower for tighter abuse thresholds.
MULTI_TENANT	false	Serve several consumers from one deployment, each scoped to its own contracts. Off means no authentication and no tenant boundary — identical to the pre-multi-tenancy build. See Multi-tenancy.
MULTI_TENANT_MAX_WATCHED	250	Cap on the union of all tenants' watch lists, bounding the ingester's RPC cost. 0 disables the cap.
MULTI_TENANT_USAGE_FLUSH	10s	How often accumulated per-tenant usage counters are persisted.
MULTI_TENANT_STREAM_SCOPE_SYNC	30s	How often an open stream re-resolves its tenant's grants, bounding how long a revoked grant keeps being served.
MULTI_TENANT_BOOTSTRAP_KEY	unset	Installs an admin API key for the seeded default tenant at startup, so a fresh multi-tenant install can mint its first keys. Rejected unless MULTI_TENANT=true.

Multi-provider failover

RPC_URL is a single point of failure. Setting RPC_URLS enables a multi-provider failover client that wraps each endpoint with health scoring and automatic promotion/demotion:

• Priority order: providers are tried in list order; the highest-priority healthy provider receives all traffic.
• Passive health scoring: real request outcomes drive demotion. Network errors (DNS, connection refused, timeout) and HTTP 5xx responses count toward the error budget. Semantic errors (IsLedgerOutOfRange, HTTP 4xx) never demote a healthy provider.
• Demotion: after 3 consecutive demotable errors a provider is degraded (still eligible but deprioritised); after 3 further errors it becomes down and is excluded from traffic.
• Probes: StateDown providers receive periodic getHealth probes (every 30s). After 2 consecutive successful probes the provider is promoted back to active. Active providers are never probed — rate limit is reserved for real work.
• Cursor re-anchor: when a provider switch occurs mid-pagination (a getEvents request carries a cursor), the failover client returns ErrFailoverReanchor. The ingester discards the cursor and resumes from the last persisted ledger position; idempotent upserts absorb the overlap.
• All-down: when every provider is down, the client enters jittered exponential backoff with clear logging.
• Per-provider rate limits: each provider gets its own token bucket at RPC_RATE_LIMIT_RPS (default 10/s).
• Head skew tolerance: small chain-head differences (≤3 ledgers) between providers do not cause flaps.
• Backward compatible: omitting RPC_URLS keeps the single-URL behaviour unchanged. RPC_URL still works and is still the default.

# Single provider (unchanged)
RPC_URL=https://soroban-testnet.stellar.org

# Multi-provider failover
RPC_URLS=https://rpc1.example.com,https://rpc2.example.com,https://rpc3.example.com
RPC_RATE_LIMIT_RPS=15

Mixed-retention caveat

Providers with different retention windows present a known risk: a failover from a long-retention provider to a short-retention one may cause IsLedgerOutOfRange errors for older ledger queries. The indexer handles this by re-clamping to the oldest retained ledger (accepting the gap), but deployments with heterogeneous providers should ensure the shortest retention window is ≥ the ingester's RETENTION_LEDGERS setting.

Ingestion behavior

• Cold start (empty database): begins at latest ledger − RETENTION_LEDGERS (clamped to what the RPC still retains) so it captures as much recent history as possible, then follows the chain head. START_LEDGER overrides this.
• Warm start: resumes from the persisted cursor / last ingested ledger.
• Events are upserted idempotently by ID, so re-scans and restarts never duplicate rows.
• If the indexer is down long enough that its resume point falls out of the RPC's retention window, it logs a warning and skips ahead to the oldest retained ledger (the gap is unrecoverable from RPC — that's the problem this project exists to prevent).
• Requests are rate-limited (~10/s, matching public endpoint limits) and errors are retried with jittered exponential backoff.
• Ingest-lag alarm: every poll cycle compares the chain head (fetched via getLatestLedger) to the last ingested ledger. When the gap exceeds LAG_WARN_LEDGERS (default 100), a single WARN-level structured log is emitted; a single INFO log fires when the gap closes. Hysteresis keeps the alarm quiet between crossings, so a stuck indexer logs once on crossing and once on recovery rather than spamming. No log is emitted on cold start (no baseline yet) or when the alarm is disabled (LAG_WARN_LEDGERS=0).
• Topics/values are stored as JSON. When the RPC supports xdrFormat: "json" its decoding is used verbatim; otherwise the base64 XDR is decoded locally into shapes like {"symbol":"transfer"}, {"u64":42}, {"i128":"-1000"}, {"address":"C..."}.
• The raw base64 XDR is stored alongside the decoded JSON, so an improved decoder can be applied to already-indexed events — see decoder replay.

Backfilling historical events

SoroTrail's live ingester only reaches as far back as the Stellar RPC's retention window (typically 24h on the public testnet, up to a couple weeks on private deployments). For contracts whose history you care about, this leaves a permanent blind spot.

sorotrail backfill closes that gap by walking /accounts/{contract_id}/transactions on a Horizon deployment that retains historical transaction meta, decoding each result_meta_xdr through the standard pipeline so backfilled rows are indistinguishable from live-ingested ones (including raw XDR for replay).

# First-pass dry run to see what would be written
sorotrail backfill \
  --contract CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC \
  --from-ledger 1 --to-ledger 250000 --dry-run

# Then commit it
sorotrail backfill \
  --contract CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC \
  --from-ledger 1 --to-ledger 250000

It is batched, resumable (Ctrl-C and re-run picks up where it stopped), idempotent (re-runs write nothing once finished), and safe alongside live ingestion. A single-row backfill_state table holds progress so resume works across restarts; idempotent upserts make the resume overlap harmless.

Source limitations are spelled out in docs/backfill.md — notably: Horizon must retain historical meta for the target network, only Soroban V3/V4 transactions carry events, and the public Stellar testnet Horizon retains everything from protocol 17 onward while mainnet varies.

Watched ingestion (per-contract cursors)

When WATCHED_CONTRACTS is set (non-empty), SoroTrail switches from the single global cursor to a per-contract cursor model backed by the contract_cursors table.

How it works

• Each watched contract has its own resume position (last ingested ledger + pagination cursor). A contract that falls behind does not hold back the others.
• The live ingester issues one getEvents call per filter batch (the existing batching logic: ≤5 contract IDs per filter, ≤5 filters per request). Contracts at different positions within the same batch share the RPC call; each contract's cursor advances independently based on the events it received.
• Adding a contract to the watch list triggers an automatic backfill for it from latest − RETENTION_LEDGERS, clamped to the RPC's oldest retained ledger and ledger 2. This uses the same cold-start rules as an initial deployment.
• Contracts that are removed from the watch list keep their cursor row. If the contract is added back later and its cursor fell outside RPC retention, the indexer skips ahead with a warning (just like a cold start).
• Unwatched mode (empty WATCHED_CONTRACTS) keeps the single global ingestion_state row exactly as before.

How to check who is behind

GET /stats now includes a contract_cursors field — the number of per-contract cursor rows tracked. Each contract's position can be queried via the store (e.g. SELECT * FROM contract_cursors) to see which contracts are lagging.

Decoder replay

Decoders improve over time. sorotrail replay re-runs the current decoder Migrations run automatically on startup.

Configuration All configuration comes from environment variables (see .env.example ):

Variable Default Description RPC_URL https://soroban-testnet.stellar.org Stellar RPC endpoint (JSON-RPC 2.0). Point at a provider URL for mainnet. DATABASE_URL — (required) Postgres connection string. POLL_INTERVAL 5s Sleep between polls once caught up. HTTP_ADDR :8080 API listen address. WATCHED_CONTRACTS empty Comma-separated contract IDs (C...). Empty = ingest all contract events. START_LEDGER unset Force cold-start ingestion from this ledger. RETENTION_LEDGERS 17280 Cold-start reach-back in ledgers (~24h at 5s/ledger). LOG_LEVEL info debug | info | warn | error. AUDIT_ENABLED false Enable the background auditor. When unset/false the binary behaves exactly like the pre-audit build. AUDIT_POLL_INTERVAL 30s Sleep between audit passes. AUDIT_BATCH_LEDGERS 100 Ledger range covered by one audit pass. AUDIT_LAG_THRESHOLD 200 Auditor sleeps until ingest is at least this many ledgers past the verified mark. AUDIT_BUDGET_SHARE 0.10 Fraction of the request budget the audit pool gets (rest goes to ingest). AUDIT_MAX_RPS 10 Total request budget (split between ingest and audit). AUDIT_MAX_REPAIR_ATTEMPTS 3 Repair iterations before a finding is kept open as unrecoverable. AUDIT_FINDING_MAX_LEDGERS 100 Largest range a single finding is allowed to span. RATE_LIMIT_RPS unset Per-client HTTP request rate limit (requests/second). Both RATE_LIMIT_RPS and RATE_LIMIT_BURST must be set together; otherwise no rate limiting is applied. RATE_LIMIT_BURST unset Maximum instantaneous burst size for the rate limiter. Pairs with RATE_LIMIT_RPS. RATE_LIMIT_TRUSTED_PROXY false Honor X-Forwarded-For for client IP detection. Must only be enabled behind a proxy you trust to strip/rewrite the header — clients control X-Forwarded-For themselves, so enabling it on an Internet-facing surface lets any caller pick their own rate-limit key. CACHE_PRIVATE false Flip cacheable responses from Cache-Control: public to private. Set this when the deployment serves per-user data behind an auth layer (see Caching). Ingestion behavior Cold start (empty database): begins at latest ledger − RETENTION_LEDGERS (clamped to what the RPC still retains) so it captures as much recent history as possible, then follows the chain head. START_LEDGER overrides this. Warm start: resumes from the persisted cursor / last ingested ledger. Events are upserted idempotently by ID, so re-scans and restarts never duplicate rows. If the indexer is down long enough that its resume point falls out of the RPC's retention window, it logs a warning and skips ahead to the oldest retained ledger (the gap is unrecoverable from RPC — that's the problem this project exists to prevent). Requests are rate-limited (~10/s, matching public endpoint limits) and errors are retried with jittered exponential backoff. Topics/values are stored as JSON. When the RPC supports xdrFormat: "json" its decoding is used verbatim; otherwise the base64 XDR is decoded locally into shapes like {"symbol":"transfer"}, {"u64":42}, {"i128":"-1000"}, {"address":"C..."}. The raw base64 XDR is stored alongside the decoded JSON, so an improved decoder can be applied to already-indexed events — see decoder replay. This intentionally duplicates payload data in events.topics_xdr and events.value_xdr; budget extra event-table storage for deployments that retain large event histories. Decoder replay Decoders improve over time. sorotrail replay re-runs the current decoder over stored raw XDR and rewrites the decoded columns, so improvements apply to everything already indexed instead of only to future events.

Shell

sorotrail replay --from-ledger 250000 --dry-run # see what would change sorotrail replay --from-ledger 250000 # rewrite it It is batched, resumable (Ctrl-C and re-run picks up where it stopped), idempotent, and safe to run against a live database while ingestion continues; a Postgres advisory lock prevents two replays at once.

See docs/replay.md for flags, the summary output, the advisory-lock strategy, and the derivation order for dependent tables.

Compression

Responses are gzip- or deflate-encoded when the client advertises support via Accept-Encoding, which matters most for event listings — a 200-event page is largely repetitive JSON keys and compresses well.

Compression is applied per response, not per route, and only once the body reaches COMPRESS_MIN_SIZE (1400 bytes by default — roughly one Ethernet MTU). Below that, encoding costs CPU on both ends and can make the body larger once the gzip header and trailer are counted, so small responses (error envelopes, /health, a single event) are sent as-is.

Clients that don't advertise an encoding get the original bytes, byte for byte. gzip is preferred over deflate when both are offered, and gzip;q=0 is honored as a refusal rather than a low ranking.

Some things are deliberately left alone:

• WebSocket upgrades (/events/ws) bypass the middleware entirely — the response is a 101 and the connection is then taken over.
• Streaming responses that flush before reaching the threshold give up on compression rather than holding bytes back, so a live stream never stalls waiting for a buffer to fill.
• 204 and 304 carry no body, and a 304 with Content-Encoding misleads caches.
• Non-compressible media types (images, already-compressed payloads) and bodies a handler already encoded itself.

Vary: Accept-Encoding is always set, so a shared cache never serves a compressed body to a client that can't decode it. Compressing produces a different representation of the same resource, so a strong ETag is weakened ("v1" → W/"v1") when a response is encoded; conditional requests still match, since If-None-Match comparison ignores the W/ prefix.

API reference

GET /health Reports the API's view of its dependencies. 200 when both the database and the RPC are reachable and healthy, 503 otherwise.

Pagination

Every list endpoint uses cursor-based pagination with a consistent contract:

• Request parameters: ?cursor=<opaque>&limit=<int> (both optional)
• Response envelope: each list response includes a "cursor" field. When non-empty, more results exist — pass it back as ?cursor= for the next page. When empty or omitted, the result set is exhausted.
• Cursor format: the cursor is the last row's sort-key value (typically an event ID or integer primary key). It is opaque — clients must never inspect or modify it. Sending an invalid cursor returns 400 Bad Request with the standard error envelope {"error": "invalid cursor ..."}.
• Defaults: limit defaults to 50 when unset; the maximum is 500 (configurable via API_MAX_LIMIT). Values outside [1, API_MAX_LIMIT] return 400 Bad Request.
• Empty result sets return an empty array with no "cursor" field (or an empty string cursor, depending on the endpoint).

# First page
curl -s 'localhost:8080/events?limit=10'
# {"events":[...],"cursor":"0001099511627776-0000000009"}

# Next page using the cursor from the previous response
curl -s 'localhost:8080/events?cursor=0001099511627776-0000000009&limit=10'

GET /health

curl -s localhost:8080/health JSON

{"status":"ok","checks":{"database":"ok","rpc":"ok"}} GET /events Lists stored events in ascending (oldest-first) or descending (newest-first) order. Defaults to ascending.

Query parameters (all optional, combinable):

Param Example Meaning contract_id CDLZ...CYSC Only events from this contract. type contract contract | system | diagnostic. topic {"symbol":"transfer"} Exact match against any topic position. A bare word is treated as a JSON string. topic_contains [{"address":"G..."}] Postgres jsonb containment (@>) against the topics array. Pass an array to match one or more topic elements: [{"address":"G..."}] matches any event where a topic contains that address; [{"symbol":"transfer"},{"address":"G..."}] requires both. Must be parseable JSON (400 otherwise). Uses the GIN index on topics. topic0 {"symbol":"transfer"} Exact match against topic position 0. topic1 {"address":"G..."} Exact match against topic position 1. topic2 {"address":"G..."} Exact match against topic position 2. topic3 {"u64":7} Exact match against topic position 3. from_ledger 250000 Inclusive lower ledger bound. to_ledger 260000 Inclusive upper ledger bound. from_time 2026-07-21T00:00:00Z Inclusive lower created_at bound (RFC 3339). Sub-second precision and missing timezone are rejected. to_time 2026-07-22T00:00:00Z Inclusive upper created_at bound (RFC 3339). Sub-second precision and missing timezone are rejected. limit 50 Page size, 1–200 (default 50). Values outside [1, 200] or non-integers return HTTP 400. cursor 0001234... Opaque pagination cursor from a previous response. Must consist of alphanumeric characters, hyphens, underscores, dots, or colons (up to 128 characters). Malformed cursors return HTTP 400. order desc asc decoded true When true, enriches events with spec-driven named fields. Contracts without a spec return flagged raw data with "decoded": false. include_xdr true When true, includes raw base64 topics_xdr and value_xdr on each event. Omitted by default to keep responses small. Topic filters may use topic for any-position matching, or topic0..topic3 for position-specific matching. topic and positional topic filters cannot be combined.

Shell

Param	Example	Meaning
contract_id	CDLZ...CYSC	Only events from this contract.
type	contract	contract | system | diagnostic.
topic	{"symbol":"transfer"}	Exact match against any topic position. A bare word is treated as a JSON string.
topic0	{"symbol":"transfer"}	Exact match against topic position 0.
topic1	{"address":"G..."}	Exact match against topic position 1.
topic2	{"address":"G..."}	Exact match against topic position 2.
topic3	{"u64":7}	Exact match against topic position 3.
tx_hash	9f5c...	Only events from this transaction hash.
in_successful_call	true	true | false; filters by whether the enclosing call succeeded. Omitted — no constraint.
from_ledger	250000	Inclusive lower ledger bound.
to_ledger	260000	Inclusive upper ledger bound.
from_time	2026-07-21T00:00:00Z	Inclusive lower created_at bound (RFC 3339). Sub-second precision and missing timezone are rejected.
to_time	2026-07-22T00:00:00Z	Inclusive upper created_at bound (RFC 3339). Sub-second precision and missing timezone are rejected.
limit	50	Page size, 1–500 (default 50). Configurable max via API_MAX_LIMIT.
cursor	0001234...	Opaque pagination cursor from a previous response.
order	desc	asc | desc, defaults to asc. Sort direction.
order_by	created_at	id | ledger | created_at, defaults to id. Sort column. Anything else is a 400.
decoded	true	When true, enriches events with spec-driven named fields. Contracts without a spec return flagged raw data with "decoded": false.

Topic filters may use topic for any-position matching, or topic0..topic3 for position-specific matching. topic and positional topic filters cannot be combined.

Ordering and pagination

order_by picks the sort column and order the direction, so they combine:

curl -s 'localhost:8080/events?order_by=created_at&order=desc&limit=100'

Pagination stays correct under every ordering. ledger and created_at both have duplicates — a ledger holds many events, and a batch insert stamps one created_at on all of them — so those orderings sort by (column, id) and their cursors carry both halves. That keeps a page boundary landing in the middle of a run of equal values from skipping or repeating rows.

Cursors are opaque and tied to the ordering that produced them: feeding a cursor from one order_by into another returns 400. Always pass back the cursor from the previous response with the same order_by/order. Cursors issued for the default id ordering are unchanged, so existing clients keep working.

curl -s 'localhost:8080/events?contract_id=CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC&topic={"symbol":"transfer"}&limit=2'
Containment search (topic_contains) lets you filter by partial topic
structure — e.g. any event involving a specific address, even when you
don't know the full topic shape:

Shell

# Events whose topics include a specific address
curl -s 'localhost:8080/events?topic_contains=[{"address":"GA...5WI"}]&limit=5'
# Events with both a transfer symbol and a specific address
curl -s 'localhost:8080/events?topic_contains=[{"symbol":"transfer"},{"address":"GA...5WI"}]&limit=5'
Semantics: topic_contains uses Postgres jsonb containment (@>),
not substring matching. topic_contains=[{"address":"G..."}] means
"the topics array contains an element that itself jsonb-contains
{"address":"G..."}", so {"address":"G...","symbol":"transfer"}
matches. For exact element equality use topic instead.

Shell

curl -s 'localhost:8080/events?topic0={"symbol":"transfer"}&topic1={"address":"GABC..."}&topic2={"address":"GDEF..."}'
JSON

```sh
curl -s 'localhost:8080/events?topic0={"symbol":"transfer"}&topic1={"address":"GABC..."}&topic2={"address":"GDEF..."}'

{
  "events": [
    {
      "id": "0001099511627776-0000000001",
      "contract_id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
      "ledger": 256000,
      "type": "contract",
      "tx_hash": "9f5c...",
      "tx_index": 1,
      "op_index": 0,
      "in_successful_call": true,
      "topics": [{"symbol":"transfer"},{"address":"G..."},{"address":"G..."}],
      "value": {"i128":"10000000"},
      "created_at": "2026-07-16T12:00:00Z"
    }
  ],
  "cursor": "0001099511627776-0000000001"
}
cursor is present when more results exist; pass it back as ?cursor= for
the next page.

When include_xdr=true, events also include the original base64 XDR payload:

JSON

{
  "topics_xdr": ["AAAADwAAAAh0cmFuc2Zlcg=="],
  "value_xdr": "AAAACgAAAAAAAAAB"
}

Events whose topics and value match a SEP-41 token standard shape (transfer, mint, burn, clawback, approve per SEP-41 / CAP-46-6) are also tagged with a sep41_event object carrying the normalized fields — addresses stay as the original G…/C… strings, amounts stay as decimal strings (no float precision loss), muxed transfers expose to_muxed_id from the data map, and CAP-0067 trailing SEP-0011 asset strings surface as asset. Non-matching events get no extra field; the augmentation is additive, never destructive.

{
  "sep41_event": {
    "standard": "sep41",
    "event": "transfer",
    "from": "GA…",
    "to": "GB…",
    "amount": "10000000",
    "asset": "native"
  }
}

A mint / burn / clawback event omits the irrelevant side (transfer has from and to; mint has only to; burn and clawback have only from; approve has from, spender, expiration_ledger).

Time filtering narrows results and does not change ordering (events remain in ascending event-ID order, which agrees with created_at order because both follow ledger sequence).

Shell

curl -s 'localhost:8080/events?from_time=2026-07-21T14:00:00Z&to_time=2026-07-21T15:00:00Z' GET /events/{id} Fetch a single event by its ID (the TOID-based identifier from the RPC). 404 if unknown.

Shell

curl -s localhost:8080/events/0001099511627776-0000000001 GET /contracts/{id}/events Convenience wrapper for GET /events?contract_id={id}; accepts the same remaining query parameters.

Shell

curl -s localhost:8080/contracts/CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC/events?limit=10 Webhooks Consumers can register callback URLs that receive matching events as they are ingested. Delivery is asynchronous — it never blocks ingestion — and includes HMAC-SHA256 signatures so subscribers can verify payload authenticity.

POST /subscriptions Register a new webhook subscription.

Shell

curl -s -X POST localhost:8080/subscriptions
-H "Content-Type: application/json"
-d '{ "url": "https://example.com/webhook", "secret": "whsec_z8eP5qL3vR2xK9yB4w", "filters": { "contract_id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC", "type": "contract", "topic": {"symbol":"transfer"} } }' Response (201 Created):

GET /contracts/{id}/export

Streams a contract's stored events for a closed ledger range as an attachment download. Either csv (default) or ndjson is supported via ?format=csv|ndjson. Ledger bounds are required; ranges larger than EXPORT_MAX_RANGE (default 17280 ledgers, roughly 24h) return 400 with the bound included in the error.

# CSV — id, ledger, type, tx_hash, topics (JSON string), value (JSON string)
curl -OJ 'localhost:8080/contracts/CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC/export?from_ledger=250000&to_ledger=260000'

# NDJSON — one event object per line, same shape as /events
curl -OJ 'localhost:8080/contracts/CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC/export?from_ledger=250000&to_ledger=260000&format=ndjson'

The response carries Content-Disposition: attachment; filename="<id>-ledgers-<from>-<to>.<format>" so downloads land under intuitive file names, and Cache-Control: no-store keeps stale browser caches from freezing a snapshot. The store is queried in fixed-size pages (200 rows, at the MaxQueryLimit ceiling), so memory usage stays bounded regardless of the ledger span.

Ingestion behavior additions

• Parallel sweeps (SWEEP_CONCURRENCY, default 1): deployments watching more than the per-request 25 contracts split the filter set into multiple request chains paged through each SweepWindow ledger span. With the default the chains are still issued sequentially; raise SWEEP_CONCURRENCY against a private RPC to fan the chains out via bounded concurrency. The HTTPClient's interval limiter caps total request rate at ~10 req/s regardless, so the parallelism only helps when the RPC has headroom past the public ceiling. The single-batch path (<=25 watched contracts) is unchanged.
• Reorg detection (REORG_CONFIRMATION_WINDOW, default 64): after every successful ingest cycle the Run loop re-fetches the range [frontier - REORG_CONFIRMATION_WINDOW, frontier - 1] and replaces any drift via the auditor's transactional delete + insert repair (ReplaceEventsInRange). Rows past the window are treated as finalized and never rewritten — once a ledger stays more than REORG_CONFIRMATION_WINDOW behind the ingest frontier the re-scan can no longer mutate its rows. Set REORG_CONFIRMATION_WINDOW=0 to disable the re-scan entirely.

Graceful shutdown

A SIGINT or SIGTERM cancels the root context, which propagates into:

• The HTTP server: Shutdown holds open connections while they finish their current request, bounded by SHUTDOWN_TIMEOUT (default 15s).
• The ingester: Run lets the current runOnce cycle wind down to its next iteration boundary, then returns context.Canceled. ingestion_state is only advanced at the end of a successful cycle, so a cancelled cycle leaves the frontier at the last fully-persisted ledger; restart resumes from there with idempotent upserts covering any in-flight rows on the next pass.

This guarantees no panics and no truncated writes on Ctrl-C / kill, and the cursor is always persisted so the next process picks up exactly where the previous one stopped.

GraphQL API

Read-only GraphQL endpoint at POST /graphql, schema-first with Shared query-builder helpers in internal/api/queries so REST and GraphQL validate filters identically. Same EventFilter shape, same Cursor semantics, no drift between transports.

Endpoints

• POST /graphql — accepts a JSON body {"query": "...", "variables": {...}, "operationName": "..."}. Returns the standard GraphQL envelope {"data": ..., "errors": ...}.
• GET /graphql?query=... — same handler; useful for browser playground GET hits.
• GET /graphiql — dev-mode GraphiQL playground. Disabled by default; set GRAPHQL_PLAYGROUND=true to serve it.

Limits

To prevent abuse, every operation runs through two pre-flight checks before any resolver runs:

• Depth: 10 (hard cap). Anything deeper is rejected with an errors[].message describing the violation.
• Complexity: 1000 (hard cap). Connection-style fields cost 25, leaf fields 1, plus a 5-unit operation overhead. Wide fanout sits comfortably within the cap; a pathological 100k-field request is rejected before SQL touches the events table.

Both limits are configurable in internal/api/graphql/limits.go. Bumping them requires rebaselining against real client queries.

Schema highlights

Queries (read-only, no mutations):

• events(filter: EventFilterInput, page: PageInput): EventConnection!
• tokenEvents(filter: EventFilterInput, page: PageInput): TokenEventConnection!
• contracts(page: PageInput): ContractConnection!
• event(id: ID!): Event — single event lookup, null when missing.
• contract(id: ID!): Contract — single watched-contract lookup.

Filter input fields mirror the REST GET /events query parameters:

• contractId, types[], topic, topics.t0..t3, topicContains, txHash, fromLedger/toLedger, fromTime/toTime.

Pagination input is Relay-style: first/after for forward, last/before for backward (currently unsupported and rejected with an error message — see queries.ResolvePage). Sort: order

• orderBy (defaults ASC/id).

Example 1 — list events with filter + cursor pagination

query EventsForContract($id: ID!, $first: Int!) {
  events(
    filter: { contractId: $id, type: CONTRACT }
    page: { first: $first, orderBy: LEDGER, order: ASC }
  ) {
    edges {
      cursor
      node { id contractId ledger type topics value createdAt }
    }
    nodes { id ledger createdAt }
    pageInfo { hasNextPage endCursor }
    totalCount
  }
}

Variables:

{
  "id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
  "first": 10
}

Response (200 OK):

{
  "data": {
    "events": {
      "edges": [
        {
          "cursor": "eyJpZCI6IjAwMDEwOTk1MTE2Mjc3NzYtMDAwMDAwMDAwMSIsIm9yZGVyX2J5IjoiaWQiLCJvcmRlciI6ImFzYyJ9",
          "node": {
            "id": "0001099511627776-0000000001",
            "contractId": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
            "ledger": 256000,
            "type": "contract",
            "topics": [{"symbol":"transfer"}, {"address":"G..."}],
            "value": {"i128":"10000000"},
            "createdAt": "2026-07-16T12:00:00Z"
          }
        }
      ],
      "nodes": [
        {
          "id": "0001099511627776-0000000001",
          "ledger": 256000,
          "createdAt": "2026-07-16T12:00:00Z"
        }
      ],
      "pageInfo": { "hasNextPage": true, "endCursor": "eyJpZCI6..." },
      "totalCount": 4321
    }
  }
}

Pass endCursor back as page.after to retrieve the next page. Cursors are opaque — base64-JSON {LastID, OrderBy, Order} — so clients should treat them as strings.

Example 2 — list token events with spec decoding

query TokenEvents($id: ID!) {
  tokenEvents(
    filter: { contractId: $id, topic: "transfer", topicContains: [{"symbol":"transfer"}] }
    page: { first: 5, orderBy: LEDGER, order: DESC }
  ) {
    edges { cursor node { id ledger } }
    nodes {
      id
      decoded
      decodedEvent { name fields }
      topics
      value
    }
    pageInfo { hasNextPage endCursor }
    totalCount
  }
}

Variables:

{ "id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC" }

Response (200 OK; decodedEvent.fields is only populated when decoded == true):

{
  "data": {
    "tokenEvents": {
      "edges": [{"cursor": "...", "node": {"id": "...", "ledger": 256100}}],
      "nodes": [
        {
          "id": "0001099511627776-0000000042",
          "decoded": true,
          "decodedEvent": {
            "name": "transfer",
            "fields": {"from": "G...", "to": "G...", "amount": "10000000"}
          },
          "topics": [{"symbol":"transfer"}, {"address":"G..."}, {"address":"G..."}],
          "value": {"i128":"10000000"}
        }
      ],
      "pageInfo": { "hasNextPage": false, "endCursor": null },
      "totalCount": 1
    }
  }
}

Filter + pagination parity with REST

Both transports share internal/api/queries: type whitelists, topic positional rules, ledger/time bounds, cursor validation, page-size caps are identical. A REST request rejected with 400 invalid cursor returns the same errors[].message from GraphQL. Conversely a GraphQL topic/topic0 conflict returns the same errors[].message as the REST /events?topic=…&topic0=… handler.

Webhooks

Consumers can register callback URLs that receive matching events as they are ingested. Delivery is asynchronous — it never blocks ingestion — and includes HMAC-SHA256 signatures so subscribers can verify payload authenticity.

POST /subscriptions

Register a new webhook subscription.

curl -s -X POST localhost:8080/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/webhook",
    "secret": "whsec_z8eP5qL3vR2xK9yB4w",
    "filters": {
      "contract_id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
      "type": "contract",
      "topic": {"symbol":"transfer"}
    }
  }'

Response (201 Created):

{
  "id": 1,
  "url": "https://example.com/webhook",
  "filters": {
    "contract_id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
    "type": "contract",
    "topic": {"symbol":"transfer"}
  },
  "secret": "whsec_z8eP5qL3vR2xK9yB4w",
  "enabled": true,
  "failure_count": 0,
  "created_at": "2026-07-24T12:00:00Z"
}

filters has the same shape as GET /events query parameters — an empty object {} matches every event. enabled defaults to true.

GET /subscriptions

List all subscriptions.

curl -s localhost:8080/subscriptions

GET /subscriptions/{id}

Fetch a single subscription by ID. 404 if unknown.

curl -s localhost:8080/subscriptions/1

PUT /subscriptions/{id}

Update a subscription. Only fields present in the body are changed; omit a field to leave it unchanged.

curl -s -X PUT localhost:8080/subscriptions/1 \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

DELETE /subscriptions/{id}

Delete a subscription and its delivery history. 204 No Content on success.

curl -s -X DELETE localhost:8080/subscriptions/1

GET /subscriptions/{id}/deliveries

List delivery attempts for a subscription, newest first. Optional ?limit= (default 50, max configurable via API_MAX_LIMIT, default 500).

curl -s localhost:8080/subscriptions/1/deliveries?limit=10

[
  {
    "id": 42,
    "subscription_id": 1,
    "event_id": "0001099511627776-0000000001",
    "status": "success",
    "response_code": 200,
    "duration_ms": 87,
    "created_at": "2026-07-24T12:01:00Z"
  },
  {
    "id": 41,
    "subscription_id": 1,
    "event_id": "0001099511627776-0000000000",
    "status": "failed",
    "response_code": 500,
    "duration_ms": 1024,
    "error": "HTTP 500",
    "created_at": "2026-07-24T12:00:55Z"
  }
]

Webhook payload

When an ingested event matches a subscription's filters, SoroTrail POSTs the event to the subscription's URL with this body:

{
  "event": {
    "id": "0001099511627776-0000000001",
    "contract_id": "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC",
    "ledger": 256000,
    "type": "contract",
    "tx_hash": "9f5c...",
    "tx_index": 1,
    "op_index": 0,
    "in_successful_call": true,
    "topics": [{"symbol":"transfer"},{"address":"G..."},{"address":"G..."}],
    "value": {"i128":"10000000"},
    "created_at": "2026-07-24T12:00:00Z"
  }
}

Every request includes an X-SoroTrail-Signature header holding the hex-encoded HMAC-SHA256 digest of the request body, keyed with the subscription's secret. Subscribers must verify this signature to confirm the payload came from SoroTrail and has not been tampered with.

When the event matches the SEP-41 token standard, the payload also includes the sep41_event envelope described under GET /events — subscribers can rely on payload.event.sep41_event (inside the existing event field of the posted JSON) to identify the transfer / mint / burn / clawback / approve semantics without re-implementing SEP-41 themselves. The signature is computed over the full body including the sep41_event field, so subscribers who add or remove the field would change the signature and fail verification.

Verifying signatures — code samples:

Go

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "io"
    "net/http"
)

func verifySignature(r *http.Request, secret string) ([]byte, bool) {
    body, _ := io.ReadAll(r.Body)
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(body)
    expected := hex.EncodeToString(mac.Sum(nil))
    if !hmac.Equal([]byte(r.Header.Get("X-SoroTrail-Signature")), []byte(expected)) {
        return nil, false
    }
    return body, true
}

Python

import hmac
import hashlib

def verify_signature(request_body: bytes, signature_header: str, secret: str) -> bool:
    expected = hmac.new(secret.encode(), request_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header)

# Example Flask endpoint:
# @app.route("/webhook", methods=["POST"])
# def webhook():
#     body = request.get_data()
#     if not verify_signature(body, request.headers.get("X-SoroTrail-Signature", ""), SECRET):
#         return "bad signature", 401
#     payload = json.loads(body)
#     # process payload["event"]

JavaScript (Node.js)

const crypto = require('crypto');

function verifySignature(body, signatureHeader, secret) {
  const expected = crypto.createHmac('sha256', secret).update(body).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(signatureHeader));
}

// Example Express endpoint:
// app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
//   const sig = req.headers['x-sorotrail-signature'] || '';
//   if (!verifySignature(req.body, sig, SECRET)) return res.status(401).end();
//   const { event } = JSON.parse(req.body);
//   // process event
// });
</details><details> <summary>TypeScript (Bun / Deno)</summary>
TypeScript

TypeScript (Bun / Deno)

async function verifySignature(req: Request, secret: string): Promise<boolean> {
  const body = await req.arrayBuffer();
  const key = await crypto.subtle.importKey('raw', new TextEncoder().encode(secret),
    { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']);
  const expected = await crypto.subtle.sign('HMAC', key, body);
  const expectedHex = Array.from(new Uint8Array(expected))
    .map(b => b.toString(16).padStart(2, '0')).join('');
  const sigHeader = req.headers.get('X-SoroTrail-Signature') || '';
  return crypto.subtle.timingSafeEqual(
    new TextEncoder().encode(expectedHex),
    new TextEncoder().encode(sigHeader)
  );
}
</details>
Delivery semantics
Delivery is at-least-once: a subscriber may receive the same event more
than once. Deduplicate by event id.
Failed deliveries are retried up to 5 times with exponential backoff
(1s → 2s → 4s → 8s → 16s).
After 5 consecutive failures the subscription is auto-disabled. A
successful delivery resets the failure counter to 0.
Delivery attempts are recorded in delivery_attempts and queryable via
GET /subscriptions/{id}/deliveries.
Subscribers should return a 2xx status code to acknowledge receipt.
Non-2xx responses are treated as failures and retried.
GET /stats
Shell

Delivery semantics

• Delivery is at-least-once: a subscriber may receive the same event more than once. Deduplicate by event id.
• Failed deliveries are retried up to 5 times with exponential backoff (1s → 2s → 4s → 8s → 16s).
• After 5 consecutive failures the subscription is auto-disabled. A successful delivery resets the failure counter to 0.
• Delivery attempts are recorded in delivery_attempts and queryable via GET /subscriptions/{id}/deliveries.
• Subscribers should return a 2xx status code to acknowledge receipt. Non-2xx responses are treated as failures and retried.

GET /stats

curl -s localhost:8080/stats JSON

{"total_events":18234,"last_ingested_ledger":260123,"verified_through_ledger":258900,"oldest_stored_ledger":242001,"chain_head_ledger":260130,"ingest_lag_ledgers":7,"contract_count":57,"watched_contracts":0,"auditor":{"passes_run":87,"ledgers_checked":1200,"findings_opened":2,"findings_repaired":1,"findings_unverifiable":0,"findings_unrecoverable":1,"rpc_requests":340}}

oldest_stored_ledger is the lowest ledger currently present in the store. chain_head_ledger is read from Stellar RPC getHealth, and ingest_lag_ledgers is chain_head_ledger - last_ingested_ledger. If the RPC is temporarily unreachable, /stats still returns HTTP 200 with the stored fields populated and the RPC-derived freshness fields (chain_head_ledger, ingest_lag_ledgers) set to null.

verified_through_ledger is the inclusive highest ledger whose stored events have been proven to match a fresh RPC fetch by the auditor. When AUDIT_ENABLED=false it stays at 0. See the Data integrity section below for the contract the field implies.

GET /metrics

Serves http_request_duration_seconds, a Prometheus histogram of HTTP request latency labeled by route (the matched chi route pattern, e.g. /events/{id} — never the raw path, so path parameters don't blow up cardinality), method, and status.

curl -s localhost:8080/metrics | grep http_request_duration_seconds

Exempt from the rate limiter for the same reason /health is: a Prometheus scraper polling this endpoint on its own schedule shouldn't be throttled like a regular client.

GET /events/ws (WebSocket live stream)

Pushes ingested events to the client over a single WebSocket connection as soon as they are written to the store. There is no replay — the stream starts at "now", and the client only sees events the indexer ingests after it connects.

Query parameters share the same EventFilter shape as GET /events (contract_id, type, topic, from_ledger, to_ledger, from_time, to_time), so any filter that works against the query Query parameters share the same EventFilter shape as GET /events (contract_id, type, topic, from_ledger, to_ledger, from_time, to_time), so any filter that works against the query API works against the stream.

Frame format:

One store.Event per WebSocket text frame, JSON-encoded. Server-to-client only — clients do not send messages; the nhooyr.io/websocket library handles ping/pong internally. The server pings every 30s so proxies don't idle the connection. Behavior:

Slow-consumer eviction: each subscriber gets a bounded channel buffer (broadcast.DefaultBufferSize = 64). A subscriber whose channel fills is evicted silently: its Events() channel is closed, the handler returns, and the WebSocket is closed from the server side. This protects the indexer from one stuck client back-pressuring the broadcaster. Broadcaster unwired: returns 501 Not Implemented (only happens if the binary was built without the broadcaster wired). Bad filter: returns 400 Bad Request before the WebSocket upgrade, with the standard {"error": "..."} JSON body. Example with websocat:

Shell

websocat 'ws://localhost:8080/events/ws?contract_id=CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC&topic=mint' {"id":"…","contract_id":"…","topics":["mint"], …} {"id":"…","contract_id":"…","topics":["mint"], …} Data integrity

• One store.Event per WebSocket text frame, JSON-encoded.
• Server-to-client only — clients do not send messages; the nhooyr.io/websocket library handles ping/pong internally.
• The server pings every 30s so proxies don't idle the connection.

Behavior:

• Slow-consumer eviction: each subscriber gets a bounded channel buffer (broadcast.DefaultBufferSize = 64). A subscriber whose channel fills is evicted silently: its Events() channel is closed, the handler returns, and the WebSocket is closed from the server side. This protects the indexer from one stuck client back-pressuring the broadcaster.
• Broadcaster unwired: returns 501 Not Implemented (only happens if the binary was built without the broadcaster wired).
• Bad filter: returns 400 Bad Request before the WebSocket upgrade, with the standard {"error": "..."} JSON body.

Example with websocat:

websocat 'ws://localhost:8080/events/ws?contract_id=CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC&topic=mint'
{"id":"…","contract_id":"…","topics":["mint"], …}
{"id":"…","contract_id":"…","topics":["mint"], …}

Data integrity

A background auditor walks recently-ingested ledger ranges (behind the ingest frontier, inside the RPC's retention window) and re-fetches each range with the same filter configuration the ingester uses, comparing the stored event counts and IDs against the fresh response. Mismatches are logged, recorded in the audit_findings table, and auto-repaired by re-ingesting the affected range with ReplaceEventsInRange (which deletes orphans and updates same-ID rows so topic/value drift on the RPC side is corrected).

Each pass advances an audit-only verified_through_ledger high-water mark past the clean prefix of the audited range; that field, exposed via GET /stats, is the strongest trust signal SoroTrail can offer: it names the highest ledger whose stored events have been verified against the RPC, not merely ingested.

Audit behaviour:

Filter parity: the auditor uses the ingester's exact filter batch (see Ingester.BuildFilterBatches), so events the RPC has for contracts you're not watching are intentionally not checked and never produce false findings. Idempotency: re-running the auditor over a clean range is a no-op; crashes mid-repair leave the finding open so the next pass can retry. Budget: the auditor shares the request-rate budget with the ingester via rpc.Budget; AUDIT_BUDGET_SHARE (default 10%) caps the audit pool while the ingest pool gets the remainder. Lag pause: if ingest hasn't moved at least AUDIT_LAG_THRESHOLD ledgers past verified_through_ledger, the auditor sleeps until it does — it never races ingestion. Retention edges: when a finding's range ages out of the RPC's retention window during repair, the auditor moves the finding to status='unverifiable' instead of crashing or false-alarming. Self-disagreement: if the RPC keeps returning different events for the same range across repair iterations, the auditor stops after AUDIT_MAX_REPAIR_ATTEMPTS attempts and keeps the finding visible with status='unrecoverable' — operators see it via /stats. Set AUDIT_ENABLED=false (the default) to disable the auditor entirely; the binary's behavior is identical to a pre-audit build.

Retention / pruning

SoroTrail exists because the RPC drops history, but “keep everything forever” is not viable for everyone indexing busy contracts. The optional pruner bounds on-disk growth without requiring an external cron job.

Configure one or both policies and the pruner runs in its own goroutine on the same lifecycle as the ingester:

• RETENTION_MAX_AGE — delete events whose created_at is older than the duration (e.g. RETENTION_MAX_AGE=2160h keeps ~90 days).
• RETENTION_MIN_LEDGER — delete events strictly below this ledger. Events at exactly RETENTION_MIN_LEDGER are kept. The SQL is ledger < bound.

If neither is set the pruner is a no-op goroutine that returns without deleting anything — matching the pre-pruner behaviour. Tuning knobs:

Variable	Default	Effect
RETENTION_BATCH_SIZE	5000	Rows per DELETE statement; smaller = less lock pressure, slower.
RETENTION_PAUSE	100ms	Sleep between DELETE batches so ingestion stays responsive.
RETENTION_INTERVAL	1h	Sleep after a sweep that found nothing to delete.

Safety guard. The pruner never deletes an event whose ledger is at or above the last ingested ledger. RETENTION_MIN_LEDGER is therefore an upper bound on the cutoff whenever it is lower than the last ingested ledger, so a misconfiguration above the chain head (e.g. setting RETENTION_MIN_LEDGER=200 when only ledger 100 has been ingested) is a no-op rather than a destructive mistake. Pruning runs in batches of RETENTION_BATCH_SIZE with a RETENTION_PAUSE between batches; rows are not selected in any particular order — the safety guarantee is the ledger < last_ingested_ledger predicate inside DELETE FROM events, not the scan order, so the ingester is never starved for IO.

Metrics. GET /stats adds a pruner object when retention is configured:

{
  "total_events": 12345,
  "last_ingested_ledger": 260123,
  "..."": "...",
  "pruner": { "runs_completed": 7, "total_rows_purged": 1804321 }
}

runs_completed is the number of full sweeps; total_rows_purged is the cumulative rows deleted since the process started.

Lifecycle. The pruner is a single background goroutine started from cmd/sorotrail/main.go alongside the ingester (and, when enabled, the auditor). On SIGINT / SIGTERM every goroutine drains through a shared error channel and the pruner exits the same way as the others — a context-cancelled return is not an error.

Dependent tables. When the SEP-41 token_events table lands, it will be defined with a foreign key to events(id) and ON DELETE CASCADE. The pruner therefore deletes only from events and lets Postgres cascade the dependent rows; today's single-table DELETE is correct because no such table exists yet, and the contract is “derived tables ride along”.

Caching

Stored events are immutable — a row written by ingest is never rewritten in normal operation — so the API serves two distinct cache policies:

• GET /events/{id} carries a strong ETag equal to the event ID and Cache-Control: public, max-age=31536000, immutable. Clients and CDNs can cache the response for a year without revalidating. Conditional requests with a matching If-None-Match return 304 Not Modified without re-serializing the row.
• List endpoints (/events, /contracts/{id}/events) split on a moving ingest frontier: a page whose upper bound (to_ledger) sits entirely below the last-ingested ledger cannot gain new rows, so the response is declared immutable with a strong ETag derived from the filter — every filter parameter that narrows the result set, so two different queries can never share a validator. Pages that are open-ended (to_ledger unset) or have bounds at/above the frontier are still growing, so they get Cache-Control: no-cache — a deliberate "when in doubt, don't cache" choice rather than a guess with a short max-age.
• GET /health and GET /stats are always Cache-Control: no-store so monitoring tooling, dashboards, and alerting see real state rather than a stale replica.

All cacheable responses set Vary: Accept-Encoding so a future compression middleware (#25) can serve distinct encoded variants without reconciling caches that warmed on a non-encoded version.

Retention/pruning (#8) Immutability is conditional on the row existing. When pruning deletes an event that was previously cached by a client or CDN, that cache will hold a stale copy until its max-age expires — clients can hit the fresh 404 immediately by sending their If-None-Match validator, at which point SoroTrail's EventExists probe correctly returns the not-found status. We accept this self-healing delay rather than arbitrarily shortening the immutable max-age, because for un-deleted rows the long max-age is the whole point of the cache.

Auth'd deployments

Caching must never leak across keys. The CACHE_PRIVATE flag flips every cacheable response from Cache-Control: public to Cache-Control: private (same max-age and immutable preset), so the same build can serve shared caches in one deployment and per-user scenarios in another. Set CACHE_PRIVATE=true whenever an auth layer sits in front of the API.

Under MULTI_TENANT=true this is not left to configuration. Responses are forced to private, Vary gains Authorization and X-API-Key, and the ETag incorporates a digest of the caller's scope — so two tenants issuing identical requests cannot share a cache entry, and a conditional request carrying another tenant's validator cannot be answered 304. The last of those matters most: it is the only one of the three that does not need a CDN to misbehave. See Caching.

Development Shell

make build # compile to bin/sorotrail make test # unit tests (Postgres tests skip without a database) make test-db # full suite incl. Postgres integration tests make lint # golangci-lint make migrate-up # apply migrations manually (needs the migrate CLI)

See [docs/architecture.md](docs/architecture.md) for the full system architecture
diagram and component descriptions. [CONTRIBUTING.md](CONTRIBUTING.md) covers extension
points and development conventions.

Roadmap / future work
Deliberately out of scope for the MVP, with seams left for contributors:

- Per-standard event decoders (e.g. SEP-41 token transfers) on top of
  `decode.Decoder`.
- Per-contract cursors (mesh-point for the parallel sweep redesign:
  the scheduler is now parallel but still shares one cursor per
  window).
- GraphQL / websocket subscriptions.
- Metrics (Prometheus) and tracing.
- Alternative storage backends behind `store.Store`.

## API documentation UI

The OpenAPI 3.1 specification lives at [`api/openapi.yaml`](api/openapi.yaml) and
is browsable through a self-hosted Swagger UI at `/docs` when the server is
running:

http://localhost:8080/docs/

No external CDN is required — all assets (HTML, CSS, JS) are compiled into the
binary via Go's `//go:embed` mechanism.

### Route-drift validation

A dedicated test ensures the router and the OpenAPI spec stay in sync:

```sh
go test ./pkg/docs/ -run TestNoRouteDrift -v

The test reads api/openapi.yaml, walks the live chi router tree, and fails with t.Fatalf if the two diverge in either direction. Run it as part of CI to catch endpoint/spec drift before it reaches production.

License

Apache-2.0