Fix/indexer scalability and readiness - #159
Open
laddyr141-ui wants to merge 5 commits into
Open
Conversation
Merging main into feat/api-hardening (59c9fdd) silently dropped the rand, metrics, and metrics-exporter-prometheus dependencies from Cargo.toml along with the PrometheusHandle/PrometheusBuilder imports and AppState::last_indexed_ledger, leaving main uncompilable. Restore the dependencies and imports, and re-add last_indexed_ledger adapted to the ArcSwap-backed snapshot (routes/mod.rs::cache_headers already calls it). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each refresh cycle read A assets fully sequentially, and within each asset issued two sequential RPC calls (get_record, balance) per allowlisted address — roughly A*(1 + 2N) sequential simulate calls every 10s poll. This doesn't scale as assets/addresses grow and risks tripping public-RPC rate limits. Index assets concurrently via a bounded futures::stream::buffer_unordered (MAX_CONCURRENT_ASSETS), and within each asset read every address's compliance record and balance concurrently as well (bounded by MAX_CONCURRENT_ADDRESS_READS), running the two per-address calls themselves in parallel via tokio::join!. Asset order is preserved by tagging each result with its original position and re-sorting after the unordered stream completes. Error-handling semantics are unchanged: a metadata/holder read failure still fails the asset (and, as before, the whole refresh) — that's addressed separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reqwest client was built with reqwest::Client::new() — no timeout or connect_timeout. A stalled RPC (no response, half-open connection) made the await in a read hang indefinitely, freezing the single-task refresh loop and serving an increasingly stale snapshot with no recovery. Build the client with .timeout()/.connect_timeout(), and additionally wrap each attempt in tokio::time::timeout as a second bound. A timeout is now a distinct IndexError::Timeout variant, treated as transient so it participates in the existing retry/backoff logic rather than immediately failing the read. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
refresh() used `?` on the per-asset metadata/holders reads, so a
single asset's token contract failing to decode or an RPC read
erroring (after retries) failed the whole refresh and left every
asset stale — not just the bad one. index_dividends was already
resilient (unwrap_or_default); the rest wasn't, so recovery behavior
was inconsistent.
A failing asset is now collected as a per-asset error (logged and
counted via rwa_indexer_asset_read_errors_total{read="asset"}) and
falls back to that asset's data from the previous snapshot when one
exists, rather than aborting the refresh. Every other asset still
publishes fresh. An asset with no prior snapshot (e.g. first refresh
after it's registered) and no successful read is omitted for that
cycle rather than fabricated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
health returned {"status":"ok"} unconditionally: on boot the snapshot
is Snapshot::default() (empty, last_updated: None), and if the indexer
never succeeds the API still reported healthy while serving empty
data. There was no readiness signal or staleness threshold.
Add GET /ready: 200 once the snapshot has been updated within the
last STALE_AFTER_SECS (3 missed poll intervals), 503 otherwise, with
the snapshot age and threshold in the body. /health stays a pure
liveness probe (process is up); /ready reflects whether the served
data is trustworthy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@laddyr141-ui Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
/healthalways returns 200 ok even when the indexer has never succeeded or is far stale #4