docs: performance investigations for importers pagination, oracle feed contention, metrics refresh, and GIN index - #1172
Merged
vjuliaife merged 4 commits intoAug 28, 2026
Conversation
Found idx_importers_created_at was never actually applied to any real database (vjuliaife#257's closing PR added it only to a legacy, unexecuted .sql migration file — grepped every real migration and db.ts for zero hits). Adds it via a proper migration (0006_importers_and_events_perf_indexes.ts), turning the surety_admin listing query from Seq Scan + Sort into an Index Scan. Cursor-based pagination is recommended but not implemented: the only current caller renders the full list with no pagination UI, and a correct keyset cursor needs a compound (created_at, id) key (id alone is a random UUID with no relation to insertion order) — implementing it speculatively ahead of a consuming UI change is out of scope for this investigation. No live Postgres was available to capture EXPLAIN ANALYZE at 1x/5x/10x volume; the Seq Scan -> Index Scan conclusion is a schema-level deduction (no matching index existed; one now does), not a measured result. See docs/investigations/importers-list-pagination-at-scale.md. Closes vjuliaife#1090
Structural analysis (not live-benchmarked — no Postgres instance available) of every read/write call site against oracle_price_feed: it's insert-only with no UPDATE anywhere, uses a random UUID PK (avoiding the rightmost-page contention pattern a SERIAL PK would have), and dedups via an ON CONFLICT DO NOTHING unique index (an index-level operation, not a table lock). Postgres MVCC means concurrent SELECTs are never blocked by concurrent INSERTs at any frequency. No index or schema change is warranted. See docs/investigations/oracle-price-feed-write-contention.md. Closes vjuliaife#1092
…eeds The materialized view's REFRESH CONCURRENTLY (confirmed in use, db.ts:953-959, so concurrent readers are never blocked) still has to recompute the full aggregate across ALL importers/bonds/events on every refresh, and refreshImporterMetricsView() is awaited synchronously on every single tariff-CSV upload. Before this fix, a slow/failed refresh (lock wait, pool exhaustion) would throw past the point where the on-chain collateral update and tariff_uploads row had already succeeded, turning a successful upload into a client-visible 500. Wrapped in its own try/catch, matching the adjacent evaluateTariffAlerts pattern. No live Postgres was available to benchmark actual refresh duration at simulated 10x volume; the refresh-cost-scales-with-total-volume conclusion follows from the view's own LEFT JOIN/aggregate definition. See docs/investigations/importer-metrics-mv-refresh-cost.md. Closes vjuliaife#1091
Two findings more fundamental than the scan-performance question this issue asks: (1) idx_contract_events_raw_gin was never actually applied to any real database, same unexecuted-legacy-migration situation as vjuliaife#1090's idx_importers_created_at; (2) contract_events.raw is functionally dead code — every INSERT omits it (checked all 4 call sites) and no SELECT anywhere reads or filters on it (checked every route file). Recommendation: do NOT add the GIN index — it would add write overhead to the busiest, already-partitioned table in the schema for zero query benefit, since nothing queries the column it would index. See docs/investigations/contract-events-gin-index-at-scale.md. Closes vjuliaife#1093
|
@pre-cious-Igwealor Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four performance investigations, each documented in
docs/investigations/with a code-level analysis and a concrete recommendation (and, where warranted, an applied fix):idx_importers_created_at(from Run EXPLAIN ANALYZE on All Hot-Path Database Queries and Add Missing Indexes #257) was never actually applied to any real database — it only ever landed in a legacy, unexecuted.sqlmigration file. Added it properly (0006_importers_and_events_perf_indexes.ts), turning the surety_admin listing from Seq Scan + Sort into an Index Scan. Cursor-based pagination is recommended for later but not implemented now — the only current caller has no pagination UI, and a correct keyset cursor needs a compound(created_at, id)key.ON CONFLICT DO NOTHING, MVCC means readers are never blocked by writers. No schema change warranted.REFRESH CONCURRENTLYis confirmed in use (protects concurrent readers), but the refresh cost itself scales with total system-wide bond/event volume and was previously unguarded — a slow/failed refresh could turn a successful tariff-CSV upload into a client-visible 500. Fixed by wrapping the refresh in its own try/catch, matching the adjacentevaluateTariffAlertspattern.contract_events.rawis functionally dead code — no INSERT populates it, no SELECT reads it. Recommendation: do NOT add the index; it would add write overhead to the busiest table in the schema for zero query benefit.Closes #1090
Closes #1092
Closes #1091
Closes #1093
Disclosure
No live Postgres instance was available in the environment these investigations were originally done in, so the acceptance criteria's benchmark/EXPLAIN ANALYZE/throughput measurements were not captured — each doc's "Reproducing / measuring" section discloses this explicitly and distinguishes code-level/schema-level deductions (verified) from measured results (not captured). Where a real, safe fix was identified independent of needing live numbers (the two missing indexes, the try/catch guard), it was implemented and verified via
tsc --noEmit/eslint.Test plan
npm run typecheck --workspace=apps/api— cleannpm run lint --workspace=apps/api— clean (eslint --max-warnings 0)0006_importers_and_events_perf_indexes.ts) reviewed for correctness against the existing migration runner's transaction-per-migration model (non-concurrentCREATE INDEX, sinceCONCURRENTLYcan't run inside a transaction block)