feat(indexer): concurrency locks + aggregation index for indexer_metrics_collector, migration hooks + diagnostics for database_writer_pool

[jonah-Elisha10]

Implements four indexer issues across two modules.

Closes #336
Closes #335
Closes #331
Closes #328

---

#336 — Race conditions in concurrent indexer_metrics_collector calls

Adds IndexerMetricsEventQueue, a bounded in-memory queue that serializes event inserts per identity (contractId|ledgerSequence|eventType, matching the events table's UNIQUE constraint).

The race was the window between the "already indexed?" check and the insert: two concurrent notifications carrying the same event could both observe "not indexed" and both write. The persisted-key set is now consulted inside the per-identity lock, so exactly one caller writes each row while unrelated events still persist concurrently. Locks are released on throw, enqueue/flush/submit are separable, and maxQueueSize bounds memory.

collectIndexerMetricsAsync() adds single-flight de-duplication on top: concurrent collections drain the queue once and share one snapshot instead of racing several transactions over the same tables.

#335 — SQLite index structures for indexer_metrics_collector

Plans were measured before choosing indexes. The events-by-type aggregation was the only query with a real problem:

before:  SCAN events USING COVERING INDEX idx_events_ledger_event_type | USE TEMP B-TREE FOR GROUP BY
after:   SCAN events USING COVERING INDEX idx_events_event_type

Migration 6 adds idx_events_event_type. Candidate indexes for the webhook-subscription and active-contract counts were measured and rejected — those plans already resolve through covering indexes, and a dead index only costs write throughput.

The exact statements the collector runs are exported as INDEXER_METRICS_QUERIES, so the query-plan assertions verify the real queries rather than a copy that can drift. INDEXER_METRICS_INDEXES names every index the collector's lookups depend on (including the two from earlier migrations), and verifyIndexerMetricsIndexes() reports any that go missing.

#331 — Migration verification hooks in database_writer_pool

• verifyWriterPoolSchema() — reports missing migrations, tables, and columns without throwing, for callers that want to log or degrade.
• assertWriterPoolSchemaReady() — throws WriterPoolSchemaError carrying the issue list.
• startWriterPool() — runs both, plus any hooks registered through registerMigrationVerificationHook(), and fails when the database state is out of sync. A hook that throws is reported as an issue rather than escaping the start call.

Start enforcement is opt-in via startWriterPool({ enforce: true }). Making a successful start mandatory for writes would have broken every existing caller that uses queueWrite directly, so the default keeps current behaviour and enforcement is something a process opts into.

#328 — Polling diagnostics logs for database_writer_pool

High-frequency debug diagnostics across the write path — enqueue, each write attempt, retries, failures, queue drains, and pool start. Every diagnostic message string carries elapsedMs=, plus payloadSizeBytes=, queueDepth= and attempt= where known, with the same values repeated in structured meta for log processors.

database_writer_pool poll diagnostics operation=insert-event status=success elapsedMs=1.204 payloadSizeBytes=42 queueDepth=0 attempt=1
database_writer_pool poll diagnostics operation=drain_write_queue status=success elapsedMs=3.881 queueDepth=0 attempt=3

---

Tests

Four new test files, 103 tests, all passing:

File	Covers
__tests__/indexer-metrics-collector-concurrency.test.ts	The required check — concurrent notifications do not duplicate entries (6 concurrent submits of the same batch → 20 rows, 20 inserts); no two persists overlap for one identity while unrelated ones do; lock release on error; single-flight collection; queue overflow
__tests__/indexer-metrics-collector-indexes.test.ts	The required check — EXPLAIN QUERY PLAN asserts indexes are used for every collector lookup, including dropping idx_events_event_type to prove it is load-bearing (the plan falls back to a temp B-tree without it)
__tests__/database-writer-pool-migration-hooks.test.ts	The required check — start fails if the database state is out of sync: missing migrations table, un-applied migration versions, dropped tables, missing columns, and failing custom hooks
__tests__/database-writer-pool-diagnostics.test.ts	The required check — every diagnostic message string contains a numeric elapsed time; payload sizes scale with returned data; queue depth, retry, failure, drain, and start diagnostics

One existing assertion needed updating: __tests__/indexer.test.ts pins the number of shipped migrations, which migration 6 changes from 5 to 6.

Full suite: 938 passed, 5 failed.

⚠️ Those 5 failures are pre-existing on main and untouched by this PR — verified by stashing these changes and re-running on a clean checkout, where they fail identically:

• __tests__/sqlite-schema-manager.test.ts (2) — references an unexported SCHEMA_MANAGER_INDEXES, plus type errors
• __tests__/failover-recovery-backoff-retry.test.ts (2)
• __tests__/failover-recovery-poll-diagnostics.test.ts (1) — uses the bare jest global, which is not injected under this repo's ESM Jest config

The same applies to the npx tsc --noEmit step in CI: 7 errors, all in test files this PR does not touch, all present on main. tsc -p tsconfig.build.json (the production build) is clean. Happy to fix that pre-existing breakage in a follow-up if wanted.

ℹ️ Note for merge ordering: PR #367 also modifies src/indexer/indexer_metrics_collector.ts (alerting and diagnostics, issues #338/#337). The two touch different parts of the file, but whichever merges second will need a conflict resolution.

🤖 Generated with Claude Code

[drips-wave]

@jonah-Elisha10 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits