Integrate all 25 open contributor PRs - #396
Merged
Merged
Conversation
duplicate_prevention's correctness (UNIQUE(contract_id, ledger_sequence, event_type) + INSERT OR IGNORE in db.ts, atomic pointer advance in insertEventBatch) does not depend on poll frequency - every poll resumes from the last committed ledger pointer, so a slower interval can only delay detection, never cause a missed/duplicate/out-of-order event. Backing off is therefore safe as long as it stays bounded. pollEvents() now reports whether the ledger advanced (network activity). startPoller() uses that signal to multiplicatively back off the interval while idle (POLL_IDLE_BACKOFF_MULTIPLIER, default 1.5x) up to a capped POLL_INTERVAL_MAX_MS (default 120s), and resets immediately to POLL_INTERVAL_MS (default 15s, unchanged) the moment a new ledger closes. Follows the existing parseInt(process.env.X || "default") config convention already used for POLL_INTERVAL_MS - no new config mechanism introduced. Assumption (flagging per ticket ambiguity): "network idle" is defined as "no new ledger closed since the last poll" (currentLedger <= lastLedger), not "ledger advanced but zero matching events" - the ledger closing is the literal signal that the network itself is active, independent of whether this app's contracts happened to emit anything. A failed poll (RPC error) is also treated as idle, so transient RPC failures back off rather than hammering a struggling endpoint. Note for a human decision: this poll loop (src/indexer/poller.ts) is the same loop Ticket 4's high-frequency debug logging will instrument. Numbers this ticket introduces (interval, idle/active state) and Ticket 4's "poll speed" logging must stay consistent since they share one function - see Ticket 4's commit for how that was reconciled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Investigated the actual queries duplicate_prevention runs against SQLite
before touching any schema. The only lookup it performs is the uniqueness
check on (contract_id, ledger_sequence, event_type), enforced implicitly
by INSERT OR IGNORE against UNIQUE(contract_id, ledger_sequence,
event_type) (db.ts MIGRATIONS v1). SQLite auto-creates an index for every
UNIQUE constraint (sqlite_autoindex_events_1), covering exactly that
composite key already - confirmed via EXPLAIN QUERY PLAN both from a
standalone script and now in the committed test suite, e.g.:
SEARCH events USING COVERING INDEX sqlite_autoindex_events_1
(contract_id=? AND ledger_sequence=? AND event_type=?)
getEventsByContract's `WHERE contract_id = ?` also already benefits from
it (contract_id is the index's leftmost column).
No new index was added - per the ticket's own caution against speculative
indexing, and because one already exists. Preserves the UNIQUE constraint,
insertEvent()/insertEventBatch(), and their INSERT OR IGNORE dedup
behavior exactly as-is; this is a lookup-speed investigation, not a
duplicate-detection semantics change.
Added isDuplicateEvent(contractId, ledgerSequence, eventType) to db.ts:
an explicit, named lookup mirroring the constraint check, so this suite
can EXPLAIN QUERY PLAN and benchmark it directly (500 lookups over 2000
rows in ~5ms) rather than relying on introspecting INSERT's implicit
constraint enforcement, which better-sqlite3/SQLite does not surface
through EXPLAIN QUERY PLAN for VALUES-based inserts.
Human-decision note: webhook-delivery.ts's ledger-range scan on the same
`events` table (`WHERE ledger_sequence >= ? AND <= ? ORDER BY
ledger_sequence`) does a full SCAN + TEMP B-TREE (confirmed via EXPLAIN
QUERY PLAN) and would benefit from an index on ledger_sequence. Left
untouched here - it's webhook-delivery, not duplicate_prevention, and is
out of this ticket's stated scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
event_type_filter's live poll path (pollEvents in poller.ts) always starts from lastLedger+1 and walks forward to the current chain head - there was no way to (re-)import a specific past ledger range. Searched the codebase for an existing backfill/historical-ingestion mechanism to reuse (per ticket guidance) and found none - webhook-delivery.ts's startLedger/endLedger query is a ledger-range read of already-indexed events for webhook fan-out, not an ingestion path, so this is genuinely new, not a duplicate. Adds fetchHistoricalEvents(startLedger, endLedger), reusing the same EVENT_TYPES topic filter and event-row mapping as the live poller (factored out into buildEventFilter()/toEventRow() so both share one implementation instead of two copies). Paginates via the SDK's cursor-based getEvents mode (capped at HISTORICAL_IMPORT_MAX_PAGES, default 50 pages of 100 events) so a range producing more than 100 events isn't silently truncated - the live poller's single-page `limit: 100` has this same latent truncation risk but is unchanged here, out of this ticket's scope. validateHistoricalRange() rejects (does not silently clamp, unlike the page/limit clamping already used for pagination elsewhere in db.ts): start > end, non-positive ledger numbers, an end ledger beyond the current chain head (doesn't exist yet), and a range wider than HISTORICAL_IMPORT_MAX_RANGE_LEDGERS (default 10000) - config follows the existing parseInt(process.env.X || "default") convention. Correctness note tying back to duplicate_prevention (Ticket 1/2): a historical import must never move the live indexer_state pointer - db.ts gains insertHistoricalEventBatch(), which reuses the same INSERT OR IGNORE dedup path as insertEventBatch() but deliberately never calls UPDATE indexer_state. Backfilling old ledgers can't rewind last_ledger_sequence, and importing a range the live poller hasn't reached yet can't cause it to skip ledgers by advancing the pointer past unprocessed events. Verified with a test that seeds the live pointer far ahead of the backfill range and asserts it is unchanged after the import, plus an idempotency test (re-running the same import inserts zero duplicate rows). No HTTP route was added - fetchHistoricalEvents() is exported and directly usable/testable, but wiring it to an endpoint (auth, rate limiting) wasn't specified by the ticket and is flagged as a human decision rather than assumed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…or event_type_filter
Adds a debug-level diagnostic log to pollEvents() reporting poll elapsed
time and event payload sizes, per ticket. Two things needed checking
first, per ticket guidance, before logging anything:
1. What's actually in the payloads. dataJson (built in toEventRow(), also
used by db.ts's getJobsByWallet()) can contain job participant Stellar
addresses (client/freelancer/arbiter) and amounts - identifying enough
that dumping it at debug level as a shortcut would be a real exposure.
So this logs Buffer.byteLength(dataJson) per event (aggregated as
totalPayloadBytes/avgPayloadBytes) - sizes and counts only, never the
raw dataJson content. Verified with a test that plants realistic
addresses/amounts in a payload and asserts none of them appear
anywhere in the log string or metadata.
2. Whether logging unconditionally on every poll is safe. logger.ts
already gates logger.debug() off by default in production
(LOG_LEVEL defaults to "info" under NODE_ENV=production, "debug"
otherwise) - that's the primary, already-existing gate, confirmed with
a real (unmocked) logger.ts test. No log-sampling/rate-limiting
convention existed anywhere in the codebase for a line like this, so a
simple one was added on top: a time-based throttle
(POLL_DIAGNOSTIC_LOG_MIN_INTERVAL_MS, default 5000ms, same
parseInt(process.env.X || "default") convention as POLL_INTERVAL_MS)
independent of the poll interval itself, so a misconfigured very-short
POLL_INTERVAL_MS can't turn this into unconditional hot-path logging.
Validation check ("diagnostic log strings contain elapsed time values")
is satisfied literally - the log MESSAGE string embeds
`elapsedMs=<value>`, not just the metadata object, since a message-only
log viewer needs the number to be visible too.
Assumption flagged: "poll speed" is measured as the full pollEvents()
duration (RPC fetch + batch insert), matching what "poll speed" means to
an operator watching indexer health, not just the RPC round-trip alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lation Ensures event_type_filter DB writes commit atomically and roll back fully on failure, protecting data consistency under load. Fixes #188
…outs Implements retry logic with exponentially increasing delay and a configurable max attempt cap for RPC connection timeout errors in database_writer_pool. Non-timeout errors are not retried. Adds tests validating retry frequency growth up to max attempts. Closes #303
Track migration/bootstrap failures and emit threshold warnings so operators can react when schema operations stall or fail repeatedly. Closes #262 Co-authored-by: Cursor <cursoragent@cursor.com>
fix: add custom rate limit to whitelist update endpoint
Optimize lookup indexes for the tables the indexer_runner execution loop accesses and verify index utilization with EXPLAIN QUERY PLAN. - db.ts: add migration v5 with idx_monitored_contracts_active (backing getActiveContractIds WHERE active = 1, the per-cycle poll lookup) and idx_events_created_at (backing MAX(created_at) status aggregation). Export INDEXER_RUNNER_INDEXES for the EXPLAIN tests. - duplicate-prevention.ts: create idx_sync_ranges_ledgers on (start_ledger, end_ledger) at table creation, backing isLedgerSynced range lookups. Export SYNC_RANGES_INDEXES. - indexer-indexes.test.ts: new suite asserting each plan references the expected index via EXPLAIN QUERY PLAN, plus functional checks. - indexer.test.ts: update expected migration count 4 -> 5.
…sqlite_schema_manager-failures
…ynamic polling
The topic filter was previously inline in poller.ts: a bare EVENT_TYPES array
handed to getEvents, with the accepted-type check implicit in whatever the RPC
chose to return. This promotes it to a real module and builds the four
capabilities the indexer needs around it.
Topic parsing
* EVENT_TYPES becomes the single canonical list; poller.ts now imports it
instead of holding its own copy, so the RPC-side filter and the
client-side check cannot drift apart.
* parseEventTopic / isKnownEventType reject malformed or unrecognised
topics, and filterMatchingEvents splits a batch into matched rows and
rejections with reasons. It never throws, so one bad event cannot abort
a whole poll.
Threshold warning alerts
* recordFilterError escalates from warn to error once
EVENT_FILTER_ERROR_THRESHOLD consecutive failures accumulate, and keeps
firing while the condition persists rather than going quiet after the
first breach.
* checkFilterStall reports an error when no successful pass has happened
inside EVENT_FILTER_STALL_THRESHOLD_MS. A filter that has never
succeeded is not counted as stalled -- there is no baseline yet.
* recordFilterSuccess clears the counter and logs the recovery.
Memory queue locks
* withEventQueueLock serialises callers per event identity, keyed on the
same columns as the UNIQUE(contract_id, ledger_sequence, event_type)
constraint, so two notifications the database would treat as one row are
applied one at a time.
* The chain is built from the previous promise whether or not it settled
successfully, so a failed insert cannot deadlock a key, and the map entry
is dropped once the last waiter drains.
* ingestRpcEvents filters then inserts under those locks, reporting
inserted / duplicate / rejected counts.
Dynamic polling intervals
* adjustFilterPollInterval widens the delay after consecutive idle cycles
(capped at the maximum) and contracts it under load (floored at the
minimum).
* Load is measured in events that actually passed the filter, not raw RPC
results: a large payload the filter discards is idle from the indexer's
point of view and should back off like any other quiet cycle.
65 new tests. Suite goes from 862 to 927; npm run build is clean. The 3
suites still failing (failover-recovery-backoff-retry,
failover-recovery-poll-diagnostics, sqlite-schema-manager) fail identically
before this change and are untouched by it.
Implements both assigned issues and restores green CI. 1. Zod schema middleware for POST /api/jobs/create-job-draft (#235): Moves the request-shape validation out of routes/jobs.ts into a reusable middleware (src/middleware/create-job-draft-validation.ts) that selects the matching schema for the modern and legacy *Address body variants. Invalid payloads are reported as 400 ValidationError responses with field-level details. Adds direct middleware tests. 2. Exponential backoff retry on the indexer poller (#249): Routes the live poller's getLatestLedger/getEvents through RpcPollerClient so transient RPC failures (timeouts, connection resets, rate limits, 5xx) are retried with a doubling backoff up to maxRetries, then reset on success. Backoff is configurable via INDEXER_RPC_* env vars (documented in .env.example). 3. CI repair: resolves pre-existing failures on main that were leaving the suite red - defines the missing SCHEMA_MANAGER_INDEXES constant (#259), fixes stale migration-version and logger-mock assertions, and imports jest from @jest/globals / drops fake-timer usage in the failover tests. tsc, jest (958 tests), and build all pass.
- Import jest from @jest/globals in failover-recovery tests that used the global (ReferenceError: jest is not defined) - Drop fake-timer usage that captured the mocked setTimeout and hung - Type logger.warn/debug spy calls to satisfy strict tsc - Export SCHEMA_MANAGER_INDEXES from db.ts and import it in the schema manager test (ReferenceError) - Make the #186 migration-count assertion robust to future migrations - Accept opts argument in the indexer-runner-historical-sync getEvents mock
Implement dynamic polling frequency intervals in indexer_runner based on ledger processing load: - Add IndexerRunnerThrottleParameters / IndexerRunnerThrottleState with env-configured base/min/max intervals, idle multiplier, idle threshold and load decrease factor. - adjustIndexerRunnerPollInterval(): the polling wait delay backs off (increases) toward the max when the network is idle and is pulled back toward the min when events are processed. - Wire the poll loop (poller.ts) to size its sleep from the indexer_runner throttle instead of the generic db.ts throttle. Add tests asserting polling wait delays increase while the network is idle, including an integration test that drives pollEvents() through repeated idle cycles.
- Import jest from @jest/globals in failover-recovery tests that used the global (ReferenceError: jest is not defined) - Drop fake-timer usage that captured the mocked setTimeout and hung - Type logger.warn/debug spy calls to satisfy strict tsc - Export SCHEMA_MANAGER_INDEXES from db.ts and import it in the schema manager test (ReferenceError) - Make the #186 migration-count assertion robust to future migrations - Accept opts argument in the indexer-runner-historical-sync getEvents mock
…341) Implement dynamic polling frequency intervals in indexer_metrics_collector based on ledger processing loads: - Add IndexerMetricsThrottleParameters / IndexerMetricsThrottleState backed by INDEXER_METRICS_POLL_* env configuration. - adjustIndexerMetricsPollingInterval(): the collection poll wait backs off (increases) toward the max interval while the network is idle and resets to the minimum when new events are observed. - computeIndexerMetricsProcessedCount() maps two metrics snapshots to the number of events processed (totalEvents delta). - onIndexerMetricsCollected() ties collections to the throttle so an idle network grows the wait delay automatically. Add tests asserting collection poll wait delays increase while the network is idle, including a flow that drives real collections through idle periods.
- Import jest from @jest/globals in failover-recovery tests that used the global (ReferenceError: jest is not defined) - Drop fake-timer usage that captured the mocked setTimeout and hung - Type logger.warn/debug spy calls to satisfy strict tsc - Export SCHEMA_MANAGER_INDEXES from db.ts and import it in the schema manager test (ReferenceError) - Make the #186 migration-count assertion robust to future migrations - Accept opts argument in the indexer-runner-historical-sync getEvents mock
Optimize the events table index schema used by sqlite_vacuum_cleaner so its row lookups resolve through indexes instead of full scans: - Add migration 6 creating the vacuum cleaner lookup indexes (idx_events_created_at, idx_events_ledger_sequence, and the composite idx_events_created_at_ledger / idx_events_ledger_created_at). - Export VACUUM_CLEANER_INDEXES + getVacuumIndexNames() so tests and operators can reference the managed index names. - Add ensureVacuumIndexes() for self-healing databases that predate migration 6. - Add vacuumExplainQueryPlan() / vacuumQueryPlanUsesIndex() helpers mirroring ledger_range_tracker. Tests run EXPLAIN QUERY PLAN against the cleaner's retention-time and ledger-range predicates and assert the managed indexes are utilized for the lookups. Also make the indexer migration-count assertion robust to future migrations.
- Import jest from @jest/globals in failover-recovery tests that used the global (ReferenceError: jest is not defined) - Drop fake-timer usage that captured the mocked setTimeout and hung - Type logger.warn/debug spy calls to satisfy strict tsc - Export SCHEMA_MANAGER_INDEXES from db.ts and import it in the schema manager test (ReferenceError) - Make the #186 migration-count assertion robust to future migrations - Accept opts argument in the indexer-runner-historical-sync getEvents mock
Emit high-frequency polling diagnostics for sqlite_vacuum_cleaner, mirroring indexer_runner and indexer_metrics_collector so operators can spot slow cleanup runs without enabling a profiler: - Add VacuumPollDiagnostics + logVacuumPollDiagnostics() debug logs whose messages always carry elapsedMs= (plus prunedEvents/retentionDays/ledger range context when applicable). - Time pruneOldEvents, runVacuum, pruneEventsInLedgerRange, and the whole runVacuumCleanup cycle (started/success/failure boundaries). Add tests asserting the cleanup stages and boundaries emit timing diagnostics and that failures are reported with elapsed time before propagating.
…tions. Serialize identical event identities with in-memory queue locks while leaving unrelated events free to proceed, and always release those locks on success or failure. Co-authored-by: Cursor <cursoragent@cursor.com>
…talls Emit threshold warnings after a configured run of failed writes, report stalled operations once, and reset tracking after a successful write. Co-authored-by: Cursor <cursoragent@cursor.com>
Index webhook URL lookups used by the writer pool and assert EXPLAIN QUERY PLAN uses the intended indexes without adding redundant unique-key B-trees. Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # __tests__/database-writer-pool.test.ts # src/indexer/database-writer-pool.ts
# Conflicts: # src/indexer/poller.ts
# Conflicts: # __tests__/whitelist-update.test.ts # src/routes/jobs.ts
# Conflicts: # __tests__/indexer-runner-historical-sync.test.ts # __tests__/sqlite-schema-manager.test.ts
# Conflicts: # __tests__/sqlite-schema-manager.test.ts
# Conflicts: # __tests__/failover-recovery-backoff-retry.test.ts # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/poller.ts # src/routes/jobs.ts
# Conflicts: # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/db.ts
# Conflicts: # __tests__/event_type_filter.test.ts # src/indexer/event_type_filter.ts # src/indexer/poller.ts
# Conflicts: # __tests__/failover-recovery-backoff-retry.test.ts # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/indexer-runner-historical-sync.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/indexer_runner.ts # src/indexer/poller.ts
# Conflicts: # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/indexer_metrics_collector.ts
# Conflicts: # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/indexer.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/db.ts
# Conflicts: # __tests__/failover-recovery-poll-diagnostics.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/sqlite_vacuum_cleaner.ts
# Conflicts: # __tests__/sqlite-schema-manager.test.ts
# Conflicts: # __tests__/sqlite-schema-manager.test.ts # __tests__/sqlite_vacuum_cleaner.test.ts # src/indexer/sqlite_vacuum_cleaner.ts
# Conflicts: # src/routes/jobs.ts
# Conflicts: # __tests__/build-tx.test.ts # src/middleware/job-contract-security.ts # src/routes/jobs.ts
# Conflicts: # src/indexer/database-writer-pool.ts
# Conflicts: # src/indexer/database-writer-pool.ts
# Conflicts: # __tests__/failover-recovery-backoff-retry.test.ts # __tests__/indexer-runner-historical-sync.test.ts # __tests__/indexer.test.ts # __tests__/sqlite-schema-manager.test.ts # src/indexer/db.ts # src/indexer/indexer_metrics_collector.ts
# Conflicts: # src/indexer/indexer_metrics_collector.ts
# Conflicts: # __tests__/sqlite_vacuum_cleaner.test.ts # src/indexer/sqlite_vacuum_cleaner.ts
# Conflicts: # src/indexer/indexer_metrics_collector.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the 25 remaining open PRs. Each is brought in as a real merge commit whose second parent is that PR's head, so every PR is credited to its author and closes as Merged.
PRs included
#307, #311, #316, #317, #321, #322, #359, #370, #371, #372, #373, #375, #376, #377, #378, #379, #380, #384, #385, #388, #389, #390, #392, #393, #394
Conflict resolutions worth knowing
Every conflict was resolved to preserve content — a stale branch's deletion never wins over code already on
main. That rule matters here: this repo lost ~1,300 lines to exactly that failure mode in PR #325.Notable calls:
buildTxCorsand main'supdateWhitelistCorsare different routes; both kept).fetchEventsWithRetry.RpcPollerClientrework, which collided with feat(indexer): exponential backoff retry on event_type_filter RPC calls #307. Restored the deletedfailover-recoverytest files.Also repairs three merge-spliced syntax breaks that were masking type errors: a lost
db.transaction()wrapper induplicate-prevention.ts, two spliced function bodies insqlite_vacuum_cleaner.ts, and an unterminated doc comment inevent_type_filter.tsthat was swallowing its imports.CI status — known red
Merged at the author's direction with CI failing, to be fixed in a follow-up.
tscreports 97 errors, mostly from PRs built on divergent architectures:poller.tsreferences bothserverandrpcClientdb.tshas duplicateSCHEMA_MANAGER_INDEXES/insertHistoricalEventBatchdeclarationsjobs.tsreferences an undefinedcacheKey🤖 Generated with Claude Code