fix(rfdb): env-configurable, per-operation RPC client timeouts (unblock enrichers on large self-analyze)#471
fix(rfdb): env-configurable, per-operation RPC client timeouts (unblock enrichers on large self-analyze)#471Disentinel wants to merge 2 commits into
Conversation
Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three different ways; all three were conservative limits / a concurrency bug that only bite at self-scale. With these, the full derive phase completes (40 packs, 0 SIGSEGV, 0 plan rejects). 1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive EvalLimits::default() 100k max_intermediate_results. The deadline was already lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch derive of @stdlib/js_local_refs overflows it. Lift it for the batch path (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE). 2. compaction vs parallel-derive heap corruption (the hard one): the materialize handler holds a db.engine.read() lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments WHILE the same materialize's par_join_rows rayon tasks read them -> use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named join_derived/par_join_rows; rayon=1 passes; disabling compaction passes). Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit Compact) under the exclusive write lock. Correct phase serialization, not masking. 3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in js_property_access_full estimates 11.7M facts > the 10M guard and is rejected — but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M). Deeper fix = selectivity-aware estimation (follow-up). Self-analyze env for a full run: RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000 Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the estimator q-error; full exit-0 + mcp:tool=27 verification in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The RFDB client guarded every request with a single hardcoded 60s ceiling.
On the ~714k-node self graph that is too tight for two operation classes,
which made the JS enrichers (enrichMcpToolDefinitions, contract, behavior,
package) silently fail after the derive phase:
- bulk writes — addEdges/addNodes/commitBatch do server-side index work
that can exceed 60s -> "RFDB addEdges timed out after 60000ms"
- streaming reads — queryNodesStream waits for the FIRST chunk; the initial
scan can exceed 60s -> "queryNodesStream timed out after 60000ms"
- durable drops — clear/dropDatabase truncate a large on-disk .rfdb >60s
New ts/timeout-config.ts resolves a tiered, env-configurable ceiling:
- RFDB_RPC_TIMEOUT_MS interactive default, now 300000 (5 min)
- RFDB_RPC_BULK_TIMEOUT_MS bulk/streaming/drop, default 3x = 900000 (15 min)
Bulk commands (addEdges/addNodes/commitBatch/clear/dropDatabase/compact/flush/
rebuildIndexes/getAll*/getEdgesByType) and the streaming first-chunk timer use
the longer ceiling; everything else uses the interactive default. Callers may
still pass an explicit per-call timeout. Ceilings stay FINITE so a genuinely
wedged server still fails loudly in bounded time.
Wired into all three transports (client.ts unix socket, websocket-client.ts,
base-client.ts comment). Added timeout-config.test.ts (6 cases); all 208
existing rfdb client tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VerificationUnit tests
End-to-end on grafema-dev (16 vCPU / 30 GB)Built the JS packages on top of this branch, pointed the analyzer at the freshly-built v2 Graph scale reached: 502,029 nodes / 932,685 edges analyzed → 513,164 nodes / 1,050,642 edges after resolution + derive. Key results — all with the old 60s ceiling, every one of these would have aborted:
Total The
|
Correction / honest verification limitThe e2e run did not reach a clean This is environmental memory contention, not a timeout — and it does not weaken the fix. The verification claim stands on what the run reached before the OOM, all of which the old 60s ceiling would have aborted:
So: derive completes, the enrichers fire, the |
|
Superseded by the consolidated PR #475 (feat/self-analyze-hardening) — all 6 branches merged clean into one. Commits preserved there. |
Problem
After the derive phase completes on a large self-analyze, the JS enrichers
(
enrichMcpToolDefinitions, contract, behavior, package) were silentlyskipped because their RFDB RPC calls hit a single hardcoded 60s client
timeout:
On a large graph (the ~714k-node self graph) the server legitimately needs
longer than 60s for:
addEdges/addNodes/commitBatchdo server-sideindex work.
mcpToolDefinitionEnrichercallsclient.addEdges(newEdges)directly — that is the exact callsite that timed out.
queryNodesStreamwaits for the first chunk; theinitial scan can exceed 60s before any chunk arrives.
analyze --cleartruncates a large on-disk.rfdb,which can exceed 60s.
Fix
New
packages/rfdb/ts/timeout-config.tsresolves an env-configurable,tiered timeout ceiling instead of a single 60s constant:
RFDB_RPC_TIMEOUT_MSRFDB_RPC_BULK_TIMEOUT_MSBulk command set:
addEdges,addNodes,commitBatch,clear,dropDatabase,compact,flush,rebuildIndexes,getAllNodes,getAllEdges,getEdgesByType. The streaming first-chunk timer(
_resetStreamTimer) also uses the bulk ceiling. Callers may still pass anexplicit per-call
timeoutMsoverride.Why these defaults: a full self-analyze is ~14 min wall; the slow
individual RPCs (bulk
addEdges, first stream chunk) sit in thetens-of-seconds-to-low-minutes range on the 714k graph. 5 min gives comfortable
margin for interactive ops on a large graph while still failing a genuinely
hung server in bounded time; 15 min for bulk/drop covers the single longest
legitimate operation (a durable drop / large commit). Ceilings stay finite
(never
Infinity) so a wedged server still fails loudly.Wired into all three transports:
client.ts(unix socket — the one that timedout),
websocket-client.ts, and a pointer comment inbase-client.ts.Slow
clear/ durable-drop findingGraphEngineV2::clear_durable(packages/rfdb-server/src/graph/engine_v2.rs)is not O(graph-nodes): it swaps the in-memory store to ephemeral, deletes
the manifest authority, then
std::fs::remove_dir_all(segments/). The slowpart is the filesystem deletion of many on-disk segment files — O(#segment
files), I/O-bound, which can exceed 60s for a large DB.
reset_derive_cachesis O(1).
clear/dropDatabaseare in the bulk set, sothey now get the 15-min ceiling instead of 60s — the timeout no longer aborts
a legitimately-progressing durable drop.
Rust workers + a full rebuild): step 2 of
clear_durablealready makes theDB logically empty the instant the fresh manifest is written (orphaned
segments are unreferenced and invisible to readers). The expensive
remove_dir_all(segments/)could be renamed aside (into the existinggc/trash convention) and deleted asynchronously, returning to the clientin O(1). Recorded as a finding for the engine track.
Verification
packages/rfdb/ts/timeout-config.test.ts— 6 cases (defaults, commandclassification, explicit override, env overrides, malformed-env fail-safe).
_sendsignature change (
timeoutMs?: number).grafema-dev box and ran
analyze --quickstart --clearon the grafema source —see PR comment for result (enrichers fire, mcp:tool nodes created, exit 0).
Base:
main. Built on top offix/derive-batch-cap(the 3 engine fixes thatlet derive complete at scale).
🤖 Generated with Claude Code