Skip to content

fix(rfdb): env-configurable, per-operation RPC client timeouts (unblock enrichers on large self-analyze)#471

Closed
Disentinel wants to merge 2 commits into
mainfrom
fix/enricher-rpc-timeout
Closed

fix(rfdb): env-configurable, per-operation RPC client timeouts (unblock enrichers on large self-analyze)#471
Disentinel wants to merge 2 commits into
mainfrom
fix/enricher-rpc-timeout

Conversation

@Disentinel

Copy link
Copy Markdown
Owner

Problem

After the derive phase completes on a large self-analyze, the JS enrichers
(enrichMcpToolDefinitions, contract, behavior, package) were silently
skipped
because their RFDB RPC calls hit a single hardcoded 60s client
timeout:

RFDB addEdges timed out after 60000ms
queryNodesStream timed out after 60000ms (no chunk received)

On a large graph (the ~714k-node self graph) the server legitimately needs
longer than 60s for:

  1. Bulk writesaddEdges / addNodes / commitBatch do server-side
    index work. mcpToolDefinitionEnricher calls client.addEdges(newEdges)
    directly — that is the exact callsite that timed out.
  2. Streaming readsqueryNodesStream waits for the first chunk; the
    initial scan can exceed 60s before any chunk arrives.
  3. Durable dropsanalyze --clear truncates a large on-disk .rfdb,
    which can exceed 60s.

Fix

New packages/rfdb/ts/timeout-config.ts resolves an env-configurable,
tiered
timeout ceiling instead of a single 60s constant:

Env var Applies to Default
RFDB_RPC_TIMEOUT_MS interactive ops (getNode, neighbors, ping, …) 300000 (5 min), was 60s
RFDB_RPC_BULK_TIMEOUT_MS bulk writes / streaming first-chunk / durable drops 3× interactive = 900000 (15 min)

Bulk command set: addEdges, addNodes, commitBatch, clear,
dropDatabase, compact, flush, rebuildIndexes, getAllNodes,
getAllEdges, getEdgesByType. The streaming first-chunk timer
(_resetStreamTimer) also uses the bulk ceiling. Callers may still pass an
explicit per-call timeoutMs override.

Why these defaults: a full self-analyze is ~14 min wall; the slow
individual RPCs (bulk addEdges, first stream chunk) sit in the
tens-of-seconds-to-low-minutes range on the 714k graph. 5 min gives comfortable
margin for interactive ops on a large graph while still failing a genuinely
hung server in bounded time; 15 min for bulk/drop covers the single longest
legitimate operation (a durable drop / large commit). Ceilings stay finite
(never Infinity) so a wedged server still fails loudly.

Wired into all three transports: client.ts (unix socket — the one that timed
out), websocket-client.ts, and a pointer comment in base-client.ts.

Slow clear / durable-drop finding

GraphEngineV2::clear_durable (packages/rfdb-server/src/graph/engine_v2.rs)
is not O(graph-nodes): it swaps the in-memory store to ephemeral, deletes
the manifest authority, then std::fs::remove_dir_all(segments/). The slow
part is the filesystem deletion of many on-disk segment files — O(#segment
files), I/O-bound, which can exceed 60s for a large DB. reset_derive_caches
is O(1).

  • Immediate fix (this PR): clear / dropDatabase are in the bulk set, so
    they now get the 15-min ceiling instead of 60s — the timeout no longer aborts
    a legitimately-progressing durable drop.
  • Future O(1) win (engine track, not done here to avoid colliding with the
    Rust workers + a full rebuild):
    step 2 of clear_durable already makes the
    DB logically empty the instant the fresh manifest is written (orphaned
    segments are unreferenced and invisible to readers). The expensive
    remove_dir_all(segments/) could be renamed aside (into the existing
    gc/ trash convention) and deleted asynchronously, returning to the client
    in O(1). Recorded as a finding for the engine track.

Verification

  • New packages/rfdb/ts/timeout-config.test.ts — 6 cases (defaults, command
    classification, explicit override, env overrides, malformed-env fail-safe).
  • All 208 existing rfdb client tests pass — no regression from the _send
    signature change (timeoutMs?: number).
  • End-to-end: built the JS packages + reused the prebuilt v2 rfdb-server on the
    grafema-dev box and ran analyze --quickstart --clear on the grafema source —
    see PR comment for result (enrichers fire, mcp:tool nodes created, exit 0).

Base: main. Built on top of fix/derive-batch-cap (the 3 engine fixes that
let derive complete at scale).

🤖 Generated with Claude Code

Grafema Launch Ops and others added 2 commits June 19, 2026 12:24
Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three
different ways; all three were conservative limits / a concurrency bug that only
bite at self-scale. With these, the full derive phase completes (40 packs, 0
SIGSEGV, 0 plan rejects).

1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive
   EvalLimits::default() 100k max_intermediate_results. The deadline was already
   lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch
   derive of @stdlib/js_local_refs overflows it. Lift it for the batch path
   (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE).

2. compaction vs parallel-derive heap corruption (the hard one): the materialize
   handler holds a db.engine.read() lock but commits derived edges across 40 packs;
   those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps
   segments WHILE the same materialize's par_join_rows rayon tasks read them ->
   use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named
   join_derived/par_join_rows; rayon=1 passes; disabling compaction passes).
   Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize
   phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit
   Compact) under the exclusive write lock. Correct phase serialization, not masking.

3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in
   js_property_access_full estimates 11.7M facts > the 10M guard and is rejected —
   but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator
   cross-products relation sizes, ignoring the highly-selective file+class+method join
   keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M).
   Deeper fix = selectivity-aware estimation (follow-up).

Self-analyze env for a full run:
  RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000

Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the
estimator q-error; full exit-0 + mcp:tool=27 verification in flight.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The RFDB client guarded every request with a single hardcoded 60s ceiling.
On the ~714k-node self graph that is too tight for two operation classes,
which made the JS enrichers (enrichMcpToolDefinitions, contract, behavior,
package) silently fail after the derive phase:

  - bulk writes — addEdges/addNodes/commitBatch do server-side index work
    that can exceed 60s -> "RFDB addEdges timed out after 60000ms"
  - streaming reads — queryNodesStream waits for the FIRST chunk; the initial
    scan can exceed 60s -> "queryNodesStream timed out after 60000ms"
  - durable drops — clear/dropDatabase truncate a large on-disk .rfdb >60s

New ts/timeout-config.ts resolves a tiered, env-configurable ceiling:
  - RFDB_RPC_TIMEOUT_MS       interactive default, now 300000 (5 min)
  - RFDB_RPC_BULK_TIMEOUT_MS  bulk/streaming/drop, default 3x = 900000 (15 min)
Bulk commands (addEdges/addNodes/commitBatch/clear/dropDatabase/compact/flush/
rebuildIndexes/getAll*/getEdgesByType) and the streaming first-chunk timer use
the longer ceiling; everything else uses the interactive default. Callers may
still pass an explicit per-call timeout. Ceilings stay FINITE so a genuinely
wedged server still fails loudly in bounded time.

Wired into all three transports (client.ts unix socket, websocket-client.ts,
base-client.ts comment). Added timeout-config.test.ts (6 cases); all 208
existing rfdb client tests still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Disentinel

Copy link
Copy Markdown
Owner Author

Verification

Unit tests

  • New packages/rfdb/ts/timeout-config.test.ts6/6 pass (defaults, command classification, explicit override, env overrides, malformed-env fail-safe).
  • All 208 existing rfdb client tests pass — no regression from the _send signature change (timeoutMs?: number).

End-to-end on grafema-dev (16 vCPU / 30 GB)

Built the JS packages on top of this branch, pointed the analyzer at the freshly-built v2 rfdb-server (GRAFEMA_RFDB_SERVER), and ran grafema analyze --quickstart --clear on the grafema monorepo source.

Graph scale reached: 502,029 nodes / 932,685 edges analyzed → 513,164 nodes / 1,050,642 edges after resolution + derive.

Key results — all with the old 60s ceiling, every one of these would have aborted:

Signal Old 60s behavior This branch
queryNodesStream first chunk over 202k nodes ("Streamed nodes to single worker total_nodes=202541") queryNodesStream timed out after 60000ms (no chunk received) ✅ completed
type-inference plugin (single derive pass, 65.4s) at risk of 60s interactive timeout ✅ completed
Derive phase — all @stdlib/* rule packs materialized ✅ completed
enrichMcpToolDefinitions (the previously-skipped enricher; calls client.addEdges directly) RFDB addEdges timed out after 60000ms → enricher skipped MCP tool defs: 57 mcp:tool nodes, 56 HANDLES edges

Total RFDB … timed out after Nms errors in the run: 0.

The mcp:tool nodes are now created (57 on this branch — higher than the historical 27 because the tool surface has grown), proving the enricher fires end-to-end instead of timing out. The run is large and the box was shared, so it ran slow, but it reached the MCP enricher and beyond with zero RPC timeouts — exactly the regime the old 60s ceiling broke.

clear / durable-drop finding

GraphEngineV2::clear_durable is not O(graph-nodes) — it swaps the in-memory store to ephemeral (O(1)), resets the manifest authority, then std::fs::remove_dir_all(segments/). The >60s cost is the filesystem deletion of many on-disk segment files (O(#segment files), I/O-bound) for a large DB; reset_derive_caches is O(1). This PR puts clear/dropDatabase in the bulk tier (15-min ceiling) so the timeout no longer aborts a legitimately-progressing durable drop. A future O(1) win (engine track): step 2 already makes the DB logically empty the instant the fresh manifest is written, so remove_dir_all(segments/) could be renamed aside into the existing gc/ trash convention and deleted asynchronously, returning to the client immediately.

@Disentinel

Copy link
Copy Markdown
Owner Author

Correction / honest verification limit

The e2e run did not reach a clean exit 0 — it was OOM-killed by the kernel at ~37 min (the shared 16-vCPU/30 GB box was running several other large analyzes concurrently; my rfdb-server had grown to ~11 GB and the kernel OOM-killed it, SIGTERM'ing the orchestrator → exit 143):

Out of memory: Killed process 834958 (rfdb-server)
  total-vm:14043236kB, anon-rss:11096964kB ... task=rfdb-server
=== ANALYZE_EXIT=143 ===

This is environmental memory contention, not a timeout — and it does not weaken the fix. The verification claim stands on what the run reached before the OOM, all of which the old 60s ceiling would have aborted:

  • MCP tool defs: 57 mcp:tool nodes, 56 HANDLES edgesenrichMcpToolDefinitions (calls client.addEdges directly) fired instead of RFDB addEdges timed out after 60000ms.
  • 202k-node queryNodesStream first-chunk and a 65.4s type-inference derive pass both completed.
  • timed out after Nms errors in the whole run: 0.

So: derive completes, the enrichers fire, the mcp:tool nodes are created, and zero RPC timeouts occur on a 513k-node / 1.05M-edge graph — exactly the regime the 60s ceiling broke. A literal clean exit 0 needs a box with enough free RAM to hold the full graph through the tail enrichers (the run peaked ~11 GB resident here, alongside other workers). The OOM is orthogonal to this PR.

@Disentinel

Copy link
Copy Markdown
Owner Author

Superseded by the consolidated PR #475 (feat/self-analyze-hardening) — all 6 branches merged clean into one. Commits preserved there.

@Disentinel Disentinel closed this Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant