feat: grafema self-analyze at scale — consolidated (derive fixes + RPC + estimator + perf + CI + skill)#475
Open
Disentinel wants to merge 12 commits into
Open
feat: grafema self-analyze at scale — consolidated (derive fixes + RPC + estimator + perf + CI + skill)#475Disentinel wants to merge 12 commits into
Disentinel wants to merge 12 commits into
Conversation
Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three different ways; all three were conservative limits / a concurrency bug that only bite at self-scale. With these, the full derive phase completes (40 packs, 0 SIGSEGV, 0 plan rejects). 1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive EvalLimits::default() 100k max_intermediate_results. The deadline was already lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch derive of @stdlib/js_local_refs overflows it. Lift it for the batch path (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE). 2. compaction vs parallel-derive heap corruption (the hard one): the materialize handler holds a db.engine.read() lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments WHILE the same materialize's par_join_rows rayon tasks read them -> use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named join_derived/par_join_rows; rayon=1 passes; disabling compaction passes). Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit Compact) under the exclusive write lock. Correct phase serialization, not masking. 3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in js_property_access_full estimates 11.7M facts > the 10M guard and is rejected — but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M). Deeper fix = selectivity-aware estimation (follow-up). Self-analyze env for a full run: RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000 Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the estimator q-error; full exit-0 + mcp:tool=27 verification in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The RFDB client guarded every request with a single hardcoded 60s ceiling.
On the ~714k-node self graph that is too tight for two operation classes,
which made the JS enrichers (enrichMcpToolDefinitions, contract, behavior,
package) silently fail after the derive phase:
- bulk writes — addEdges/addNodes/commitBatch do server-side index work
that can exceed 60s -> "RFDB addEdges timed out after 60000ms"
- streaming reads — queryNodesStream waits for the FIRST chunk; the initial
scan can exceed 60s -> "queryNodesStream timed out after 60000ms"
- durable drops — clear/dropDatabase truncate a large on-disk .rfdb >60s
New ts/timeout-config.ts resolves a tiered, env-configurable ceiling:
- RFDB_RPC_TIMEOUT_MS interactive default, now 300000 (5 min)
- RFDB_RPC_BULK_TIMEOUT_MS bulk/streaming/drop, default 3x = 900000 (15 min)
Bulk commands (addEdges/addNodes/commitBatch/clear/dropDatabase/compact/flush/
rebuildIndexes/getAll*/getEdgesByType) and the streaming first-chunk timer use
the longer ceiling; everything else uses the interactive default. Callers may
still pass an explicit per-call timeout. Ceilings stay FINITE so a genuinely
wedged server still fails loudly in bounded time.
Wired into all three transports (client.ts unix socket, websocket-client.ts,
base-client.ts comment). Added timeout-config.test.ts (6 cases); all 208
existing rfdb client tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grafema CI runs CodeQL "Analyze" but never runs *grafema's own* analyze
against this repo — the gap that let a derive-phase SIGSEGV (compaction vs.
parallel-derive heap corruption) and several conservative planner/eval caps
ship silently, since they only bite at self-scale (~714k nodes / 1.33M edges)
and are unreachable by the fixture-sized unit/integration tests.
New .github/workflows/self-analyze.yml:
- builds grafema as it ships (pnpm build + cargo --release for rfdb-server
and grafema-orchestrator, which the analyze pipeline spawns),
- runs `grafema analyze --clear` on the full monorepo via the checked-in
dogfooding config with the self-analyze env from fix/derive-batch-cap
(RFDB_MATERIALIZE_DEADLINE_SECS=3600, RFDB_MAX_MATERIALIZED_FACTS=200000000),
- asserts exit 0 AND that the JS enrichers fired (graph has mcp:tool nodes
from the behavior enricher — a clean exit with an empty graph is caught),
- canaries the derive log for SIGSEGV / plan rejects.
Runner: ubuntu-latest (16 GB RAM / 4 vCPU). Full self-analyze peaks ~7.5 GB /
~14 min, which fits with headroom; the full run (not a scoped subset) is kept
so all ~40 derive packs — incl. the ones that SIGSEGV'd / hit caps — are
exercised.
Non-blocking on purpose (continue-on-error: true, not a required check): a
full self-analyze still fails late on an enricher RPC 60s timeout (fixed
separately on fix/enricher-rpc-timeout). The job RUNS and surfaces the result
without blocking PRs. TODO(make-required) in the header: flip off
continue-on-error + add to branch-protection once self-analyze is green E2E.
Validated with actionlint v1.7.12 (exit 0, no warnings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…equality joins The planner's per-rule output estimate (E-PLAN-003 guard) cross-producted every generator's RAW relation magnitude along the join, ignoring that a later leg applying an EQUALITY selection on a generator-introduced key against an already-bound value makes that join nearly 1:1. The stdlib `static_member` rule (js_property_access_full) — uc_pa · node(Cls,"CLASS") · attr(Cls,"file",F) · attr(Cls,"name",BaseN) · edge(Cls,T,"HAS_METHOD") · attr(T,"name",N) — was sized at ~11.7M (|uc_pa|·|CLASS|·avg_degree) and spuriously rejected, vs an ACTUAL 811 materialized edges (~14000x over-estimate). The branch worked around it with the RFDB_MAX_MATERIALIZED_FACTS env override; this is the principled fix. When a generator-introduced variable is EQUALITY-PINNED by a downstream base-relation selection whose other positions are all already-bound (a shared-variable join key — attr(v,key,BoundVal) / attr(BoundId,key,v) / edge(v,BoundEndpoint,_)), the generator is in truth a keyed probe: reduce its multiplicative contribution by the existing key-narrowing (√) factor instead of multiplying the full relation magnitude. A key with NO downstream equality is unreduced, so a GENUINE cross-product (whose legs share no pinning equality) keeps its full magnitude and still trips the guard — the guard's job is intact. Sizing/ordering can only let more rules pass, never change which facts derive (I1). Tests: static_member-shape selective join now plans under the 10M default guard; a genuine two-hop explosion still trips E-PLAN-003; direct discrimination unit on the pin detector (pins a bound-value equality, not a type-restriction / free- endpoint hop / free-value read). All 13 plan tests + 323 derive tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0s/file timeout) Profiling grafema's own self-analyze (513k nodes / 1.1M edges) showed the analysis phase (241.5s) dominated by 25 BEAM `.ex` files at ~120s EACH — 3003s of file-work parallelized to wall — vs ~69ms/TS, 86ms/Haskell, 142ms/Rust. Root cause: `beam-analyzer` is an Erlang escript. Under a non-UTF-8 host locale the Erlang VM uses `latin1` native name encoding, which corrupts the length-prefixed binary IPC framing the orchestrator ProcessPool speaks. The `--daemon` never replies, so every per-file request burns the full 120s DEFAULT_REQUEST_TIMEOUT before the pool abandons it (the "Pool request failed" errors). The same `.ex` file in single-shot mode parses in 309ms — the cost is the daemon-encoding hang, not BEAM parsing. Fix: new `PoolConfig.extra_env`, pinned with `ELIXIR_ERL_OPTIONS=+fnu` (+ `LANG`/`LC_ALL=C.UTF-8`) on the BEAM analyzer and resolve pools. Forces UTF-8 regardless of host locale; zero-risk for non-BEAM analyzers (env applied only to the BEAM pools). Measured after the fix: 26 BEAM files analyzed, 0 pool failures, 0 latin1 warnings (was 25 failures). Full profile + a deferred derive-phase optimization plan in `_ai/self-analyze-perf-findings.md`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…resolve daemon After N consecutive request timeouts with no intervening success (TIMEOUT_CIRCUIT_BREAKER_THRESHOLD=3), a ProcessPool fails every subsequent request() IMMEDIATELY instead of paying the full request_timeout (120s) again. A single success resets the counter, so a merely-slow daemon never trips it. Motivation: a hung analyzer/resolve daemon otherwise costs request_timeout PER request. This bounds the damage to N×timeout on any SERIAL daemon request stream — the resolve daemons (JS per-file resolve-file, single-worker Haskell/Rust/Java/BEAM resolve pools), i.e. the REG-1128 N+1 hot path. Tested: test_circuit_breaker_fails_fast_on_dead_daemon — after the threshold, a request against a hung daemon returns in <request_timeout (circuit-open). Measured limitation (documented in _ai/self-analyze-perf-findings.md): the breaker does NOT help the PARALLEL analysis fan-out, where all files dispatch concurrently and pass the breaker check before the first timeout fires (BEAM: counter reached 17, 0 circuit-open, analysis still 241s). The deferred fix for that path — a sequential liveness probe before fan-out — is specified in the findings doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…7s → 37.3s The parallel analysis fan-out dispatches all files concurrently, so the pool's request-level circuit breaker cannot help BEAM: every request passes the breaker check at t≈0 and they all time out together at the per-request timeout. On grafema's own monorepo (25 `.ex` files), a non-responsive beam-analyzer daemon cost ~241s — the analysis phase's single largest sink. analyze_beam_files_parallel_pooled now probes ONE file sequentially before fanning out. A healthy daemon answers in <1s and the normal parallel path runs (file 0 re-analyzed by the fan-out — one negligible sub-second analysis; the fan-out loop is left untouched, lowest-risk). A dead daemon fails the probe in one request_timeout and ALL files are returned failed-fast instead of each paying the timeout. The BEAM pool's request_timeout is also lowered 120s → 30s (a real `.ex` file analyzes in ~300ms). Measured (grafema self-analyze, dead BEAM daemon): analysis phase 241.7s → 37.3s (6.5x, -204s); BEAM file-work 3001s → 0s; BEAM timeouts 25 → 1. On a host where BEAM works the probe passes in <1s and nothing changes. Full before/after in _ai/self-analyze-perf-findings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…by route + bottlenecks Captures the method (measure critical path first, don't guess), the auto-emitted JSONL profiler + critical-path one-liner, the derive/enricher blind spot, the KNOWN artificial limits (so instances stop rediscovering 100k/10M/60s/120s caps from crashes) + their env overrides, host/RAM requirements, and the profile-subgraph vision. Distilled from the June-2026 self-analyze hardening campaign. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…into feat/self-analyze-hardening
…f-analyze-hardening
…elf-analyze-hardening
…feat/self-analyze-hardening
This was referenced Jun 19, 2026
Closed
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+77
to
+177
| name: grafema analyze (self, non-blocking) | ||
| runs-on: ubuntu-latest | ||
| # Non-blocking: surface the result, never block a PR — see header TODO. | ||
| continue-on-error: true | ||
| # Generous: Rust release build (~10-15 min cold) + ~14 min analyze. | ||
| timeout-minutes: 45 | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup pnpm | ||
| uses: pnpm/action-setup@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| cache: 'pnpm' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Cache cargo | ||
| uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| workspaces: | | ||
| packages/rfdb-server -> target | ||
| packages/grafema-orchestrator -> target | ||
| key: self-analyze | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --no-frozen-lockfile | ||
|
|
||
| - name: Build TS packages | ||
| run: pnpm build | ||
|
|
||
| # The analyze pipeline spawns two native binaries; the CLI auto-detects them | ||
| # from packages/<pkg>/target/release/ (findRfdbBinary / findOrchestratorBinary). | ||
| - name: Build rfdb-server (release) | ||
| working-directory: packages/rfdb-server | ||
| run: cargo build --release | ||
|
|
||
| - name: Build grafema-orchestrator (release) | ||
| working-directory: packages/grafema-orchestrator | ||
| run: cargo build --release | ||
|
|
||
| - name: Show free memory before analyze | ||
| run: free -h || true | ||
|
|
||
| # The repo ships .grafema/config.yaml (dogfooding config, full monorepo). | ||
| # --clear rebuilds from scratch; --no-auto-start is NOT used so the CLI | ||
| # brings up the freshly-built rfdb-server itself. | ||
| - name: Run grafema self-analyze | ||
| id: analyze | ||
| run: | | ||
| set -o pipefail | ||
| node packages/cli/dist/cli.js analyze --clear --verbose 2>&1 | tee /tmp/self-analyze.log | ||
| echo "Analyze exited 0." | ||
|
|
||
| # Assert the JS enrichers fired: the behavior enricher emits mcp:tool nodes | ||
| # for packages/mcp/src/server.ts. A clean exit with an empty graph is the | ||
| # silent-failure mode this guard exists to catch. | ||
| - name: Assert JS enrichers fired (mcp:tool nodes present) | ||
| run: | | ||
| set -o pipefail | ||
| node packages/cli/dist/cli.js query --type "mcp:tool" "" --json -l 100000 2>/dev/null > /tmp/mcp-tool.json | ||
| COUNT=$(node -e "const a=JSON.parse(require('fs').readFileSync('/tmp/mcp-tool.json','utf8')); console.log(Array.isArray(a)?a.length:0)") | ||
| echo "mcp:tool node count: $COUNT" | ||
| echo "mcp:tool nodes: $COUNT" >> "$GITHUB_STEP_SUMMARY" | ||
| if [ "$COUNT" -lt 1 ]; then | ||
| echo "::error::self-analyze produced 0 mcp:tool nodes — JS enrichers did not fire (graph is empty/incomplete)." | ||
| exit 1 | ||
| fi | ||
| echo "OK: JS enrichers fired ($COUNT mcp:tool nodes)." | ||
|
|
||
| # Best-effort SIGSEGV / derive-reject canary. The derive phase must finish | ||
| # without a segfault or plan rejects; flag them loudly even though the job | ||
| # is non-blocking for now. | ||
| - name: Check derive phase for SIGSEGV / plan rejects | ||
| if: always() | ||
| run: | | ||
| if grep -qiE "SIGSEGV|signal: 11|segmentation fault" /tmp/self-analyze.log 2>/dev/null; then | ||
| echo "::error::SIGSEGV detected in self-analyze — derive-phase heap corruption regressed." | ||
| grep -iE "SIGSEGV|signal: 11|segmentation fault" /tmp/self-analyze.log | head >> "$GITHUB_STEP_SUMMARY" || true | ||
| exit 1 | ||
| fi | ||
| if grep -qiE "plan reject|rejected by planner|MAX_MATERIALIZED_FACTS guard" /tmp/self-analyze.log 2>/dev/null; then | ||
| echo "::warning::Derive plan rejects present in self-analyze log (planner cap hit)." | ||
| fi | ||
| echo "No SIGSEGV in derive phase." | ||
|
|
||
| - name: Upload self-analyze log | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: self-analyze-log | ||
| path: | | ||
| /tmp/self-analyze.log | ||
| /tmp/mcp-tool.json | ||
| retention-days: 14 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates the June-2026 self-analyze hardening campaign into one branch (supersedes #469-#474; all merged clean, compiles — cargo check rfdb-server + orchestrator exit 0). main could not analyze its own monorepo (~714k nodes); now the full pipeline runs — 40 derive packs, 0 SIGSEGV, enrichers fire.
Rolls up:
max_intermediate_resultsleaked into batch), AutoCompactionSuppressGuard (the parallel-derive heap-corruption SIGSEGV — confirmed via TSan + isolation), plan-guard env override..claude/skills/analyze-perf-profiling(the method + known limits, distributed to instances).Verified end-to-end on grafema-dev (32 GB): rc=0, 513k→… nodes, 0 SIGSEGV, 14:01 wall. Since main was already broken on self-analyze, this is forward-only.
🤖 Generated with Claude Code