ci: non-blocking grafema self-analyze regression guard#470
Closed
Disentinel wants to merge 2 commits into
Closed
Conversation
Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three different ways; all three were conservative limits / a concurrency bug that only bite at self-scale. With these, the full derive phase completes (40 packs, 0 SIGSEGV, 0 plan rejects). 1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive EvalLimits::default() 100k max_intermediate_results. The deadline was already lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch derive of @stdlib/js_local_refs overflows it. Lift it for the batch path (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE). 2. compaction vs parallel-derive heap corruption (the hard one): the materialize handler holds a db.engine.read() lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments WHILE the same materialize's par_join_rows rayon tasks read them -> use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named join_derived/par_join_rows; rayon=1 passes; disabling compaction passes). Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit Compact) under the exclusive write lock. Correct phase serialization, not masking. 3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in js_property_access_full estimates 11.7M facts > the 10M guard and is rejected — but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M). Deeper fix = selectivity-aware estimation (follow-up). Self-analyze env for a full run: RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000 Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the estimator q-error; full exit-0 + mcp:tool=27 verification in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grafema CI runs CodeQL "Analyze" but never runs *grafema's own* analyze
against this repo — the gap that let a derive-phase SIGSEGV (compaction vs.
parallel-derive heap corruption) and several conservative planner/eval caps
ship silently, since they only bite at self-scale (~714k nodes / 1.33M edges)
and are unreachable by the fixture-sized unit/integration tests.
New .github/workflows/self-analyze.yml:
- builds grafema as it ships (pnpm build + cargo --release for rfdb-server
and grafema-orchestrator, which the analyze pipeline spawns),
- runs `grafema analyze --clear` on the full monorepo via the checked-in
dogfooding config with the self-analyze env from fix/derive-batch-cap
(RFDB_MATERIALIZE_DEADLINE_SECS=3600, RFDB_MAX_MATERIALIZED_FACTS=200000000),
- asserts exit 0 AND that the JS enrichers fired (graph has mcp:tool nodes
from the behavior enricher — a clean exit with an empty graph is caught),
- canaries the derive log for SIGSEGV / plan rejects.
Runner: ubuntu-latest (16 GB RAM / 4 vCPU). Full self-analyze peaks ~7.5 GB /
~14 min, which fits with headroom; the full run (not a scoped subset) is kept
so all ~40 derive packs — incl. the ones that SIGSEGV'd / hit caps — are
exercised.
Non-blocking on purpose (continue-on-error: true, not a required check): a
full self-analyze still fails late on an enricher RPC 60s timeout (fixed
separately on fix/enricher-rpc-timeout). The job RUNS and surfaces the result
without blocking PRs. TODO(make-required) in the header: flip off
continue-on-error + add to branch-protection once self-analyze is green E2E.
Validated with actionlint v1.7.12 (exit 0, no warnings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+77
to
+177
| name: grafema analyze (self, non-blocking) | ||
| runs-on: ubuntu-latest | ||
| # Non-blocking: surface the result, never block a PR — see header TODO. | ||
| continue-on-error: true | ||
| # Generous: Rust release build (~10-15 min cold) + ~14 min analyze. | ||
| timeout-minutes: 45 | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup pnpm | ||
| uses: pnpm/action-setup@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| cache: 'pnpm' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Cache cargo | ||
| uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| workspaces: | | ||
| packages/rfdb-server -> target | ||
| packages/grafema-orchestrator -> target | ||
| key: self-analyze | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --no-frozen-lockfile | ||
|
|
||
| - name: Build TS packages | ||
| run: pnpm build | ||
|
|
||
| # The analyze pipeline spawns two native binaries; the CLI auto-detects them | ||
| # from packages/<pkg>/target/release/ (findRfdbBinary / findOrchestratorBinary). | ||
| - name: Build rfdb-server (release) | ||
| working-directory: packages/rfdb-server | ||
| run: cargo build --release | ||
|
|
||
| - name: Build grafema-orchestrator (release) | ||
| working-directory: packages/grafema-orchestrator | ||
| run: cargo build --release | ||
|
|
||
| - name: Show free memory before analyze | ||
| run: free -h || true | ||
|
|
||
| # The repo ships .grafema/config.yaml (dogfooding config, full monorepo). | ||
| # --clear rebuilds from scratch; --no-auto-start is NOT used so the CLI | ||
| # brings up the freshly-built rfdb-server itself. | ||
| - name: Run grafema self-analyze | ||
| id: analyze | ||
| run: | | ||
| set -o pipefail | ||
| node packages/cli/dist/cli.js analyze --clear --verbose 2>&1 | tee /tmp/self-analyze.log | ||
| echo "Analyze exited 0." | ||
|
|
||
| # Assert the JS enrichers fired: the behavior enricher emits mcp:tool nodes | ||
| # for packages/mcp/src/server.ts. A clean exit with an empty graph is the | ||
| # silent-failure mode this guard exists to catch. | ||
| - name: Assert JS enrichers fired (mcp:tool nodes present) | ||
| run: | | ||
| set -o pipefail | ||
| node packages/cli/dist/cli.js query --type "mcp:tool" "" --json -l 100000 2>/dev/null > /tmp/mcp-tool.json | ||
| COUNT=$(node -e "const a=JSON.parse(require('fs').readFileSync('/tmp/mcp-tool.json','utf8')); console.log(Array.isArray(a)?a.length:0)") | ||
| echo "mcp:tool node count: $COUNT" | ||
| echo "mcp:tool nodes: $COUNT" >> "$GITHUB_STEP_SUMMARY" | ||
| if [ "$COUNT" -lt 1 ]; then | ||
| echo "::error::self-analyze produced 0 mcp:tool nodes — JS enrichers did not fire (graph is empty/incomplete)." | ||
| exit 1 | ||
| fi | ||
| echo "OK: JS enrichers fired ($COUNT mcp:tool nodes)." | ||
|
|
||
| # Best-effort SIGSEGV / derive-reject canary. The derive phase must finish | ||
| # without a segfault or plan rejects; flag them loudly even though the job | ||
| # is non-blocking for now. | ||
| - name: Check derive phase for SIGSEGV / plan rejects | ||
| if: always() | ||
| run: | | ||
| if grep -qiE "SIGSEGV|signal: 11|segmentation fault" /tmp/self-analyze.log 2>/dev/null; then | ||
| echo "::error::SIGSEGV detected in self-analyze — derive-phase heap corruption regressed." | ||
| grep -iE "SIGSEGV|signal: 11|segmentation fault" /tmp/self-analyze.log | head >> "$GITHUB_STEP_SUMMARY" || true | ||
| exit 1 | ||
| fi | ||
| if grep -qiE "plan reject|rejected by planner|MAX_MATERIALIZED_FACTS guard" /tmp/self-analyze.log 2>/dev/null; then | ||
| echo "::warning::Derive plan rejects present in self-analyze log (planner cap hit)." | ||
| fi | ||
| echo "No SIGSEGV in derive phase." | ||
|
|
||
| - name: Upload self-analyze log | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: self-analyze-log | ||
| path: | | ||
| /tmp/self-analyze.log | ||
| /tmp/mcp-tool.json | ||
| retention-days: 14 |
Owner
Author
|
Superseded by the consolidated PR #475 (feat/self-analyze-hardening) — all 6 branches merged clean into one. Commits preserved there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Grafema's CI runs CodeQL Analyze jobs — but it never runs grafema's own
grafema analyzeagainst this repo. That gap is what let a whole class of self-scale bugs ship silently:max_intermediate_results,MAX_MATERIALIZED_FACTS)These only bite when grafema analyzes its own ~714k-node / 1.33M-edge monorepo, and are unreachable by the unit/integration tests (which run on tiny fixture graphs). All of the above were fixed on
fix/derive-batch-cap(#469); this PR adds the guard so they can't regress silently again.What this adds
New workflow
.github/workflows/self-analyze.yml:pnpm build(TS) +cargo build --releasefor the two native binaries the analyze pipeline spawns (rfdb-server+grafema-orchestrator, auto-detected frompackages/*/target/release/).grafema analyze --clearon the whole monorepo via the checked-in dogfooding config (.grafema/config.yaml), with the self-analyze env fromfix/derive-batch-cap:RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000.mcp:toolnodes (produced by the behavior enricher onpackages/mcp/src/server.ts). A clean exit with an empty graph is the silent-failure mode this guard explicitly catches.SIGSEGV/ plan rejects.Runner / scope choice
ubuntu-latest(GitHub-hosted, 16 GB RAM / 4 vCPU). A full self-analyze peaks ~7.5 GB RAM / ~14 min — it fits in 16 GB with headroom.@stdlib/js_local_refs) and the cap rejects (js_property_access_full). A scoped single-package analyze would not reproduce those cross-package derive paths, so scoping was rejected. If a future graph outgrows 16 GB, the header documents the escape hatch (larger runner label, or scope via--service).Non-blocking (intentional)
The job is non-blocking:
continue-on-error: true, not a required status check. As offix/derive-batch-capthe derive phase completes with 0 SIGSEGV, but a full self-analyze still fails late on an enricher RPC 60s client timeout (being fixed separately onfix/enricher-rpc-timeout). Until self-analyze is green end-to-end, this job runs and surfaces the result without blocking PRs.There is a
TODO(make-required)in the workflow header: oncefix/enricher-rpc-timeoutlands and a full self-analyze reaches exit 0 withmcp:tool > 0reliably, removecontinue-on-error: trueand addself-analyzeto branch-protection required checks.Validation
grafema query --type mcp:tool "" --jsonexits 0 and returns a parseable JSON array ofmcp:toolnodes (61 on that graph). Confirms the exit-0 + non-empty-enricher assertion works against a real self-analyze graph.🤖 Generated with Claude Code