Add Codex exec auth flow and sandboxed analysis support

[poppaLol]

Summary

This branch adds an authenticated codex exec analysis path alongside the existing Claude Code / external LLM orchestration paths, and fixes the sandbox handling needed for Codex and CodeQL tool subprocesses.

It also improves run-level logging ergonomics so operators can reduce console noise without changing logger call sites or weakening audit/file logging.

What Changed

• Added Codex CLI readiness/auth helpers in core/startup/codex.py.
• Added doctor/login support for Codex browser and device auth.
• Added --codex-exec analysis mode for agentic/autonomous LLM analysis flows.
• Added packages/llm_analysis/codex_dispatch.py to call codex exec with schema-constrained JSON output.
• Integrated Codex exec into orchestration as an analysis-only dispatch mode.
• Skips exploit, patch, retry, consensus, judge, aggregation, and grouped analysis stages for Codex exec mode.
• Skips LLM dispatcher startup and model-config construction for Codex-only runs, since Codex uses its own authenticated CLI session rather than RAPTOR’s SDK credential dispatcher.
• Runs Codex exec through run_untrusted_networked with a narrow host allowlist and scoped Codex config/auth paths.
• Redacts Codex diagnostics before surfacing startup/dispatch errors, matching the existing Claude-style secret-handling posture.
• Adds run-level agentic --log-level to reduce console noise without changing logger call-site levels.
• Passes trusted strict env into CodeQL database creation and analysis subprocesses so RAPTOR’s pid1 shim can dispatch while still stripping dangerous env vars.
• Clarifies that _RAPTOR_TRUSTED is an internal pid1-shim marker, stripped before target execution, and is not equivalent to --trust-repo.

Why

Codex exec and CodeQL are RAPTOR-owned tool subprocesses, but they execute inside RAPTOR’s sandbox. That means they need a careful two-part contract:

• treat target repositories and build commands as untrusted;
• allow RAPTOR’s own pid1 shim to dispatch internal tool subprocesses by passing _RAPTOR_TRUSTED=1.

Without this, sandboxed tool runs can fail with:

raptor-pid1-shim: internal dispatch script.
Run via 'bin/raptor' instead.

Codex also needs controlled access to its existing auth/config state and network access to OpenAI/Codex endpoints.

CodeQL needs the same internal shim treatment during database creation and query execution while preserving strict environment sanitisation before target build execution.

Validation

Focused Python validation passed:

.venv/bin/python -m pytest \
  core/startup/tests/test_codex.py \
  core/startup/tests/test_doctor.py \
  core/startup/tests/test_cli_smoke.py \
  core/startup/tests/test_check_lang_and_project.py \
  packages/llm_analysis/tests/test_codex_dispatch.py \
  packages/llm_analysis/tests/test_orchestrator.py \
  packages/llm_analysis/tests/test_verbose_flag.py \
  packages/codeql/tests/test_database_manager.py \
  packages/codeql/tests/test_query_runner_env.py -q

Result:

190 passed, 8 deselected

Manual validation:

• CodeQL CLI installed outside the RAPTOR repo.
• CUDA samples vectorAdd target built under CodeQL tracing.
• CodeQL database creation completed.
• CodeQL analysis completed with zero findings, which is plausible for vectorAdd.
• grm-rs was scanned with combined Semgrep + CodeQL + Codex exec analysis.

Full pytest was also attempted locally:

.venv/bin/python -m pytest -q

That did not complete cleanly in this local environment. The failures were concentrated around environment/tooling constraints, including local socket bind restrictions, a missing python2 fixture dependency for CodeQL Python e2e tests, and optional tool integration tests.

Notes

This branch intentionally does not include repo-local .tools/ or .gitignore changes. CodeQL should be supplied as a system/operator dependency and pointed to via CODEQL_CLI or --codeql-cli.

This branch also intentionally does not retain verbose CodeQL log-tail collection. That was useful while diagnosing local build failures, but it was too much diagnostic collection for the PR surface.

Follow-up worth considering: audit other tool integrations such as AFL++, Semgrep, Frida, GDB/rr, Coccinelle, jadx, and similar tools for the same sandbox/tool-env pattern.

[poppaLol]

Ran this against my own code base grm-rs to test, but that output is long. Here is a simple run using vectorAdd from CUDA Samples as the build target, with local paths are redacted to [~]

~/source/raptor$ python raptor.py codeql \
  --repo ../cuda-samples \
  --languages cpp \
  --codeql-cli "$CODEQL_CLI" \
  --build-command 'cmake --build build-codeql-prefetched --target vectorAdd --clean-first --parallel' \
  --sandbox network-only \
  --phase-timeout 1800 \
  --out out/cuda-samples-codeql-vectorAdd-2
OUTPUT_DIR=[~]/source/raptor/out/codeql_20260625_123243_pid300845_8121
Target license: BSD-3-Clause
Target: C++ (93.4%, 2.2M LOC), cmake, c.userspace-daemon — $25-$50, 40-75 min estimated

[*] Running CodeQL analysis...

[INFO] llm-dispatcher server.start ok pid=None uid=None token=- label=-
[INFO] llm-dispatcher token.issue ok pid=None uid=None token=sfUaELFPH27I label=raptor_codeql.py
binary-oracle: no locally-built debug binaries detected; running unfiltered. Build the target or pass --binary <path> for dead-code filtering (--no-binary-oracle to silence).
[WARNING] Sandbox profile forced to 'network-only' by CLI --sandbox flag
[INFO] ======================================================================
[INFO] RAPTOR CODEQL - AUTONOMOUS SECURITY ANALYSIS
[INFO] ======================================================================
[INFO] ======================================================================
[INFO] PHASE 1: CODEQL SCANNING
[INFO] ======================================================================
[INFO] Database manager initialized: [~]/source/raptor/codeql_dbs
[INFO] CodeQL CLI: [~]/source/raptor/.tools/codeql/codeql/codeql
[INFO] Query runner initialized with CodeQL: [~]/source/raptor/.tools/codeql/codeql/codeql
[INFO] ======================================================================
[INFO] RAPTOR CODEQL AGENT
[INFO] ======================================================================
[INFO] Repository: [~]/source/cuda-samples
[INFO] Output: out/cuda-samples-codeql-vectorAdd-2
[INFO] ======================================================================
[INFO] PHASE 1: LANGUAGE DETECTION
[INFO] ======================================================================
[INFO] Using specified languages: cpp
[INFO] ✓ Detected 1 language(s):
[INFO]   - cpp: 0 files (confidence: 1.00)
[INFO] ======================================================================
[INFO] PHASE 2: BUILD SYSTEM DETECTION
[INFO] ======================================================================
[INFO] cpp: Using custom build command
[INFO] ======================================================================
[INFO] PHASE 3: DATABASE CREATION
[INFO] ======================================================================
[INFO] Creating 1 databases in parallel (max workers: 2)
[INFO] ======================================================================
[INFO] Creating CodeQL database for cpp
[INFO] ======================================================================
[INFO] ✓ Using cached database for cpp: [~]/source/raptor/codeql_dbs/5b3964c1a9666dba/cpp-db
[INFO] ✓ cpp database completed
[INFO] ✓ Created 1 database(s):
[INFO]   - cpp (cached)
[INFO] ======================================================================
[INFO] PHASE 4: SECURITY ANALYSIS
[INFO] ======================================================================
[INFO] Analyzing 1 databases in parallel (max workers: 2)
[INFO] ======================================================================
[INFO] Running CodeQL analysis for cpp
[INFO] ======================================================================
[INFO] Using security-and-quality suite: codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls
[INFO] Executing: [~]/source/raptor/.tools/codeql/codeql/codeql database analyze [~]/source/raptor/codeql_dbs/5b3964c1a9666dba/cpp-db codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls --format=sarif-latest --output=out/cuda-samples-codeql-vectorAdd-2/codeql_cpp.sarif --no-rerun -j 0 -M 7735
[INFO] Timeout: 2400s
[INFO] Sandbox (net+pid+nproc:1024+limits): [~]/source/raptor/.tools/codeql/codeql/codeql database analyze
[INFO] ✓ Analysis completed for cpp
[INFO]   Findings: 0
[INFO]   Queries executed: 179
[INFO]   Duration: 6.4s
[INFO]   SARIF: out/cuda-samples-codeql-vectorAdd-2/codeql_cpp.sarif
[INFO] ✓ cpp analysis completed: 0 findings
[INFO]   - cpp: 0 findings
[INFO] ======================================================================
[INFO] PHASE 5: REPORT GENERATION
[INFO] ======================================================================
[INFO] ✓ Report saved: out/cuda-samples-codeql-vectorAdd-2/codeql_report.json
[INFO] ✓ Phase 1 complete: 0 findings
[INFO] Scan-only mode - skipping autonomous analysis

======================================================================
CODEQL ANALYSIS SUMMARY
======================================================================
Repository: [~]/source/cuda-samples
Duration: 6.8s
Status: ✓ SUCCESS

Languages detected: 1
Databases created: 1
Analyses completed: 1

Total findings: 0
SARIF files: 1

SARIF outputs:
  - out/cuda-samples-codeql-vectorAdd-2/codeql_cpp.sarif

Output directory: out/cuda-samples-codeql-vectorAdd-2
======================================================================

[grokjc]

Hi - this is a substantial piece of work, thanks! A few observations/questions if I may:

Bug

cost_usx_unknown typo in test_codex_dispatch.py - the production code sets cost_usd_unknown but the test asserts cost_usx_unknown. This is a KeyError at runtime; needs to be cost_usd_unknown.

Design questions

1. What does the Codex CLI path provide over the existing OpenAI API path?

RAPTOR already supports OpenAI models via the external LLM dispatch - set OPENAI_API_KEY and you get structured output, defense envelope, cost tracking, consensus, the works. This PR adds a second route to OpenAI models that bypasses all of that infrastructure.

If the value is subscription-based access without an API key, that's worth understanding but the tradeoffs (below) should be visible to the operator, not silent.

If the value is that Codex can read the full repository via --cd for richer analysis, that's a genuinely different capability. But it also means the agent sees files RAPTOR didn't curate, and crafted repo files could influence its analysis directly - that needs its own defense story rather than inheriting a pipeline designed for controlled prompt/response.

It would help to have the intended use case spelled out so the design choices make sense to future readers.

2. Defense envelope is bypassed for Codex dispatch

The canary probe (orchestrator.py:~717-721) builds _models_to_probe from analysis_models_all, which is [] in Codex mode - so the entire envelope probe, nonce-tagged datamarking, and XML envelope are skipped. The only injection defense for finding content (which comes from scanning untrusted repos) is the plaintext CODEX_EXEC_TRUSTED_PREAMBLE.

The external LLM and CC paths both get structural defenses verified by probe. The Codex path doesn't, and nothing tells the operator.

Suggestion: if Codex can't support the envelope, surface the weaker posture in operator-facing output (not just JSON metadata) and require something like --accept-weakened-defenses so the operator consciously opts in.

3. No cost or budget control

invoke_codex_exec returns cost=0.0, tokens=0, so the CostTracker adaptive cutoff never fires. On a large codebase the run dispatches unbounded Codex calls with no budget gate. cost_usd_unknown=True in the report is honest but doesn't protect the operator mid-run.

Suggestion: a finding-count cap defaulting lower for Codex mode, or at least a banner like "Codex mode: analysing N findings (no cost tracking available)" so operators know what they're getting into.

4. Single-pass unverified verdicts

Every post-analysis quality gate is disabled: retry, consensus, judge, aggregate, group. The result is single-pass unverified verdicts on security findings which is a meaningful reliability gap compared to every other dispatch path.

Suggestion: enable the self-consistency retry stage at minimum as it's lightweight (re-dispatches only contradictory or low-confidence results) and catches the worst single-pass errors. If that's not feasible here, an operator-visible note in the report summary that verdicts are single-pass would help.

---

None of this is meant to block the direction and the sandbox work and auth delegation are great pieces of work. But taken together, the Codex path has fundamentally weaker assurances than the paths it sits alongside, and nothing surfaces that to the operator at run time. A finding-count cap, enabling retry, and gating on --accept-weakened-defenses would close the main gaps without a lot of additional work.

What do you think?

[poppaLol]

@grokjc - thanks for looking! 🙏

If this direction is adopted for future then it needs more work to fit the existing sequence in the pipeline - again I can probably see how to adopt that in a better way. Its a first attempt, and although the connection works, its done with very little understanding of the whole codebase and the reasons behind how its developed - your comments here are really helpful with that.

Your other observation here is important too - the target repo under inspection is not to be trusted - and there needs to be some amount of work to ensure the code under the microscope is not affecting the analysis with hidden prompts in any way. I can certainly keep that separation in mind. I think what this points to is that I need to also get my head around how the current structure works to derisk that. Or like you suggest in various areas - be super clear with the user that there is a risk.

The use case:

For me its just to be able to use this tool with a lowly ChatGPT Plus account, where I dont think I can generate API keys. As a result I thought I would take a crack at setting it up and seeing what it was like. One possible use case I kind of wondered about, was consensus between LLM's having seen the same prompts, or comparative sample exploit code. But this isnt something I am actually working on, just theorizing.

The bug

I think can probably fix that up - I will take a look! Seems low effort.

1. Yeah good point! I hadn't realised the wider implications, and certainly (because of my lowly "plus" level token access) I had seen this exhaust tokens and then say nothing much about the analysis which continued after... so I think have misunderstood the overall pipeline, and integrate with that
2. I think I should probably try and understand the defense envelope - and see if its plausible to support it. Otherwise the suggestion is worth exploring - thanks!
3. Definitely this was evident in some of the test runs I went through, and that confirms its another part of the pipeline to integrate. Let me see what I can do
4. Again - I think I like this part of the suggested approach: enable the self-consistency retry stage at minimum as it's lightweight (re-dispatches only contradictory or low-confidence results) and catches the worst single-pass errors.

I think maybe I should just keep chipping away - but I am also conscious that this change will become quite weighty and difficult to review. If I carry on on my own fork branch and see how it looks there, then maybe I can break it out into meaningful chunks too. I will let you know if this is necessary. In any case - this is all "learning by doing" so appreciate the comments!

[grokjc]

Having taken another look at this I think some of your changes here are genuine improvements to main with no Codex dependency whatsoever and splitting them out would shrink the review surface and get value merged sooner:

• --log-level flag + the _configure_run_logging refactor in raptor_agentic.py (with its test). Useful for anyone wanting to quiet sandbox/proxy chatter.

• Type annotation fixes — Optional[str] = None in metadata.py / output.py. Pure cleanup, zero runtime change.

• Test mock target fixes five tests in test_orchestrator.py patch packages.llm_analysis.cc_dispatch.subprocess.run, but cc_dispatch.py already uses core.sandbox.run_untrusted_networked on main. The mocks are dead, they never fire, and the tests pass coincidentally. Your fix to patch the real target is 100% correct. Worth checking that the mock side_effect functions handle run_untrusted_networked's kwargs (target=, output=, proxy_hosts=) since they'll actually intercept calls now.

The _intersect_profiles empty-set guard and _detect_multi_model_collapse cleanup are also harmless to include, cosmetic rather than hardening, but your versions are cleaner.

If you fancy landing some or all of that as a separate PR I'd appreciate it!

CodeQL _RAPTOR_TRUSTED env var needs a second look

The query_runner.py strict_env=True addition is fine, codeql database analyze doesn't execute target code, and the env was already going through get_safe_env() so strict_env is effectively a no-op albeit cosmetically cleaner.

database_manager.py is different. That call already uses sanitise_host_fingerprint=True specifically to prevent the target's build scripts from detecting the analysis environment. Adding _RAPTOR_TRUSTED=1 to the same env works against that, it's a trivial tell for any anti-analysis-aware build. The pid1 shim strips it before exec, but that only applies in namespace mode; on Landlock-only hosts (Ubuntu 24.04+ default) it leaks through to the build. I'd hold that change out of a split PR until the stripping is reliable across both paths.

On the Codex-as-agent point

One thing worth keeping in mind as you look at the envelope integration: codex exec with --cd repo_path gives the agent direct read access to the scanned repository. The existing dispatch paths (external LLM, CC) only see what RAPTOR puts in the prompt: curated finding snippets with envelope wrapping. The Codex agent can read arbitrary repo files on its own initiative, so even if you wire up the envelope for the prompt content, a crafted file elsewhere in the repo could still influence the analysis. That may need its own defense story beyond what the envelope provides.

Thanks for the support!

[poppaLol]

Yeah - sounds sensible 👍

I will split the commits up a bit and maybe retarget the effort towards that then - to be able to test it out on my own set up I will need to either (1) rebase the Codex connection on the new logging/annotion/mocks fixes and test - or - (2) part with the cash for some claude tokens... - or - maybe also hook up claude to my GPU in my home lab... its all possible tho!

Testing across setups will represent other use cases, and hopefully introduce robust changes.. 🤞

Probably be able to look at it over the weekend, but might be next week before I have the change in place. Happy to help tho! 😉

[danielcuthbert]

Echo'ng what @grokjc said here, this is bloody epic work!! thank you

[grokjc]

Good progress, thanks! Two gaps remain from the prior review, plus a few new things...

No fallback when all Codex calls fail

The CC-fallback block after the analysis dispatch loop is gated on dispatch_mode == "external_llm". When dispatch_mode == "codex_exec" and every call errors out (network hiccup, auth token expires mid-run, CLI crash), the fallback is silently skipped and the orchestrator proceeds with all-error results. You get a garbage report with no warning. Every other dispatch mode has a graceful degradation path; Codex doesn't. You don't need automatic CC fallback for v1, but if all results are errors, print something like "All Codex exec calls failed, check codex login status" with a recovery path. The operator shouldn't have to dig through JSON to discover nothing worked. That said, automatic fallback would be cool 😄

Defense probe still silently skipped

The canary probe verifies a model can parse the defense envelope (XML tags, datamarking, nonce). For Codex, analysis_models_all is empty so _models_to_probe is empty and the probe loop never runs. The CONSERVATIVE profile IS applied to the prompt, which is the right default, but there's no verification Codex actually handles it correctly. The report then says defense_profile: conservative, which reads as "validated" to an operator. The --accept-weakened-defenses gate is never triggered even if Codex mangles the envelope.

Two options: either wire up a synthetic probe against codex_dispatch_fn so the existing gate kicks in naturally, or if that's not feasible yet, set defense_profile to "conservative-unverified" and log a warning so operators know the difference.

Schema conversion misclassifies two production fields

_json_type_from_description uses substring matching (if "bool" in lowered, if "int" in lowered) to convert RAPTOR's compact schema descriptions to JSON Schema types. The check order matters: "bool" is checked before "list", and "int" before the "string" fallback.

The production ANALYSIS_SCHEMA field path_conditions has a description that starts with "list of strings or null" but later mentions "Boolean predicates" in its body text. "bool" in lowered matche first, so the Codex output schema gets {"type": ["boolean", "null"]} instead of {"type": ["array", "null"]}. Same issue with path_profile, its description mentions "int32" / "int64", producing ["integer", "null"] instead of ["string", "null"]. I verified this by running the converter against the real schema.

The fix is to match the type keyword at the start of the description rather than anywhere in the body. The compact format is always "type_keyword ...", so splitting on the first word or a leading-word regex would be reliable.

_RAPTOR_TRUSTED changes in CodeQL files are unrelated, and redundant

The database_manager.py and query_runner.py changes add env["_RAPTOR_TRUSTED"] = "1" before passing the env dict into the sandbox. These aren't Codex-related, so they'd want to be a separate PR regardless.

But beyond that, the change doesn't actually do anything. Here's how the existing mechanism works:

The pid1 shim (libexec/raptor-pid1-shim) checks for CLAUDECODE or _RAPTOR_TRUSTED at startup. If neither is set it exits with code 2. Both of those markers are on the get_safe_env() allowlist in core/config/__init__.py:401, with an explicit comment saying they need to propagate through because the sandbox spawns its own libexec scripts using this env. So when CodeQL calls sandbox_run(cmd, env=RaptorConfig.get_safe_env(), ...), whichever marker is already in the parent process (CLAUDECODE when running under Claude Code, _RAPTOR_TRUSTED when running under bin/raptor) passes straight through to the child. The shim sees it and proceeds normally.

The PR's explicit env["_RAPTOR_TRUSTED"] = "1" is guarding against a situation where neither marker is set. But that can't happen: you can't reach CodeQL analysis without going through bin/raptor (which sets _RAPTOR_TRUSTED) or Claude Code (which sets CLAUDECODE). The inline comment ("this authorises RAPTOR's own sandbox wrapper to dispatch CodeQL") makes it sound like the shim would reject the call without this line, but it won't, because the marker is already there via the allowlist.

I'd say drop these two hunks from this PR entirely. If you want them as extra safety, that's a conversation for a separate PR, but given how the propagation already works they're adding a redundant write that could mislead future readers into thinking the allowlist doesn't cover this case.

invoke_codex_exec bypasses RAPTOR's sandbox layer

cc_dispatch.py routes through core.sandbox.run_untrusted_networked with Landlock filesystem restrictions, network namespace isolation, and a proxy host allowlist (that's what #861 updated the mock targets for). The Codex path calls subprocess.run() directly with get_safe_env(). Codex's own --sandbox read-only provides some protection, but RAPTOR's defense-in-depth doesn't apply. If routing through run_untrusted_networked with a Codex-specific host allowlist isn't feasible in v1, a comment and a TODO would at least document the intent.

Smaller items

• The assert dispatch_fn is not None after the if/elif/else dispatch construction is currently unreachable (every branch either sets it or returns), but assert is stripped by python -O. A raise RuntimeError(...) would hold up better for future refactors.

• args.codex_exec appears as a third or term in four separate conditionals across raptor_agentic.py (~2474, 2530, 2602, 2628). Each is correct on its own, but add a fifth dispatch mode and you'll need to find all four sites. Worth considering adding codex_exec to llm_env so llm_env.llm_available accounts for it.

• The --codex-exec + --sequential mutual exclusion at line 1439 is good validation but has no test coverage.

• codex_login_command returns ["login"] without the executable. Only caller is run_codex_login which prepends it. The name suggests a full command; _login_args would be clearer and private.

• The duplicate elif mode == "codex_exec": via = "Codex exec" blocks at ~3214 and ~3240 could be unified into a small helper.

This branch almost certainly needs a rebase onto current main now that #860 and #861 have landed. Should be clean since those PRs touched non-overlapping areas.

That's a lot of text to say there really isn't that much left to fix. Thanks for your work on this! 👍

[poppaLol]

Got into some work - so I am distracted at the moment - but I will come back to this soon!

Thanks @grokjc for checking over the last update tho - will digest comments and get it ready

[grokjc]

I took the liberty of merging the recent changes for you, which turned out to be a pain 😄 We got there in the end ... this will teach me to refactor things!

[poppaLol]

I took the liberty of merging the recent changes for you, which turned out to be a pain 😄 We got there in the end ... this will teach me to refactor things!

Thanks for doing this!! Will pick up the changes and continue soon!