Skip to content

PR-05: Deterministic runner (scanner) - #28

Merged
emmanuelgjr merged 1 commit into
mainfrom
improve/pr-05-deterministic-runner
Jul 18, 2026
Merged

emmanuelgjr merged 1 commit into
mainfrom
improve/pr-05-deterministic-runner

Conversation

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Phase 1 linchpin. A deterministic engine owns pattern matching; the LLM's job becomes orchestration and judgment. Everything in Phases 2–4 verifies against this. Depends on PR-03 (rules) + PR-04 (fixtures).

cli/dsgai_scan.py — single file, stdlib-only at runtime

  • Runs the ruleset via rg --pcre2, emits DSGAI-scan.json + SARIF 2.1.0 + a table summary — identically every run (SOURCE_DATE_EPOCH honored; collections sorted).
  • Redaction is structural, not behavioral: value-bearing rules run rg -o --replace '' so ripgrep erases the match text before emitting path:line:. The secret never enters this program, the pipe, the checkpoint, or the report. Findings have no content/match_text/value field — by construction.
  • Compound logic: subtract, requires_nearby (proximity + module scope), exclude_globs, gated_on (NOT APPLICABLE gating via a detect stage).
  • Per-control classification (PASS/WARN/FAIL/NOT VALIDATED/NOT APPLICABLE).
  • Discovery via git ls-files (honors .gitignore, still includes tracked files like the fixture .env); os.walk fallback; --scope/--exclude.
  • scan / detect / doctor subcommands; --version; exit codes 0/1/2 + --fail-on.
  • detect adds raw-endpoint signals (api.openai.com, …) and C# signals (Semantic Kernel / Azure.AI.OpenAI) ahead of PR-15.

tests/test_runner.py — 9 tests

Every PCRE compiles under rg --pcre2; the fixture scan matches expected-findings.yaml exactly (25 findings, compound logic resolved); must_not_flag holds; known_bugs tracked; no content fields; no fixture-secret substring in output; SARIF structure valid; a source-level guard that value-bearing execution always passes --replace; and the JSON stays in sync with the YAML.

Verification (all local)

  • pytest tests/ -q9 passed.
  • scan table + JSON match the answer sheet exactly (0 diff).
  • Two runs byte-identical with SOURCE_DATE_EPOCH fixed.
  • No FAKE substring in JSON or SARIF (redaction proof).

Also in this PR

  • requirements-dev.txt (PyYAML/pytest/jsonschema — dev only; runtime is stdlib) + dependabot pip for /dsgai_scanner_tool.
  • Repaired .github/dependabot.yml: its Docs comment had wrapped onto a new line without a #, leaving a stray scalar (ndabot.yml-file) that made the whole file invalid YAML — so dependabot couldn't parse it and the new pip block would have been inert. Diff is limited to the comment + pip block.

Follow-ups

  • Per-rule × per-file rg batching passes all candidate files on one command line; very large repos may want chunking (arg-length). Not needed for the fixture / typical repos.
  • The CI self-test workflow that gates external rule PRs lands in PR-06.

…-05)

A single-file, stdlib-only-at-runtime CLI that runs the ruleset via rg --pcre2
and emits DSGAI-scan.json + SARIF 2.1.0 + a table, identically every run.

- Value-bearing rules run location-only (rg -o --replace ''): ripgrep erases
  the match before emitting, so secrets never leave rg. Findings carry no match
  content by construction (no content/match_text/value fields exist).
- Compound logic: subtract, requires_nearby (proximity + module scope),
  exclude_globs, gated_on (NOT APPLICABLE gating via a detect stage).
- Per-control classification (PASS/WARN/FAIL/NOT VALIDATED/NOT APPLICABLE).
- File discovery via git ls-files (honors .gitignore, includes tracked files
  like the fixture .env); os.walk fallback. --scope / --exclude.
- Subcommands scan/detect/doctor; --version; exit codes 0/1/2 with --fail-on;
  SOURCE_DATE_EPOCH honored for reproducible artifacts.
- detect adds raw-endpoint signals (api.openai.com etc.) and C# signals.

tests/test_runner.py (9 tests): every PCRE compiles; the fixture scan matches
expected-findings.yaml EXACTLY (25 findings); must_not_flag holds; known_bugs
tracked; no content fields; no fixture secret substring in output; SARIF
structure; source-level guard that value-bearing always passes --replace; JSON
stays in sync with YAML.

Also: requirements-dev.txt + dependabot pip for /dsgai_scanner_tool, and repair
of the pre-existing broken wrapped comment that made dependabot.yml invalid YAML.

Acceptance: pytest green; scan output matches the answer sheet; two runs are
byte-identical with SOURCE_DATE_EPOCH fixed.
@emmanuelgjr
emmanuelgjr merged commit c6a119e into main Jul 18, 2026
8 checks passed
@emmanuelgjr
emmanuelgjr deleted the improve/pr-05-deterministic-runner branch July 18, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant