poc: Mokka + AICR pre-silicon coverage sweep, measured twice

[ArangoGutierrez]

Problem

The 2026-07-25 POC (#503) answered whether AICR runs against Mokka at all. It could not answer the
question the AICR maintainer actually put on 2026-07-24:

Does Mokka + AICR materially reduce GB200 bring-up time, or does it merely prove the stack deploys
against mocked APIs?

Answering that needs a permutation matrix and a way to say what each check proves when no GPU is
present, not a single recipe and a pass count.

Approach

A harness at tests/aicr-sweep/ classifies every AICR check into A (meaningful), B (trivial),
C (hardware-dependent) or G (closable Mokka gap), and records a cause code for anything that did not
reach a verdict. Blocked cells stay in the denominator, so coverage cannot be inflated by deleting
the work that did not happen.

It is built so an incomplete run degrades to "not run" rather than to a favourable number. Seven
invariants each have a test that goes red if it regresses, and invariant 1 was mutation-checked:
flipping OutcomeNotRun to OutcomePass in Classify turns two tests red.

Two runs are recorded, and both regenerate byte-identically:

Run	Cells	Results	Mokka
2026-08-03	cells.yaml	results/	v0.3.0
2026-08-26	cells-57ef016.yaml	results-57ef016/	57ef01659

AICR is pinned to 0752ea14 in both, so Mokka is the only axis that moved between them. Verified
rather than assumed: the base cell's generated recipe is byte-identical across the two runs.

What it found

The re-run's headline is negative, and that is the point. #672 fixed the PCI bus ID gap the first
run reported (0000:01:00.0 to 00000000:01:00.0, confirmed by a differential probe of the two
published images). gpu-operator-health did not move. GFD now receives a well-formed BDF and
still cannot read the vendor file, because Mokka's rendered PCI tree uses relative symlinks and is
not grafted onto GFD's /sys. Both statements are true at once, and AICR's check cannot tell them
apart: it only sees ClusterPolicy state=notReady.

One verdict flipped. dra-support passes now, because k8s-dra-driver-gpu v0.5.0 released
altProcDevices on 2026-08-19. D-009 had recorded that flag as unreleased, which was correct when
written. Mokka's IMEX rendering already worked at v0.3.0; a released consumer was what was missing.

Coverage moved a long way. 13 of 210 check-results reached a verdict in the first run; 53 in the
second. The BUDGET and U blocks are both zero. The largest single gain is the h100 cell, where
AICR dispatches 14 of 21 checks against 9 for gb200.

Two cells the first run blocked were misattributed, and this PR reclassifies them with CLI output
as evidence: both declare intent=training, which resolves no recipe on the kind service for
gb200. That is X, not BUDGET/U.

Two consumers, one shape of problem. GFD is dynamically linked and fails on a sysfs path.
AICR's own snapshot agent is a static ko-built Go binary with no PT_INTERP, so library
interception cannot reach it at all, and it fails on sysfs content. Raised by @giuliocalzo in
review and verified here against the published image. The common remedy is a filesystem graft, not
library interception.

An injected uncorrectable ECC fault moves no verdict. With the mock reporting 6 and 9 DRAM
Uncorrectable errors and all four GPUs still enumerating, every dispatched check is identical to the
base cell. That is a statement about which checks this recipe dispatches, not about Mokka.

Testing done

go build ./tests/aicr-sweep/...
go test -race -count=1 ./tests/aicr-sweep/
golangci-lint run ./tests/aicr-sweep/...
go run ./tests/aicr-sweep -catalog ... -cells ... -results ...   # both runs

Build clean, tests pass, lint reports 0 issues, and both recorded runs regenerate byte-identical
coverage.md and report.json. Every number in the write-ups comes from that last command rather
than from memory.

Known limits, stated rather than buried

• Nothing here ran on silicon. Every record carries sim provenance. Bucket C is non-empty by
  construction: 4 checks cannot be judged without hardware.
• The chart came from the checkout, not from a release. Chart.yaml on main is still 0.3.0, so
  the published OCI chart does not carry the 60 commits under test. The image is the published
  sha-57ef016, so anyone can pull it by digest.
• The GFD failure is still cause K, not G. D-014 named the experiment that would settle it
  (run with cdi.enabled=true and the toolkit installed). It has not been run, the honest status is
  "not distinguished", and it is the single most valuable cell still outstanding.
• dra-support needed --set nameOverride. v0.5.0's chart was renamed, so its default resource
  names no longer match what AICR looks for. AICR itself was not modified.

Breaking changes

None. This adds a test harness and documentation. No production code path changes.

Relationship to other work

Supersedes and replaces #503, now closed. Its code had already landed on main via #541 and #542, and
the IMEX renderer it introduced now lives at internal/imex/ after #729 relocated it. The
preflight-workflow diagram and the diagrams index are carried forward here; its harness and
2026-07-25 results are deliberately left behind, because that harness will not be in the tree and
those numbers could not be regenerated or checked. They remain reachable at pull/503/head.

Related: #502 tracks the coverage gaps, and its body still carries the July figures that this PR
supersedes. Not closing it here, since correcting those numbers is a separate call.

[greptile-apps]

Greptile Summary

This PR adds a reproducible AICR-on-Mokka coverage sweep, two recorded result sets, supporting evidence, and documentation explaining the pre-silicon coverage boundary.

• Introduces CTRF parsing, conservative result classification, matrix roll-ups, and JSON/Markdown rendering.
• Records and compares sweeps against Mokka v0.3.0 and commit 57ef016.
• Adds operational documentation and diagrams describing coverage, failure attribution, and the silicon readiness gate.

Confidence Score: 4/5

The PR should fix cell-ID validation before merging because duplicate IDs can reuse and double-count a single result directory; duplicate CTRF handling is additional non-blocking hardening.

The harness trusts operator-supplied cell IDs as unique result identities, allowing repeated IDs to classify the same on-disk results more than once and corrupt the coverage roll-up.

Files Needing Attention: tests/aicr-sweep/main.go, tests/aicr-sweep/classify.go, tests/aicr-sweep/ctrf.go

Important Files Changed

Filename	Overview
tests/aicr-sweep/main.go	Orchestrates matrix loading and report generation, but does not enforce unique cell IDs before mapping cells to result directories.
tests/aicr-sweep/classify.go	Implements conservative classification and roll-up invariants; duplicate runtime test names are silently overwritten.
tests/aicr-sweep/ctrf.go	Parses and merges phase reports deterministically, although merged duplicate names have no explicit resolution contract.
tests/aicr-sweep/report.go	Produces deterministic machine- and human-readable roll-ups with explicit simulation and roadmap qualifications.
tests/aicr-sweep/classify_test.go	Covers core classification invariants and catalog validation but omits duplicate merged-result and cell-ID cases.
tests/aicr-sweep/catalog.yaml	Defines the evidence-backed A/B/C/G classification used by both recorded sweeps.
docs/aicr-preflight/coverage.md	Documents measured versus analytical coverage and preserves the boundary between simulated evidence and silicon-only checks.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  C[Catalog and cell matrix] --> R[Per-cell CTRF reports]
  R --> P[Parse and merge reports]
  P --> K[Classify each catalog check]
  K --> U[Roll up checks across cells]
  U --> J[report.json]
  U --> M[coverage.md]
  K --> N[Blocked and not-run remain in denominator]
  K --> S[Simulated hardware-only passes marked suspect]

Loading

_{Reviews (1): Last reviewed commit: "fix(docs): unbreak the site build the ai..." | Re-trigger Greptile}

[greptile-apps]

Duplicate cell identities

When an operator-supplied cells file repeats an ID, both entries load the same CTRF directory and classify it as separate matrix cells, causing duplicate attribution and an incorrect aggregate coverage result. Validate that cell IDs are unique before processing the matrix.

[greptile-apps]

Duplicate verdict overwrite

When multiple phase or retained rerun reports contain the same check name, this assignment silently keeps whichever entry sorts last by filename. The generated verdict therefore depends on incidental file ordering and hides the other result; reject duplicates or define an explicit resolution rule.