testing.md

Testing

Use the narrowest check that can falsify the change, then widen only when the changed contract crosses more layers. Local checks are fast proof of syntax, schema, generation, and transformation behavior. GPU execution is proof of allocation, serving, workload, and artifact behavior. A smoke run reduces feedback time, but it is not the full-sweep-and-eval evidence required for merge review.

Index

• Sources of truth
• Testing layers
• Test quality
• Local checks
• Smoke, sweep, and eval
• Evidence standard
• Review gates
• Stop conditions

Sources of truth

• .github/AGENT_OPERATIONS.md defines sweep labels and modifiers. Its dispatch section defines manual runs and artifact inspection.
• .github/workflows/README.md documents matrix generation, e2e-tests.yml, PR sweeps, and reuse.
• run-sweep.yml is the executable PR sweep gate, and merge-ingest.yml is the push-to-main reuse and ingest gate. e2e-tests.yml is the manually dispatched end-to-end path.
• docs/PR_REVIEW_CHECKLIST.md is the merge-review standard. The verifier prompt states how sweep and eval evidence is independently checked.

These sources outrank this guide when behavior changes. Update this English guide to match them.

Testing layers

CI runs Lint and Tests in parallel for PRs (including forks) and pushes to main that change Python files, .github/scripts/ helpers, ci.yml, pyproject.toml, uv.lock, .python-version, Ruff configuration, or pytest.ini. Workflow security runs Zizmor for changes to workflows, action definitions, Dependabot, pre-commit, or zizmor configuration. Python-only changes do not trigger Zizmor; other workflow-only changes do not trigger Lint or Tests. Editing ci.yml triggers all three jobs. Each workflow can be dispatched manually. Changes only to other docs, shell scripts, or benchmark YAML do not trigger either workflow; run the applicable checks locally or dispatch them manually.

Tests runs suites under infx/tests/, utils/, runners/, ../collectivex/tests/, and ../operatorx/tests/ with four pytest workers, excluding the upstream utils/srt-slurm/ suite. New tests in those directories are discovered automatically. CI installs infx as a wheel with uv sync --locked --all-extras --group test --no-editable, using Python 3.12 and CPU-only PyTorch. A failing job does not cancel the other; a newer PR update cancels the superseded CI run. Branch pushes without a PR no longer start a separate changelog-test run.

Layer	What it can prove	What it cannot prove
Parse and syntax	Edited YAML loads, and edited Bash parses	Schema validity, runtime routing, or GPU behavior
Schema and matrix	A config key validates and emits the intended matrix fields	Runner availability, server startup, or performance
Focused Python tests	Changed generator, changelog, result, eval, collection, or reuse contracts behave on covered inputs	Container, accelerator, network, or Slurm behavior
Smoke run	One tightly filtered path allocates, starts a server, runs a workload, and emits artifacts	The complete concurrency/search space or merge eligibility
Manual trimmed run	Each selected single-node and multi-node deployment shape runs its lowest concurrency (e2e-tests.yml with trim-conc: true; in changelog-ref mode only throughput rows are trimmed, and evals keep their selected concurrency). No PR label trims concurrency	Intermediate concurrency points required by a full sweep
Full sweep and eval	The selected untrimmed matrix and eval jobs execute on the reviewed commit	Correctness of evidence that was not inspected, or unrelated configurations

A green later layer does not erase missing earlier evidence. For example, a green collector can aggregate an empty set, so review must inspect the underlying executed jobs and artifacts.

Test quality

The mandatory test rules (forbidden test types, what a kept test looks like, and the four review questions) live in AGENTS.md. Two additions apply here:

• Preserve genuine contracts: numerical results, rejected invalid inputs, stable artifact formats, and agreement between independently consumed configurations. Assert only the parts of the contract the consumer needs.
• Control clocks and long waits in timing tests. Synchronize on observable readiness, keep process termination and artifact writes real when testing those contracts, and bound waits and cleanup so regressions cannot strand test workers.

Local checks

Run end-to-end checks from inferencex-e2e/ and replace placeholders with the exact changed path or key.

Python environment

pyproject.toml defines the infx package and its dependencies; uv.lock records their resolved versions. Run uv sync --locked to install the core tooling, then uv run --locked python -m infx.matrix.generate ... to use the existing module commands. Use --extra workflows for CODEOWNER/GitHub integrations, --extra results for eval summaries/database comparisons. The test group holds the CPU test dependencies.

For development, uv installs the package in editable mode, so source edits apply immediately. CI installs a regular wheel, but the main pytest suite imports the source checkout. Dedicated installed-package tests create clean environments with only core or results dependencies and run outside the source checkout. They check recipe node counts, runner metadata, matrix rejection, packaged threshold loading, score validation, BFCL license attribution, and generated eval rows and summaries. Repository-dependent commands use their source checkout when installed editable; with a wheel, run them from inferencex-e2e/. Eval YAML/JSON resources and the Apache license ship with the package.

Use uv add for core dependencies, uv add --optional <extra> for an integration, or uv add --group test for test-only tools. Commit both the manifest and lockfile. uv lock --upgrade-package <name> updates a dependency; resolution enforces a 12-hour age cutoff from pyproject.toml. Ruff and Zizmor remain outside the lockfile so their CI checks keep using the latest eligible release. Benchmark images, vendor eval environments, and commands that measure historical checkouts retain their existing dependency setup.

Dependencies stay within the latest supported major; minor and patch upgrades are recorded in uv.lock and validated in CI. For pre-1.0 packages, review minor upgrades too. New releases do not automatically change a locked environment.

Result collection, comparisons, and run statistics use infx from current main, resolved once during sweep setup and shared across those jobs. Klaud resolves main once and passes that commit to every candidate; candidate hooks use a separate tooling checkout so recipe edits cannot replace the imported package. Sign-off also checks out current trusted main. These tools have no release-version pin or cooldown on repository commits. PR CI tests the PR's package, while matrix generation, benchmark scripts, and recipes retain the selected checkout. Sweep summaries show the resolved setup and tooling commits, plus the source head when reusing benchmarks. E2E benchmark and eval jobs use the same resolved SHA as matrix generation, even if the requested branch moves while jobs queue. Klaud summaries show the tooling commit used as the candidate base.

Python lint and formatting

Ruff checks infx/ with the rules in infx/ruff.toml, targeting Python 3.12 and a line length of 100. CI runs on any Python-file change and uses the latest Ruff release at least 12 hours old.

uvx --exclude-newer PT12H ruff@latest check --fix infx
uvx --exclude-newer PT12H ruff@latest format infx

Fix findings where practical. Justified exceptions use inline # noqa: CODE; unused ignores are checked. Preview rules and automatic unsafe fixes are not enabled.

GitHub Actions security

CI uses the latest zizmor release at least 12 hours old, with its strictest auditor persona, strict input collection, all supported input kinds, and online action-reference checks. Every unsuppressed finding fails the job, including informational and low-confidence findings. Run the same audit locally with an authenticated GitHub token:

GH_TOKEN="$(gh auth token)" uvx --exclude-newer PT12H zizmor@latest \
  --persona auditor --strict-collection --collect all --no-config --no-progress .

All third-party actions remain pinned to commit SHAs. Same-repository workflow calls use $/, which resolves the workflow's exact commit and requires Actions runner 2.336.0 or newer. Dependabot waits seven days before action updates. The combined Claude workflow keeps separate review and coding jobs with their own permissions; the pinned Claude action installs its supported CLI version.

Auditor mode also reports deliberate architecture choices. Exceptions are attached to the exact affected YAML line with a reason, never disabled globally:

• Independent GPU dispatches, comment requests, and Klaud waves must not supersede one another. The priority scheduler and candidate ownership claims handle their resource limits.
• Trusted external dispatch uses pull_request_target; it executes trusted control code and enforces authorization before privileged operations.
• Existing repository-scoped integration credentials are retained. Moving them into protected GitHub Environments requires migrating the actual stored secrets; adding an empty environment: field is not a fix.
• The profiling storage checkout retains its scoped SSH deploy key only because the next step pushes a trace commit to that separate repository. Benchmark checkouts do not retain credentials.

Add --no-ignores to review all of these exceptions. Keep new findings blocking, and review an exception again if its trigger, checkout, credential consumer, or authorization changes. No GPU execution is needed to run this security audit.

Parse and syntax

python3 -c "import yaml; yaml.safe_load(open('configs/<nvidia|amd>-master.yaml')); yaml.safe_load(open('configs/runners.yaml')); yaml.safe_load(open('perf-changelog.yaml'))"
bash -n benchmarks/<path>/<script>.sh
uv run python -c 'from infx.clusters import load_clusters; load_clusters()'

Parsing is only the first gate. Do not report a YAML parse as matrix validation.

Exact config, then filtered family

uv run --locked \
  python -m infx.matrix.generate test-config \
  --config-files configs/<nvidia|amd>-master.yaml \
  --runner-config configs/runners.yaml \
  --config-keys <exact-key>

uv run --locked \
  python -m infx.matrix.generate full-sweep \
  --config-files configs/<nvidia|amd>-master.yaml \
  --runner-config configs/runners.yaml \
  --model-prefix <prefix> \
  --framework <framework> \
  --precision <precision> \
  --runner-type <runner> \
  --seq-lens 8k1k

Inspect the emitted values, not only the exit code or row count: config key, model, image, runner, scenario, concurrency, max-model-len, TP/PP/EP/DCP/PCP, prefill/decode workers, hardware, router, KV transfer, eval flags, additional-settings, and spec-decoding. The schema lives in validation.py, and the generator is generate.py.

Focused suites by changed contract

Change	Focused command
Matrix schema or generation	python -m pytest infx/tests/matrix/ -v
Changelog content or PR gating	python -m pytest infx/tests/matrix/test_process_changelog.py infx/tests/workflows/test_validate_perf_changelog.py infx/tests/workflows/test_prepare_perf_changelog_merge.py -v
Result processing and topology	python -m pytest infx/tests/results/power/test_process_result.py infx/tests/results/agentic/test_process_agentic_result.py infx/tests/results/power/test_aggregate_power.py infx/tests/workflows/test_calc_success_rate.py -v
AgentX aggregation and artifact loading	python -m pytest infx/tests/results/agentic/ -v
Eval dispatch, batching, staging, or patches	python -m pytest infx/tests/bench/test_eval_command.py infx/tests/bench/test_eval_meta.py infx/tests/bench/test_vendor_eval.py infx/tests/evals/ -v
Container-side python3 -m infx.bench commands	python -m pytest infx/tests/bench/ -v
Eval collection	python -m pytest infx/tests/results/test_collect_eval_results.py -v
Sweep reuse or reusable artifacts	python -m pytest infx/tests/test_github.py infx/tests/workflows/test_find_reusable_sweep_run.py infx/tests/workflows/test_acknowledge_sweep_reuse.py infx/tests/workflows/test_validate_reusable_sweep_artifacts.py -v

For an edited changelog, also run the same matrix-compatibility validator used by setup, with real base and head refs:

python3 -m infx.workflows.validate_perf_changelog \
  --changelog-file perf-changelog.yaml \
  --base-ref <base-ref> \
  --head-ref <head-ref>

Its contract is implemented in validate_perf_changelog.py. This check validates the generated matrix and rejects prohibited content changes, but whitespace-only historical deletions can be invisible to its diff reader. Inspect the exact byte diff as a separate evidence gate. Do not rewrite or normalize historical perf-changelog.yaml bytes.

A local matrix cannot prove Slurm allocation or llm-d endpoint discovery. Multi-node recipe changes still require the upstream recipe checker and an execution on the intended fleet.

Full local suite in parallel

The existing Python suites cover workflow contracts too. infx/tests/matrix/test_validation.py tests the workflow input schemas and runs both preparation scripts with controlled generator output. Invalid rows must fail before publishing job outputs; accepted rows must remain unchanged, including when manual dispatch measures an older checkout. infx/tests/results/power/test_process_result.py executes the shipped launch step with a recording launcher for current and historical checkouts. These tests do not emulate GitHub's expression engine or prove GPU performance; review expression changes with workflow validation and applicable smoke evidence.

Run the same locked environment and four-worker suite as CI:

uv run --locked --all-extras --group test --no-editable \
  python -m pytest infx/tests/ utils/ runners/ ../collectivex/tests/ ../operatorx/tests/ --ignore=utils/srt-slurm -n 4

Use -n 0 for serial debugging. Tests must keep temporary files and ports isolated and collect deterministic parameter cases across workers.

Smoke, sweep, and eval

Smoke

A smoke run is a manually dispatched e2e-tests.yml run whose generator command is restricted to the changed model/framework/runner/scenario and a minimal supported concurrency. Generate that exact command locally first. Use smoke evidence to answer a narrow question such as “does this image start this server on this runner and produce a result artifact?”

A smoke run is not merge evidence: it intentionally omits configurations and concurrency points. Likewise, agentx-fast is an AgentX preflight, not a canonical AgentX result. The dispatch reference defines the reduced warmup/profile behavior.

Trimmed and full sweeps

• full-sweep-fail-fast is the recommended full-sweep label. Unless the sweep has only multi-node fixed-sequence or eval entries, it first runs a canary (the lowest-concurrency non-eval single-node entry, or a multi-node AgentX entry when no single-node entry qualifies). It then stops each matrix after that matrix's first failure while preserving completed results.
• Use non-canary-full-sweep-enabled only when the canary is known to be flaky or unrepresentative. It also runs without fail-fast. Use full-sweep-enabled instead of fail-fast only when every matrix job must continue despite a failure.
• Apply exactly one primary sweep label. Modifier-only or conflicting primary labels do not constitute a valid sweep.
• A trimmed sweep (lowest concurrency only) can be run manually via e2e-tests.yml with the trim-conc input.

The current meanings and eligibility rules are in PR primary and modifier labels and implemented by run-sweep.yml.

Eval

Throughput and evals are separate jobs. The default sweep evaluates the selected 8k1k subset and the AgentX GSM8K subset. all-evals expands eval selection, and evals-only suppresses throughput. Choose modifiers from the changed scope, but do not substitute an eval-only or preflight run for the required full sweep.

Eval completion is not just a green job. Preserve and inspect meta_env.json, the results*.json files, the score-validation output, the inference image, and the aggregated eval artifact. infx/evals/EVALS.md owns task and artifact behavior. validate_scores.py rejects missing result files, below-threshold scores, and runs with no checked metrics. When expected concurrency metadata is available, it also rejects invalid/incomplete/failed batches. The single-node workflow invokes it without --expected-concs, so reviewers must verify meta_env.json independently for single-concurrency artifacts.

Evidence standard

Record enough information for another reviewer to reproduce the claim without guessing:

1. Exact commit SHA and whether it is still in the PR.
2. Exact local command or workflow generator command, including every filter and modifier.
3. Workflow URL, run ID, attempt, job/check name, and conclusion. Preserve failed logs before rerunning.
4. Config key plus resolved model, image, runner, framework, precision, scenario, topology, and concurrency range.
5. Artifact names and the relevant structured fields or summarized metrics. Do not paste an unbounded raw aggregate.
6. For evals, task, expected threshold, observed score, completion metadata, and proof that the evaluated image matches the PR config.
7. For a failure, record the first failing layer and the evidence that rules out earlier layers. Use the classification in troubleshooting.md.

“CI is green,” a screenshot without a run URL, a collector success without executed jobs, or an artifact from a rebased-out commit is not sufficient evidence.

Review gates

1. Before GPU work: parsing, exact-key generation, relevant focused suites, and inspection of emitted fields are green. The exact dispatch generator command has been run locally.
2. Before broadening: a smoke or canary proves the changed runtime path. If it fails, diagnose that layer before spending a full sweep.
3. Before CODEOWNER sign-off: follow PR_REVIEW_CHECKLIST.md, including its code-quality, architecture, image provenance, upstream recipe, patch/waiver, chat-template, and AgentX requirements where applicable.
4. For sweep/eval acceptance: at least one commit currently in the PR has successful, non-skipped executed single-node */ and eval / checks. A successful collect-evals alone is insufficient. Download the corresponding eval artifacts and confirm non-empty, passing accuracy and the same inference image. These are the executable rules in verifier Checks 1 and 2.
5. For reuse at merge: an authorized OWNER, MEMBER, or COLLABORATOR posts /use <run_id> on its own line to select the eligible source run (the legacy /reuse-sweep-run command remains supported) before the supported merge path. The verifier treats a missing or unauthorized command as a failure. See verifier Check 4 and the reuse procedure.
6. At merge: satisfy GitHub's Core-team and CODEOWNER approval requirements, or use an authorized maintainer bypass. Automated checklist verification posts an advisory comment; see the contribution guide.
7. After merge: the author confirms the main-branch jobs pass, as required by CONTRIBUTING.md.

Stop conditions

Stop and fix or escalate instead of widening the run or approving when:

• syntax, schema, exact-key generation, changelog validation, or a focused suite fails.
• generated fields differ from the intended config even if the command exits zero.
• the runner/fleet prerequisite cannot be exercised for a multi-node change.
• a smoke or canary fails and the failure layer is still unknown.
• expected jobs are skipped, cancelled, absent, or attached only to a commit no longer in the PR.
• eval artifacts are missing, empty, incomplete, below threshold, or use a different image.
• required evidence, an upstream recipe, a waiver, or a reviewer-owned checklist fact is unknown.

Do not convert an unknown into a pass, rerun until a flake disappears without preserving the original failure, or spend a broader sweep to hide a narrower failure.