Skip to content

cli.md

Latest commit

 

History

History
142 lines (107 loc) · 8.45 KB

cli.md

File metadata and controls

142 lines (107 loc) · 8.45 KB
Raw

CLI reference

Every ifixai command and ifixai run flag. Guided first run: get-started.md.

Commands

ifixai init                  # check env for provider keys, suggest a first run
ifixai run                   # run inspections (Standard or Full mode)
ifixai run --fixture FILE    # custom fixture (YAML or JSON)
ifixai list tests            # all 50 inspections (32 core + 18 extended)
ifixai list fixtures         # registered named fixtures
ifixai validate [FILE]       # per-test layout, or a fixture against schema.json
ifixai compare A B           # diff two scorecard reports
ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION   # example: one category

ifixai run flags

System under test (SUT)

Flag Default Does
--provider, -p none mock, openai, openrouter, anthropic, gemini, azure, bedrock, huggingface, http, langchain.
--api-key, -k none SUT API key. Always passed explicitly, never read from the environment.
--model, -m provider default Model identifier override.
--endpoint, -e none Endpoint URL (required for http and azure).
--system-prompt, -s none Custom system instructions sent before each inspection.
--grounding sut Governance context source: sut (baked-in), fixture (system prompt derived from fixture), none.
--sut-temperature 0.0 SUT sampling temperature. B22 needs 0 or --sut-seed.
--sut-seed none SUT sampling seed (recorded in the manifest either way).

Judge

Flag Default Does
--eval-mode auto deterministic, single, full, or self. See How a run is judged.
--judge-provider none Judge provider. Repeat >=2 times for a Full-mode ensemble.
--judge-api-key, --judge-model none Key(s) and model(s), paired with --judge-provider.
--judge-budget 0 Max judge LLM calls per run (0 = unlimited).

Fixture and governance

Flag Default Does
--fixture, -f auto Fixture name or YAML/JSON path.
--governance none GovernanceFixture YAML path; structural inspections score against your declared policies. See fixture_authoring.md.
--mode standard standard or full (hand-built fixture + >=2 distinct judge providers). Deprecated alias --profile (quick → standard).

Suite subset

Flag Does
--strategic Top 8 strategic tests only.
--test, -b Test(s) by ID, repeatable: -b B01 -b B08.
--category, -c Category name(s), case-insensitive, repeatable; beats --strategic. Names: FABRICATION, MANIPULATION, DECEPTION, UNPREDICTABILITY, OPACITY, SABOTAGE, SUBVERSION, CONCEALMENT, SANDBAGGING, INSUBORDINATION, USURPATION, SYSTEMIC_RISK, MISCALIBRATION, STAKEHOLDER_CONFLICT, PERCEPTION_GOVERNANCE, OVERSIGHT_ATROPHY, PERSISTENCE, IDENTITY_ATTESTATION, INFLUENCE.

Output and reporting

Flag Default Does
--output, -o ./ifixai-results/ Report directory.
--format both json, markdown, or both.
--name, --version provider name, 1.0 System name and version label in reports.
--min-score 0.85 Exit code 2 if overall score is below (CI gate).
--quiet, -q off Suppress banner and summary; stdout still carries scores.

Execution and reliability

Flag Default Does
--timeout, -t 30 Per-request timeout in seconds.
--concurrency, -j 5 Max in-flight LLM requests (1-20). Overrides IFIXAI_CONCURRENCY.
--no-parallel off Alias for --concurrency 1.
--dry-run off Print inspection and judge-call estimates, then exit.
--reliability-out runs Directory for manifest.json, one subdir per run.
--run-nonce fresh Replay-protection nonce (16 hex chars), recorded in the manifest.
IFIXAI_JUDGE_FALLBACKS (env) packaged JSON Path to the judge fallback-model chain. See Judge fallback models.
--holdout-seed, --b{12,14,28,29,30,32}-seed fresh random Pin, or set matching IFIXAI_*_SEED, to replay a run. See reproducibility.md.

How a run is judged

The SUT (system under test) is the agent being graded via the SUT flags above; its key is never read from the environment. The judge grades the SUT's answers; a citable grade needs a judge from a second, different provider.

Judge auto-pairing

With no --judge-* flags, iFixAi picks a judge from a different provider whose key it finds in the environment, never the SUT's own. With only one credential it refuses unless you pass --eval-mode self. Preference order: testing-your-agent.md.

Evaluation modes (--eval-mode)

Mode What it does
deterministic Structural inspections only; no judge call.
single One cross-provider judge (--judge-provider required).
full Multi-judge ensemble (>=2 --judge-provider; Full mode only).
self SUT grades itself; grade prints but is flagged, not citable.

Standard vs Full

Same 50 inspections either way. Standard: auto fixture (the bundled default is a seeded-defect demo — expect 15/50 FAILs by design; see ifixai/fixtures/default/README.md), one auto-paired judge. Full (--mode full): requires --fixture (the default fixture is refused) and >=2 judge providers; majority vote, tie-break fail > partial > pass.

# Standard: export a second provider key and the judge auto-pairs.
ifixai run --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN"

# Full: hand-built fixture, two independent judges.
ifixai run --mode full \
  --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN" \
  --fixture ./my-fixture.yaml \
  --judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" \
  --judge-provider openai    --judge-api-key "$OPENAI_API_KEY"

Judge design rationale: methodology.md.

Judge fallback models

When the judge provider itself errors — an OpenRouter 502/503, a timeout, a dropped connection — the probe is retried on the next model in a priority-ordered fallback chain instead of killing the run. If every model is down, that one probe drops as INCONCLUSIVE and the run continues. A malformed judge reply is not a provider fault and never switches models.

The chain is plain JSON. Defaults ship at ifixai/judge/judge_fallbacks.json (OpenRouter: Gemini 2.5 Flash → GPT-4o mini → Haiku 4.5 → DeepSeek V3.2 → Qwen3 235B → GLM-5.2). The order is cheapest-and-tersest first: a verbose model burns real credit before it fails. Override by dropping an ifixai.judge-fallbacks.json next to your run, or point IFIXAI_JUDGE_FALLBACKS at any path:

{
  "providers": {
    "openrouter": {
      "attempts_per_model": 2,
      "models": [{ "model": "z-ai/glm-5.2" }, { "model": "openai/gpt-4o-mini" }]
    }
  }
}

attempts_per_model is how many provider errors one model absorbs before the chain advances. An empty models list disables fallbacks; with no fallback declared, the configured judge keeps its full retry budget instead. Substitutions are counted under fallback_grades in the run's judge stats, so you can see which model actually graded.

The SUT's own --model is dropped from the chain: a fallback must never turn a declared cross-provider grade into silent self-judging. Pointing the configured judge at the SUT is still --eval-mode self, which is declared and flagged.

Cost and time controls, so a dead judge cannot drain a key or stall a run:

  • Judge calls ask OpenRouter for no reasoning tokens. A hybrid-reasoning model asked for a verdict otherwise thinks out loud until the token ceiling cuts it off — billed in full, worthless as a verdict.
  • A truncated reply retires that model immediately rather than re-buying the same overrun.
  • A model that fails is retired for the rest of the run. The chain is not re-walked from the dead primary on every grade. Once all models are retired, remaining probes drop for free with no further calls.
  • The scorecard reports substitute judge graded this run: and judge calls that failed and were retried: whenever either happened, so a grade always names its origin.