An evidence-tracking diagnostic for structured-decision RAG.
Is your pipeline reading the evidence, or reacting to its presentation?
Production RAG pipelines fail in two directions, and most evals can't tell them apart:
- Presentation-reactive — reorder the retrieved sections and the decision flips, even though the facts are identical.
- Evidence-blind — swap the governing rule for a different valid one and the decision doesn't move, even though it should.
Labeled evaluations are expensive and go stale the moment a prompt or model changes. LLM-as-judge inherits the judge's blind spots and is structurally circular. Neither catches both failures above reliably.
Kelvin asks something stricter than "does the answer look right?": does the answer move only when the evidence that should determine it moves?
Kelvin applies metamorphic perturbations to the context a pipeline receives — reordering retrieved units, padding with peer units from other cases, and swapping governing units for same-type units from other cases.
The unit boundaries come from the corpus itself. In v1, the user asserts the unit typing by writing markdown section headers (e.g. ## Gate Rule, ## Market Evidence); automatic schema inference is the load-bearing v2 piece. See the whitepaper for the formal claim and its current limits.
The core signal is two measurements, together:
- Invariance — reorder or pad. Decision-relevant content hasn't changed; the output shouldn't either. Drift here means the system is responding to presentation, not substance.
- Sensitivity — swap a governing unit for a different valid one. The decision criterion has changed; the output should move with it. Insensitivity here means the system isn't reading the governing evidence at all.
Why the pair matters. Invariance alone is trivially gamed by a constant-output pipeline (return "stay_in_validate" scores 1.0 on every invariance test while being useless). Sensitivity alone can reward noise-reactivity. Only the pair separates a grounded pipeline from both failure modes. This is the central methodological claim — see whitepaper §3.4.
A concrete failure the tool caught. On the Envelop venture-assessment case, Kelvin's reorder perturbations flipped the stage decision from pre-seed to seed whenever the ## Gate Rule section was moved to the top of the input — classic retrieval-position bias. The facts were identical across reorderings; only the presentation changed. Exactly what the invariance signal is built to expose.
Range [0, 2], lower-is-better. K = 0 means perfectly anchored: invariant where it should be, sensitive where it should be. Higher = more thermal noise, borrowed from the absolute-temperature analogy.
The CLI reports K alongside Invariance and Sensitivity in the terminal summary and in kelvin/report.json. Per-stage decomposition (retrieval / reranking / generation) is v2.
Good fit — structured-decision RAG. Pipelines whose output is a categorical or scalar decision over a small set of values:
- Stage-gate assessors (
idea/pre-seed/seed/growth/scale) - Resume screening (
advance/manual_review/reject) - Underwriting, routing, triage, grading
- Any decision a human would make from a discrete set after reading typed evidence units
The evidence must arrive as discrete, identifiable units (documents, rows, chunks, clauses, tickets) — that's where the metamorphic relations live.
Not a fit (v1) — prose-output RAG. Summarization, open QA, chat. Kelvin scores a designated decision field; free-form rationales are recorded for inspection but ignored by the scorer. Tools like RAGAS and ARES target prose RAG and are complementary to Kelvin, not replaced by it.
Out of scope entirely for v1: perturbations that rewrite content inside a unit (paraphrasing, synonym swapping). Deferred to a later phase once the structural approach is proven.
pip install kelvin-evalOr from source:
git clone https://github.com/SbenaVision/kelvin
cd kelvin
pip install -e .1. Write a kelvin.yaml in your working directory:
run: python -m your_pipeline --input {input} --output {output}
cases: ./cases
decision_field: recommendation
governing_types: [gate_rule]
seed: 0run— shell command to invoke your pipeline. Must contain{input}and{output}.cases— folder of*.mdcase files. One file per case. Sections start with## Heading— each heading becomes a typed unit.decision_field— the JSON key your pipeline writes. Must be a top-level key in the output (no dotted paths —factsheet.delivery_modelis not supported). If your pipeline produces nested output, flatten it in a thin adapter harness before Kelvin sees it. The value must resolve to a scalar (str / number / bool / null).governing_types— unit types used for swap perturbations. Normalize to lowercase+underscores (e.g.Gate Rule→gate_rule).cache_dir— optional. Path to an on-disk invocation cache, relative to the working directory. Omit (the default) to disable caching; set to e.g..kelvin-cacheto skip pipeline invocations whose(run_template, rendered_markdown, decision_field)tuple has been seen before. Only successful invocations are cached; transient failures remain retryable on the next run.
2. Write a case file (cases/acme.md):
## Gate Rule
Revenue must exceed $10k MRR before Series A.
## Market Evidence
TAM estimated at $2B based on 2024 industry report.
## Customer Signal
Three enterprise pilots confirmed willingness to pay $500/mo.3. Run:
kelvin checkOptional flags: --only <case_name> to run a single case, --seed 42 to override the seed.
4. Inspect results:
jq '.invariance, .sensitivity, .sensitivity_by_type' kelvin/report.json
diff kelvin/acme/baseline/output.json kelvin/acme/perturbations/swap-gate_rule-01/output.jsonEverything lands under ./kelvin/:
| Path | Contents |
|---|---|
kelvin/report.json |
Cross-case scores: invariance, sensitivity, sensitivity_by_type, warnings |
kelvin/<case>/report.json |
Per-case: baseline decision, every perturbation's distance and notes |
kelvin/<case>/baseline/ |
Unperturbed input + output |
kelvin/<case>/perturbations/<variant>/ |
Each perturbation's input + output |
Exit codes: 0 success · 1 config or cases-dir problem · 2 decision field missing or every baseline failed.
- Teams shipping stage-gate, screening, triage, routing, or grading pipelines that need a repeatable evidence-tracking signal in CI.
- Engineers debugging why a model upgrade or prompt change quietly shifted a decision distribution.
- Anyone who has seen a production RAG "work" on spot-checks and later found it was reacting to retrieval position rather than content.
| Component | Status |
|---|---|
Core perturbations (reorder, pad, swap) |
✅ Done |
| Scorer — Invariance + Sensitivity | ✅ Done |
CLI (kelvin check) |
✅ Done |
| Terminal reporter | ✅ Done |
Kelvin score K (formalized, emitted) |
✅ Done |
| Grounded-vs-degenerate empirical table (Table 3) | ✅ Done |
| Corpus scaled to n=6 realistic cases | ✅ Done |
| Envelop harness — exponential backoff on transient 5xx | ✅ Done |
Pad split (pad_length / pad_content) |
✅ Done |
| Footgun warnings (governing-type validation, type discovery, single-case banner, cost preamble) | ✅ Done |
On-disk invocation cache (opt-in cache_dir) |
✅ Done |
Rule-condition swap (swap_condition) |
🔜 v0.3 — design in progress |
| HTML / markdown reporters | 🔜 Upcoming |
kelvin init wizard |
🔜 Upcoming |
| CI/CD integration | 🔜 Upcoming |
| Automatic schema inference | 🔜 v2 (load-bearing) |
| Stage decomposition (retrieval / reranking / generation) | 🔜 v2 |
The formal treatment of Kelvin's paired metamorphic diagnostics is in docs/whitepaper.md.
Apache 2.0. See LICENSE.