MC-30: Sync Database-entity-details; drop the vendored pipeline-records schema

docs/design/Database-entity-details.md

@@ -12,12 +12,12 @@
 > the PRD's adjudication field list contradicting its own routing section — is
 > tracked on the PRD follow-ups list.
 >
-> A machine-readable companion, `schemas/pipeline-records.schema.json`, mirrors every
-> entity here as a JSON Schema — emit cores plus landed `*_row` variants derived
-> from them. It is generated from POC-3's `publish/pipeline_records_contract.py`
-> and committed byte-identical in both repos. It is the artifact the CI contract
-> test validates pipeline-emitted records against, so drift between the pipeline
-> and the DDL fails a build instead of surfacing at integration.
+> A machine-readable companion JSON Schema mirrors every entity here — emit
+> cores plus landed `*_row` variants derived from them. It is generated from
+> the pipeline's `publish/pipeline_records_contract.py` and lands in this repo
+> with the POC-3 migration. It is the artifact the CI contract test validates
+> pipeline-emitted records against, so drift between the pipeline and the DDL
+> fails a build instead of surfacing at integration.
 
 This document lists the entities and fields the pipeline hands the storage
 database. It is the **logical contract** between the pipeline (producer) and the
@@ -62,7 +62,7 @@ shared number that bumps for everything conflates unrelated changes:
 
 | Version | Field | What bumps it |
 |---|---|---|
-| Envelope shape | `contract_version` | A change to the emitted record structure — the envelope (this contract). Already implemented as `SIDECAR_CONTRACT_VERSION` (currently `3` — ADR 0017 split the knob; POC-3 ADR 0020 bumped it for the four #318 per-record additions); envelope evolution bumps this knob without invalidating adjudications. Transform migrations key off it. |
+| Envelope shape | `contract_version` | A change to the emitted record structure — the envelope (this contract). Already implemented as `SIDECAR_CONTRACT_VERSION` (currently `4` — ADR 0017 split the knob; POC-3 ADR 0020 bumped it for the four #318 per-record additions, POC-3 ADR 0025 for the `documented` predicate); envelope evolution bumps this knob without invalidating adjudications. Transform migrations key off it. |
 | Scoring methodology | `scoring_plan_version` | Rules, prompts' meaning, adjustment magnitudes — methodology semantics only (ADR 0017 moved sidecar-shape changes to `contract_version`). Reporting, score labels, and curation staleness key off it. |
 | State source documentation | `snapshot_id` + `snapshot_digest` | Content identity of the elements artifact a run consumed. Already implemented (ADR 0018): `snapshot_id` is `{filename stem}@{digest[:16]}`; `snapshot_digest` is sha256 over a canonical form of the artifact (volatile run timestamps stripped), so same content means same digest across re-ingests and formatting differences. This is the PRD's `state_requirements_tag` — the doc uses the pipeline's names. |
 
@@ -74,6 +74,45 @@ Every score row's run traces to all of these: which envelope shape it was
 emitted under, which methodology produced it, and which edition of the state's
 documentation it scored — provably, via the digest.
 
+### Fact-run identity
+
+`release_id` and the snapshot digests cover ingestion inputs and scored
+records. Neither covers a **fact-run header** — the phase-A artifact's `mode`,
+`model`, `prompt_version`, `status`, `cost_cap_hit`, and counts — so a
+correction that changes only header metadata leaves every identity above
+byte-identical. That is not hypothetical: the `fact_run.mode` `"api"` → `"llm"`
+resolution (MC-24 §8.8) changed 135 headers with no change to any run identity,
+and a loader deduping on `ingestion_run_id` would silently keep the stale
+header.
+
+Two digests on `fact_runs` close that gap (nachos-ai-poc-3#345):
+
+- **`observations_digest`** — sha256 over the artifact's observation rows.
+  *What was extracted.*
+- **`fact_run_digest`** — sha256 over the identity-recipe version, every
+  non-volatile header field, and `observations_digest`. *The run.* **Dedup fact
+  runs on this.** Equal digest ⇒ the held row is current, skip. Different digest
+  ⇒ re-land, *even when `ingestion_run_id` and the snapshot digests are
+  unchanged*. Comparing the pair distinguishes a header-only correction
+  (observations equal, run digest moved) from changed data (both moved).
+
+The digest deliberately does not cover `scored_at` or the token/cost/cache
+counters — those describe the *occurrence*, and a cache-warm replay of identical
+facts legitimately reports `$0` and different token counts. The exact carve-out,
+the recipe, and the ordering tiebreak are published in the companion schema's
+`x-fact-identity` block, so both sides agree on precisely what the digest
+covers rather than inferring it.
+
+ADR 0027 (nachos-ai-poc-3#378/#381) adds two non-volatile header fields —
+`prompt_fingerprint` (the prompt bytes that produced the observations) and
+`answered_by_host` (which endpoint served the run). The deny-list covers both by
+construction, so the recipe is unchanged; the identity-recipe version moves
+1 → 2 purely as an epoch marker, published in `x-fact-identity.version_history`.
+The consequence is one-time and expected: every `fact_run_digest` in the tree
+changes at once while every `observations_digest` stays put, so the load reads it
+as a single producer-side recipe epoch — re-land every fact run once — rather
+than as changed data. Both fields are nullable and carry no backfill obligation.
+
 ### Assessor identity
 
 A scoring run is one assessor's sweep over a whole snapshot. The run record
@@ -140,11 +179,10 @@ views. This is the lowest-effort shape for the pipeline port (no per-table
 typed writes to maintain) and keeps every physical choice on the database side
 of the boundary. What the contract records about it:
 
-- **The emit contract is the payload shape**, pinned by
-  `schemas/pipeline-records.schema.json`. A JSONB landing zone accepts
-  anything, so drift would otherwise surface as a broken transform or a
-  silently-NULL dashboard column. The CI schema check is the primary
-  drift gate.
+- **The emit contract is the payload shape**, pinned by the companion
+  schema. A JSONB landing zone accepts anything, so drift would otherwise
+  surface as a broken transform or a silently-NULL dashboard column. The CI
+  schema check is the primary drift gate.
 - **Run identity is stamped at emit, or never.** `snapshot_id` /
   `snapshot_digest`, `release_id`, assessor identity — the transform cannot
   reconstruct these later; they arrive on every landed payload, and the
@@ -278,6 +316,8 @@ state explicitly authored or backfilled from the Ed-Fi swagger.
 | `source_page_or_section` | Location within the source document | TEXT |
 | `documented` | Whether the state source explicitly documents this element | BOOLEAN |
 | `documentation_source` | Provenance label: `source_doc`, `swagger`, or `swagger_leaf` | TEXT |
+| `source_document_description` | *(nachos-ai-poc-3#351)* Operator-supplied framing of the state documentation package this artifact was built from — e.g. "TEA publication Jan 2026" — sourced from the pipeline's committed `data/source_documents.yaml` registry. Required non-blank (schema `minLength: 1`) and uniform across every record of one artifact; display-only provenance, never scored. | TEXT |
+| `school_year` | *(nachos-ai-poc-3#351)* Operator-supplied school year the documentation targets, e.g. `2025-2026`. NULL when the publication does not target a single year. Uniform across every record of one artifact, alongside `source_document_description`. | TEXT |
 
 **Does ingest erase and recreate all rows for a state?** Under retain-by-run,
 each run inserts a fresh snapshot keyed by `ingestion_run_id`; rows from prior
@@ -419,7 +459,7 @@ progresses/completes.
 | `lens` | `source` or `spine` | TEXT |
 | `ingestion_run_id` | The ingestion snapshot this extraction ran against | BIGINT |
 | `snapshot_id` | Snapshot identifier of that ingestion run | TEXT |
-| `contract_version` | Envelope-shape version of the emitted records (see the version numbers) | TEXT |
+| `contract_version` | *(meaning clarified in the contract — touchdownllc/dataload_poc#27)* The **fact-identity recipe epoch** this run was landed under: the string form of `x-fact-identity.version`, read from the vendored contract at landing time rather than hand-typed. **Not** the sidecar envelope version that `scoring_runs.contract_version` carries — a fact run has no sidecar, and the two families version on independent lifecycles (the fact-identity recipe is ADR 0023's; the sidecar envelope is ADR 0017's). Landing-supplied by design: the phase-A header carries no `contract_version` to project. | TEXT |
 | `fact_name` | Fact label, e.g. `business_rules_present`, `has_conditional_logic`, or the v30 extension-necessity pair `state_requirement_basis` / `core_can_express_requirement` (which replaced the retired `extension_is_necessary`). Emitted as `fact` in the artifact header. | TEXT |
 | `artifact_name` | Short label for the run, e.g. `AZ_source_business_rules_present` | TEXT |
 | `artifact_path` | Path to the emitted artifact file (optional but useful for replay/debug) | TEXT |
@@ -430,7 +470,11 @@ progresses/completes.
 | `cost_cap_hit` | Whether the run hit its configured cost cap | BOOLEAN |
 | `schema_error` | Header-level schema validation error text, when present | TEXT |
 | `scored_at` | Header timestamp for artifact completion | TIMESTAMPTZ |
-| `source_hash` | Optional checksum for deduplicating repeated submissions of the same job | TEXT |
+| `observations_digest` | sha256 over the artifact's observation rows — what was extracted. Equal digests mean the same facts | TEXT |
+| `fact_run_digest` | sha256 over the identity-recipe version, every non-volatile header field, and `observations_digest`. **The dedup key for fact runs** — see *Fact-run identity* above | TEXT |
+| `source_hash` | Optional checksum for deduplicating repeated submissions of the same job. Predates `fact_run_digest` and is not the dedup key; dedup on `fact_run_digest` | TEXT |
+| `prompt_fingerprint` | *(fact identity v2 — nachos-ai-poc-3#378, POC-3 ADR 0027)* sha256 over every rendered prompt batch of the pair — the prompt bytes that produced the observations. Nullable: NULL/absent on artifacts extracted before the field existed and where a cache replay could not prove the answers match the current prompt (absent means "not established", never "unchanged"); omitted on deterministic-fact runs (no prompt). Non-volatile, so it rides inside `fact_run_digest`; consumers land it as provenance and never recompute it. | TEXT |
+| `answered_by_host` | *(fact identity v2 — nachos-ai-poc-3#381, POC-3 ADR 0027)* Transport identity of the endpoint that served the run: `anthropic` (direct Messages API), `azure:<resource-host>/<deployment>`, `deterministic`, `cache:<host>` (every batch replayed and every entry names the same producing host), `cache` (every batch replayed but the producer isn't recorded or the entries disagree), `mixed:<host>` (some batches replayed and some answered live, every recorded producer being that same host), bare `mixed` (a replayed entry names a *different* producer, so naming the live host would wrongly credit it — split on `:` to read the host; bare `mixed` yields none), or `unknown` on artifacts migrated from before the field. The `cache:`/`mixed:` producer tags were added by nachos-ai-poc-3#392 and refined by #394; the prefixed host is the recorded producer of the replayed answers, never an inference. Records what `model` cannot — an Azure-hosted run pins the production cache namespace yet still reports the base `model` id. Nullable; non-volatile, so it rides inside `fact_run_digest`. | TEXT |
 | `record_count` | Number of observation rows in the run | INTEGER |
 | `scored_count` | Number of rows scored in the run | INTEGER |
 | `skipped_count` | Number of rows skipped in the run | INTEGER |
@@ -532,7 +576,7 @@ load, with `assessor_type = 'human'`).
 | `scored_at` | Timestamp when the scoring pipeline produced the result set | TIMESTAMPTZ |
 | `status` | Run status, e.g. `complete` | TEXT |
 | `scoring_plan_version` | Methodology version that produced this run's scores | TEXT |
-| `contract_version` | Envelope-shape version of the emitted records (`SIDECAR_CONTRACT_VERSION`, currently `3`) | TEXT |
+| `contract_version` | Envelope-shape version of the emitted records (`SIDECAR_CONTRACT_VERSION`, currently `4`) | TEXT |
 | `snapshot_id` | Ingestion snapshot this run scored — `{artifact stem}@{digest[:16]}`, already emitted in the sidecar header (ADR 0018). The PRD's `state_requirements_tag`. | TEXT |
 | `snapshot_digest` | sha256 canonical-form digest of that snapshot — proves the content, not just the edition. Already emitted. | TEXT |
 | `release_id` | Deterministic hash over the full run identity — the landing zone's idempotency key. Already emitted. | TEXT |
@@ -599,6 +643,7 @@ together — not from `fact_observations` alone.
 | `tier_name` | *(emitted since sidecar contract v3 — nachos-ai-poc-3#318, POC-3 ADR 0020)* Name of the rule-cascade tier that matched (the cascade token, e.g. `tier_3_aggregation`); NULL when the dimension could not evaluate. Display labels and thresholds for it live in `score_labels`, keyed by `scoring_plan_version` and joined on this token. | TEXT |
 | `confidence_composite` | Minimum confidence across all dimensions: `high`, `medium`, or `low` | TEXT |
 | `adjusted_nachos_score` | Final NACHOS score after extension and multi-entity adjustments, capped at 4.5 | DOUBLE PRECISION |
+| `base_nachos_score` | Raw NACHOS complexity tier the scoring rules assigned this element *before* extension/multi-entity adjustment: 0–3. Landing-derived from `dimensions.nachos_score.value` (the value the rule cascade in `tier_name` produced), the same way `documentation_gap` is surfaced from the `dimensions` payload. The base half of the base/adjusted headline; `adjusted_nachos_score` is the adjusted half. Distinct from `complexity_score`: the two coincide on most rows but diverge where a rule (e.g. concatenation) sets the NACHOS tier independently of the business-logic complexity tier. | INTEGER |
 | `in_scope` | Whether this element is in scope for NACHOS methodology scoring (the value the run consumed; classification originates on the element record) | BOOLEAN |
 | `documentation_style` | *(payload field; typed-column materialization is MC-6's call)* `prescriptive`, `conceptual`, `cross_reference`, `regulatory`, or `unspecified` — the extracted fact behind the `documentation_style_tier` dimension. (The PRD calls this business_logic_complexity and its first value "Conditional"; the pipeline's fact name and tokens are used here.) | TEXT |
 | `adjustment_drivers` | *(emitted since sidecar contract v3 — nachos-ai-poc-3#318, POC-3 ADR 0020)* Array of the pipeline's canonical adjustment tokens: `unnecessary_ext`, `necessary_ext`, `multi_entity`, `fidelity_divergent_explained`, `fidelity_divergent_unclear` — built at the same rule sites as the rendered labels (no string parsing); the byte-pinned substrings of `nachos_justification` remain for display. | JSONB |
@@ -608,6 +653,7 @@ together — not from `fact_observations` alone.
 | `nachos_justification` | Human-readable rule label and adjustment breakdown string | TEXT |
 | `discovery_lens` | Provenance: `source` for source-doc rows, `spine_anchored` for gap-recovered rows | TEXT |
 | `documentation_source` | Provenance label: `source_doc`, `swagger`, or `swagger_leaf` | TEXT |
+| `documented` | *(emitted since sidecar contract v4 — nachos-ai-poc-3#364, POC-3 ADR 0025)* Whether the state source explicitly documents this element — the headline predicate. Aggregate computes every reported mean, dimension distribution, and review count over the rows where this is true (the run header's `in_scope_count` is their count), so carrying it per-record lets a consumer reproduce the headline population from the score envelope alone, without a cross-artifact join. Distinct from `documentation_source`, which discriminates only on the source lens (every API-model-lens row is `source_doc`) and so cannot stand in for it. | BOOLEAN |
 | `dimensions` | Per-dimension score objects (value, rule_matched, inputs_used, confidence). The full firing-rule path for a row is the per-dimension `rule_matched` entries plus `nachos_justification`; it stays inspectable without re-running a model. | JSONB |
 | `fact_provenance` | Per-fact audit trail (value, confidence, downgraded, downgrade_reason, spans). When an analyst fact correction replaced the extracted value at aggregate time, the entry additionally carries `provenance = 'human_corrected'` with confidence forced to `high`, downgrade flags cleared, and spans dropped; the key is emitted only when set, so uncorrected facts are byte-identical to before. Absent `provenance` means model-extracted. NULL on `legacy_import` rows. | JSONB |
 | `review` | Review block (needs_review, reasons, route) | JSONB |
@@ -852,26 +898,25 @@ to physical tables.
 
 ## The CI contract test
 
-The companion `schemas/pipeline-records.schema.json` (JSON Schema, draft 2020-12)
-defines every record shape above under `$defs`. POC-3 already commits its own
+The companion pipeline-records schema (JSON Schema, draft 2020-12) defines
+every record shape above under `$defs`. POC-3 already commits its own
 contract artifact for the score sidecar — `docs/contracts/assessment-release.schema.json`,
 generated from `score/release_contract.py`, with an envelope/payload split —
 and states that the Metadata Catalog CI validates emits against it. The two
-must not drift: for score records this file defers to the release contract
-(same envelope roster), and adds what it does not cover — element records,
-the ingestion run ledger, fact artifacts, curation sidecars, and gap logs.
-This file is generated from POC-3's `publish/pipeline_records_contract.py` and
-committed **byte-identical** in both repos — POC-3's copy at
-`docs/contracts/pipeline-records.schema.json` is validated against every
-artifact family on every publish run (POC-3 ADR 0021). Emit cores state what
-the pipeline emits; the landed `*_row` variants derive from them by
-construction — identical emit-field constraints plus the landing-generated
-additions (surrogate keys, import audit stamps, denormalized snapshot
-identity) the pipeline can never emit — so emit/landed alignment is
-mechanical, not a change-flow discipline. The CI contract test (MC-18)
-enforces the byte identity on this side. The contract test validates a
-sample of pipeline-emitted records against it on every build, and the
-transform/DDL review checks the schema file against what the database extracts.
+must not drift: for score records the pipeline-records schema defers to the
+release contract (same envelope roster), and adds what it does not cover —
+element records, the ingestion run ledger, fact artifacts, curation sidecars,
+and gap logs. It is generated from POC-3's `publish/pipeline_records_contract.py`
+and validated there against every artifact family on every publish run (POC-3
+ADR 0021); it lands in this repo with the POC-3 migration, which settles the
+path it lives at. Emit cores state what the pipeline emits; the landed `*_row`
+variants derive from them by construction — identical emit-field constraints
+plus the landing-generated additions (surrogate keys, import audit stamps,
+denormalized snapshot identity) the pipeline can never emit — so emit/landed
+alignment is mechanical, not a change-flow discipline. The CI contract test
+(MC-18) validates a sample of pipeline-emitted records against the schema on
+every build, and the transform/DDL review checks the schema against what the
+database extracts.
 A field added to the emit without a schema update — or a schema update without
 a doc update — fails the build. Under the JSONB-landing delivery model the
 landing zone accepts any payload, so this check is the only place drift fails