Add wafer comparison and stream analysis scans

CHANGELOG.md

@@ -58,10 +58,14 @@ Shared retained-evidence adapter also ships:
 
 - `site-variation-request-v1` / `prr-site-variation-v1`
 - `lot-comparison-request-v1` / `prr-lot-comparison-v1`
+- `wafer-comparison-request-v1` / `prr-wafer-comparison-v1`
 - `retest-impact-request-v1` / `prr-retest-impact-v1`
 
 Public entry: `zstdf-cli snapshot analyze <dataset> <request.json>` (attempt-yield
-remains available). Dashboard `coordinate-v1` merging is unchanged.
+remains available). Dashboard `coordinate-v1` merging is unchanged. Absent and
+empty wafer IDs stay distinct (`__missing__` / `__empty__`). Pareto remains
+deferred. Analysis scans stream per evidence source after inventory collection
+so corrupt indexes still fail before a complete receipt.
 
 ### Run telemetry
 

docs/architecture/data-contracts.md

@@ -169,13 +169,18 @@ distinct request/result versions:
 
 - `site-variation-request-v1` → per `(head, site)` yield; unresolved visible
 - `lot-comparison-request-v1` → per `lot_id` yield; unresolved visible
+- `wafer-comparison-request-v1` → per wafer yield; absent wafer groups as
+  `__missing__`, empty-string wafer as `__empty__` (kept distinct); unresolved
+  visible
 - `retest-impact-request-v1` → per-source ECID first/last attempt impact,
   recovery/degradation among known verdicts; unresolved identity counted
   separately; `latest_device` keeps full history only for ECIDs with a selected
   latest flag
 
-Pareto remains deferred. Site differences alone do not prove hardware causality
-or GR&R; qualification/reliability decisions need study metadata.
+Pareto / FTR pattern-attempt population remains deferred (needs execution
+evidence beyond PRR companions). Site or wafer differences alone do not prove
+hardware causality or GR&R; qualification/reliability decisions need study
+metadata.
 
 ## Receipt and reproducibility
 

docs/architecture/implementation-backlog.md

@@ -85,7 +85,7 @@ or installer evidence. Do not alter current CI merely to hide extension failures
 
 | Priority / issue | Scope | Gate |
 | --- | --- | --- |
-| P1 / ANA-002 | Latest yield, lot comparison, site variation, retest impact and Pareto | Latest yield + site/lot/retest shipped over retained evidence; Pareto still deferred; Dashboard merge untouched |
+| P1 / ANA-002 | Latest yield, lot comparison, site variation, wafer comparison, retest impact and Pareto | Latest yield + site/lot/wafer/retest shipped over retained evidence; Pareto / FTR pattern population still deferred; Dashboard merge untouched |
 | P1 / API-001 | Read-only typed local tool adapter | Schema, authorization, cancellation, payload limits and denied operations tested; no LLM needed |
 | Optional / SQL-001 | Embedded DuckDB query adapter | Pinned version, fixed operations, restricted object access, resource tests, semantic parity; no install required for ordinary users |
 | Optional / AI-001 | Local or opted-in cloud explanation adapter | Authorized data scope, no arbitrary tool capability, provenance-grounded output, prompt-injection tests |

docs/architecture/se-matrix-reassessment.md

@@ -1,8 +1,7 @@
 # SE matrix reassessment (post low-score fixes)
 
-Status: evidence-based rescore after Waves 1–4 remainders plus adversarial
-corner-case adapter tests on branch `feature/corner-case-tests-and-rescore`
-(baseline main includes `#1`–`#5`).
+Status: evidence-based rescore after TDD residuals on branch
+`feature/se-tdd-residuals` (baseline main includes `#1`–`#6`).
 
 Companion canvas:
 `canvases/stdf-se-matrix-reassessment.canvas.tsx` (Cursor project canvases).
@@ -11,37 +10,39 @@ Companion canvas:
 
 Same 1–5 rubric as the prior review. Scores move only when shipped code/tests/docs
 changed the evidence. Dashboard `coordinate-v1` yield math was not modified.
-Pareto ANA-002 and dashboard SpillStore rewrite remain open (documented skip).
-Identity policies stay distinct (attempt-yield / latest-device vs Dashboard merge).
+Pareto ANA-002 and dashboard SpillStore rewrite remain open (documented skip with
+spike evidence). Identity policies stay distinct (attempt-yield / latest-device vs
+Dashboard merge).
 
 ## Before → prior → latest
 
-| Dimension | Original | Prior (~Waves 1–4) | Latest | Why (latest delta) |
+| Dimension | Original | Prior (~corner) | Latest | Why (latest delta) |
 | --- | ---: | ---: | ---: | --- |
 | Layering & modularity | 4.0 | 4.0 | 4.0 | Shared reducers in `stdf-analytics`; CLI still I/O host |
 | Cohesion of concerns | 3.0 | 3.5 | 3.5 | `summary_ops` + `attempt_yield_op` host analytics |
-| Extensibility seams | 3.0 | 3.5 | 3.5 | Versioned request dispatch for site/lot/retest/yield |
-| Domain fidelity | 4.5 | 4.5 | 4.5 | Tied/missing START_T + unresolved overlap asserted on adapter |
-| Schema & versioning | 4.0 | 4.5 | 4.5 | `analysis-receipt-v1` + ANA-002 result versions |
-| Provenance & reproducibility | 3.5 | 4.0 | 4.5 | `result_hash` stable under reorder; ANA-002 receipt replay |
-| Reliability & failure model | 3.5 | 4.0 | 4.5 | Mid-scan cancel inject + corrupt ANA-002 empty-scope fail |
-| Performance engineering | 3.5 | 3.5 | 3.5 | SpillStore not wired into dashboard (documented skip) |
+| Extensibility seams | 3.0 | 3.5 | 4.0 | Wafer comparison joins versioned analyze dispatch |
+| Domain fidelity | 4.5 | 4.5 | 4.5 | Missing vs empty wafer labels kept distinct |
+| Schema & versioning | 4.0 | 4.5 | 4.5 | `wafer-comparison-*-v1` + existing ANA-002 versions |
+| Provenance & reproducibility | 3.5 | 4.5 | 4.5 | Wafer receipt replay stable |
+| Reliability & failure model | 3.5 | 4.5 | 4.5 | Stream scan still fails corrupt empty-scope evidence |
+| Performance engineering | 3.5 | 3.5 | 4.0 | Per-source stream reduce; no multi-source attempt Vec |
 | Trust boundary | 3.5 | 4.0 | 4.0 | Path-traversal denial tests retained |
 | Observability | 2.0 | 3.0 | 3.0 | `run-telemetry-v1` on convert/view/analyze |
 | Operability & portability | 3.0 | 3.5 | 3.5 | Linux CI job (Rust/clippy/test; no Edge smoke) |
-| Test strategy | 4.0 | 4.0 | 4.5 | 12 adversarial CLI corner cases over shipped adapter |
+| Test strategy | 4.0 | 4.5 | 4.5 | Wafer + streaming corrupt fixtures; goldens held |
 | CI & automation | 3.0 | 3.5 | 3.5 | Clippy -D warnings + Linux job + cargo-deny |
 | Process & governance | 4.5 | 4.5 | 4.5 | Unchanged multi-agent / review policy |
-| Maintainability | 3.0 | 3.5 | 4.0 | Analysis adapter + dashboard HTML/JSON render split (`#5`) |
-| Analytics completeness | 2.5 | 3.5 | 3.5 | Latest yield + site/lot/retest; Pareto still deferred |
+| Maintainability | 3.0 | 4.0 | 4.0 | Adapter + dashboard render split retained |
+| Analytics completeness | 2.5 | 3.5 | 4.0 | Site/lot/wafer/retest + latest; Pareto still deferred |
 | API / release stability | 3.0 | 3.5 | 3.5 | Public `snapshot analyze` + CHANGELOG contracts |
 
-Approximate weighted maturity: **~3.3 → ~3.8 → ~4.0** (same weights as prior canvas).
+Approximate weighted maturity: **~3.3 → ~4.0 → ~4.1** (same weights as prior canvas).
 
 ## Residuals
 
-1. Pareto / full 10F step analytics still deferred.
-2. Dashboard SpillStore rewrite skipped to protect yield math.
+1. Pareto / FTR pattern-attempt population still deferred (needs execution evidence).
+2. Dashboard SpillStore rewrite skipped again after spike: SpillStore is an
+   external-sort store, not a map backend; no bitwise KPI golden suite yet.
 3. Linux CI omits Playwright/Edge viewer smoke by design.
 4. Loopback-only multi-user trust model unchanged.
 5. Independent semantic review of this delta still required before calling the
@@ -51,12 +52,11 @@ Approximate weighted maturity: **~3.3 → ~3.8 → ~4.0** (same weights as prior
 
 ```text
 cargo test -p stdf-analytics --locked -- summary_ops attempt_yield
-cargo test -p stdf-cli --locked -- attempt_yield analysis_receipt latest_device site_variation lot_comparison retest_impact analysis_path
+cargo test -p stdf-cli --locked -- attempt_yield analysis_receipt latest_device site_variation lot_comparison wafer_comparison retest_impact streaming_scan analysis_path
 cargo fmt --all
 cargo clippy -p stdf-analytics -p stdf-cli --locked -- -D warnings
 ```
 
-Corner cases added in this pass (CLI `analysis_tests.rs`): absent vs empty scope
-`result_hash`, unresolved pass/fail/unknown overlap, tied/missing START_T latest,
-reorder-stable `result_hash`, mid-scan cancel, site scope/excludes + zero-denom +
-corrupt empty scope, multi-lot comparison, lot `latest_device`, retest receipt replay.
+This pass added: `wafer-comparison-request-v1` reducer + CLI adapter coverage,
+streaming inventory→scan reduce (inventory parity enforced), and documented
+Dashboard spill skip spike evidence.

docs/cli.md

@@ -153,6 +153,7 @@ versions and publish the same `analysis-receipt-v1` layout. Dispatch via
 | --- | --- | --- |
 | `site-variation-request-v1` | `site-variation-result-v1` / `prr-site-variation-v1` | Yield by `H{head} / S{site}` |
 | `lot-comparison-request-v1` | `lot-comparison-result-v1` / `prr-lot-comparison-v1` | Yield by `lot_id` |
+| `wafer-comparison-request-v1` | `wafer-comparison-result-v1` / `prr-wafer-comparison-v1` | Yield by wafer; `__missing__` / `__empty__` reserved |
 | `retest-impact-request-v1` | `retest-impact-result-v1` / `prr-retest-impact-v1` | Per-source ECID first/last; unresolved identity counted separately |
 
 Populations remain `all_attempts` and `latest_device` (with
@@ -161,7 +162,7 @@ Populations remain `all_attempts` and `latest_device` (with
 
 ```powershell
 .\target\release\zstdf-cli.exe snapshot analyze path\to\dataset site-request.json
-cargo test -p stdf-cli site_variation lot_comparison retest_impact --locked
+cargo test -p stdf-cli site_variation lot_comparison wafer_comparison retest_impact --locked
 ```
 
 ## Multi-file Conversion (Phase 10B)

docs/performance_baseline.md

@@ -26,10 +26,14 @@ nonzero. Analysis callers can pass the same helper around `analysis::execute`.
 ## Dashboard spill rewrite (skipped)
 
 `stdf-analytics::SpillStore` remains available for future disk-backed dashboard
-aggregations. Wiring it into `stdf-cli/src/dashboard.rs` in-memory maps was
-skipped because changing aggregation order or intermediate materialization
-risks Dashboard `coordinate-v1` yield math. Keep current in-memory KPIs until a
-separate experiment proves bitwise-identical yields under spill.
+aggregations. A 2026-11 residual spike reconfirmed the skip: Dashboard KPI /
+wafer / site / bin yields are computed from in-memory `BTreeMap` part merges
+(`coordinate-v1`), while `SpillStore` is an external-sort record store—not a
+drop-in map backend. There is no locked bitwise-identical KPI golden suite that
+would prove yield math under a rewrite. Keep current in-memory KPIs until a
+separate experiment adds golden KPI fixtures and proves identical yields under
+spill. Analysis adapters (attempt-yield / ANA-002) already stream per-source
+attempt scans and do not depend on Dashboard spill.
 
 Run:
 

stdf-analytics/src/summary_ops.rs

@@ -1,5 +1,5 @@
 // Copyright 2026 zstdf contributors. SPDX-License-Identifier: Apache-2.0
-//! ANA-002 site / lot / retest summaries over physical PRR attempts.
+//! ANA-002 site / lot / wafer / retest summaries over physical PRR attempts.
 //!
 //! These reducers reuse the attempt-yield request envelope and
 //! [`YieldAccumulator`] counting rules. Dashboard `coordinate-v1` merging is
@@ -23,6 +23,14 @@ pub const LOT_REQUEST_VERSION: &str = "lot-comparison-request-v1";
 pub const LOT_RESULT_VERSION: &str = "lot-comparison-result-v1";
 pub const LOT_OPERATION: &str = "prr-lot-comparison-v1";
 
+pub const WAFER_REQUEST_VERSION: &str = "wafer-comparison-request-v1";
+pub const WAFER_RESULT_VERSION: &str = "wafer-comparison-result-v1";
+pub const WAFER_OPERATION: &str = "prr-wafer-comparison-v1";
+/// Group label when retained attempt wafer is absent (`None`).
+pub const WAFER_MISSING_LABEL: &str = "__missing__";
+/// Group label when retained attempt wafer is present but empty.
+pub const WAFER_EMPTY_LABEL: &str = "__empty__";
+
 pub const RETEST_REQUEST_VERSION: &str = "retest-impact-request-v1";
 pub const RETEST_RESULT_VERSION: &str = "retest-impact-result-v1";
 pub const RETEST_OPERATION: &str = "prr-retest-impact-v1";
@@ -31,6 +39,7 @@ pub const RETEST_OPERATION: &str = "prr-retest-impact-v1";
 pub enum SummaryKind {
     SiteVariation,
     LotComparison,
+    WaferComparison,
     RetestImpact,
 }
 
@@ -39,6 +48,7 @@ impl SummaryKind {
         match self {
             Self::SiteVariation => SITE_REQUEST_VERSION,
             Self::LotComparison => LOT_REQUEST_VERSION,
+            Self::WaferComparison => WAFER_REQUEST_VERSION,
             Self::RetestImpact => RETEST_REQUEST_VERSION,
         }
     }
@@ -47,6 +57,7 @@ impl SummaryKind {
         match self {
             Self::SiteVariation => SITE_RESULT_VERSION,
             Self::LotComparison => LOT_RESULT_VERSION,
+            Self::WaferComparison => WAFER_RESULT_VERSION,
             Self::RetestImpact => RETEST_RESULT_VERSION,
         }
     }
@@ -55,6 +66,7 @@ impl SummaryKind {
         match self {
             Self::SiteVariation => SITE_OPERATION,
             Self::LotComparison => LOT_OPERATION,
+            Self::WaferComparison => WAFER_OPERATION,
             Self::RetestImpact => RETEST_OPERATION,
         }
     }
@@ -64,6 +76,7 @@ pub fn kind_of_version(version: &str) -> Option<SummaryKind> {
     match version {
         SITE_REQUEST_VERSION => Some(SummaryKind::SiteVariation),
         LOT_REQUEST_VERSION => Some(SummaryKind::LotComparison),
+        WAFER_REQUEST_VERSION => Some(SummaryKind::WaferComparison),
         RETEST_REQUEST_VERSION => Some(SummaryKind::RetestImpact),
         _ => None,
     }
@@ -130,6 +143,14 @@ fn lot_key(row: &AttemptRecord) -> String {
     row.lot.clone()
 }
 
+fn wafer_key(row: &AttemptRecord) -> String {
+    match &row.wafer {
+        None => WAFER_MISSING_LABEL.to_owned(),
+        Some(wafer) if wafer.is_empty() => WAFER_EMPTY_LABEL.to_owned(),
+        Some(wafer) => wafer.clone(),
+    }
+}
+
 fn disposition(row: &AttemptRecord, excludes: Option<&HashSet<u64>>) -> AttemptDisposition {
     AttemptDisposition {
         verdict: Verdict::from_prr_flags(row.flags),
@@ -138,7 +159,7 @@ fn disposition(row: &AttemptRecord, excludes: Option<&HashSet<u64>>) -> AttemptD
     }
 }
 
-/// Streaming site or lot aggregation. Caller owns cancel checks between rows.
+/// Streaming site / lot / wafer aggregation. Caller owns cancel checks between rows.
 pub fn accumulate_groups(
     kind: SummaryKind,
     rows: impl IntoIterator<Item = AttemptRecord>,
@@ -156,6 +177,7 @@ pub fn accumulate_groups(
         let key = match kind {
             SummaryKind::SiteVariation => site_key(&row),
             SummaryKind::LotComparison => lot_key(&row),
+            SummaryKind::WaferComparison => wafer_key(&row),
             SummaryKind::RetestImpact => {
                 return Err(io::Error::new(
                     io::ErrorKind::InvalidInput,
@@ -172,7 +194,9 @@ pub fn accumulate_groups(
 
 fn row_in_summary(kind: SummaryKind, row: &AttemptRecord, population: Population) -> bool {
     match kind {
-        SummaryKind::SiteVariation | SummaryKind::LotComparison => in_population(row, population),
+        SummaryKind::SiteVariation | SummaryKind::LotComparison | SummaryKind::WaferComparison => {
+            in_population(row, population)
+        }
         // Retest needs full attempt history for selected devices; handled separately.
         SummaryKind::RetestImpact => true,
     }
@@ -363,6 +387,7 @@ pub fn build_grouped_result(
     let group_field = match kind {
         SummaryKind::SiteVariation => "sites",
         SummaryKind::LotComparison => "lots",
+        SummaryKind::WaferComparison => "wafers",
         SummaryKind::RetestImpact => "groups",
     };
     let mut items = Vec::with_capacity(groups.len());
@@ -484,12 +509,25 @@ mod tests {
         flags: u8,
         ecid: Option<&str>,
         latest: Option<bool>,
+    ) -> AttemptRecord {
+        row_wafer(attempt, head, site, lot, Some("W1"), flags, ecid, latest)
+    }
+
+    fn row_wafer(
+        attempt: u64,
+        head: u8,
+        site: u8,
+        lot: &str,
+        wafer: Option<&str>,
+        flags: u8,
+        ecid: Option<&str>,
+        latest: Option<bool>,
     ) -> AttemptRecord {
         AttemptRecord {
             attempt,
             run: 1,
             lot: lot.into(),
-            wafer: Some("W1".into()),
+            wafer: wafer.map(str::to_owned),
             head,
             site,
             flags,
@@ -545,6 +583,55 @@ mod tests {
         assert_eq!(groups["L2"].summary().fail, 1);
     }
 
+    #[test]
+    fn wafer_comparison_keeps_missing_and_empty_distinct() {
+        let scope = compile_scope(None, &HashMap::new()).unwrap();
+        // Hand-computed: W1 pass, W2 fail, missing unknown, empty pass.
+        let rows = [
+            row_wafer(1, 1, 1, "L1", Some("W1"), 0, Some("a"), Some(true)),
+            row_wafer(2, 1, 1, "L1", Some("W2"), 8, Some("b"), Some(true)),
+            row_wafer(3, 1, 1, "L1", None, 4, None, None),
+            row_wafer(4, 1, 1, "L1", Some(""), 0, Some("c"), Some(true)),
+        ];
+        let (total, groups) = accumulate_groups(
+            SummaryKind::WaferComparison,
+            rows,
+            &"s".repeat(64),
+            Population::AllAttempts,
+            &scope,
+            None,
+        )
+        .unwrap();
+        let summary = total.summary();
+        assert_eq!(summary.pass, 2);
+        assert_eq!(summary.fail, 1);
+        assert_eq!(summary.unknown, 1);
+        assert_eq!(summary.unresolved, 1);
+        assert_eq!(groups["W1"].summary().pass, 1);
+        assert_eq!(groups["W2"].summary().fail, 1);
+        assert_eq!(groups[WAFER_MISSING_LABEL].summary().unknown, 1);
+        assert_eq!(groups[WAFER_EMPTY_LABEL].summary().pass, 1);
+        let encoded = build_grouped_result(
+            SummaryKind::WaferComparison,
+            Population::AllAttempts,
+            &summary,
+            &summaries_of(&groups),
+            &Coverage {
+                snapshot_sources: 1,
+                selected_sources: 1,
+                missing_source_ids: Vec::new(),
+                partial: false,
+                expected_inventory: "unknown".into(),
+                content_alias_source_count: 0,
+                catalog_run_status: "complete".into(),
+                catalog_failure_count: 0,
+            },
+        );
+        assert_eq!(encoded["version"], WAFER_RESULT_VERSION);
+        assert_eq!(encoded["operation"], WAFER_OPERATION);
+        assert_eq!(encoded["wafers"].as_array().unwrap().len(), 4);
+    }
+
     #[test]
     fn retest_impact_recovery_and_unresolved_are_visible() {
         let scope = compile_scope(None, &HashMap::new()).unwrap();