Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
187 lines (147 loc) · 9.17 KB

ARCHITECTURE.md

File metadata and controls

187 lines (147 loc) · 9.17 KB
Raw

Architecture

Data flow

DQP pipeline

DatasetSource ──┬─> session-level metrics ──> MetricResult (per session)
                │                                   │
                └─> dandiset-level metrics ─────────┤
                                                    ├─> gate evaluation (per result)
                                                    ├─> aggregate: session → subject → dandiset
                                                    └─> dimension roll-up (weighted)
                                                            │
                                                            └─> Passport → JSON

Four layers, each replaceable independently:

Layer Module Replaceable by
Ingestion / inspection sources/ any backend satisfying the protocols in sources/base.py
QC evidence qc.py declare a QCFamily; three metrics follow
Metric engine metrics/, registry.py add or version metrics without touching the runner
Aggregation aggregate.py new rules registered in AGGREGATORS
Presentation report.py, cli.py UI/API consuming the JSON contract

Why the source abstraction exists

Metrics never import pynwb, h5py, or dandi. They talk to SessionSource and DatasetSource. This buys:

  • Testability without data. The full suite runs in under a second with no network. Regression cases for pathological Dandisets are written as JSON describing the shape of the problem, not by pinning a 40 GB asset.
  • Backend swappability. Local file, streamed asset, LINDI index, or a cached metadata snapshot are all the same to a metric.
  • Cost control. The backend decides what is expensive. Combined with the assess-don't-compute boundary below, this means the whole-archive pass reads metadata and tables only — there is no sample-level pass to schedule.

Gates vs. scores

Gates are evaluated against individual results, never against aggregates. Averaging a gate away is precisely the failure mode gates exist to prevent: a Dandiset with one unreadable file out of forty should show a gate on that file, not a slightly-lower structure score.

Gate codes are stable strings (GATE_SYNC_UNDOCUMENTED) so the UI, the depositor dashboard, and support documentation can key off them.

Current gates:

Code Metric Trips when
GATE_MINIMAL_METADATA metadata.required_completeness score ≤ 1
GATE_VALIDATION_FAILED structure.nwb_validation critical validator issues
GATE_UNREADABLE structure.readability asset cannot be opened
GATE_TIMESTAMP_BROKEN integrity.timestamp_continuity non-monotonic time
GATE_SYNC_UNDOCUMENTED integrity.cross_stream_sync multi-stream, no sync evidence

Note what is not gated: documentation, provenance, and raw-signal quality. Documentation gaps are cheap to fix, and gating them would penalise exactly the depositors most likely to fix them. Raw-signal quality is the most gameable class of metric in the framework — a depositor who drops bad sessions improves their distribution — so it informs rather than gates.

The assess-don't-compute boundary

dqp/qc.py is the module to read before touching a metric pack.

DQP computes structural/integrity properties (timestamps, shapes, schema). It does not compute domain QC (isolation metrics, motion residual, pose confidence). A test (test_dqp_computes_no_domain_qc) enforces this by rejecting any import of SpikeInterface, suite2p, CaImAn, DeepLabCut, or SLEAP anywhere in the package.

Three reasons this boundary is load-bearing:

  1. Correctness. Domain tools compute these better and are versioned by people who own the methods.
  2. Cost. DQP never reads sample data, so there is no expensive pass and no scheduling problem. This is the single biggest simplification in the design.
  3. Gaming. DQP cannot be gamed through DQP's own computation choices, because it makes none. It reports what the deposit claims, who computed it, and how the values compare to convention.

The three questions

Every family of derived output gets assessed three ways, and the three land in different dimensions because they are different failures with different fixes:

Question Helper Dimension Failure it catches
Coverage assess_coverage processed QC was never computed
Provenance assess_qc_provenance provenance QC exists but isn't comparable to anyone else's
Standing assess_standing processed QC exists and the values are poor

QC provenance is not pipeline provenance. Kilosort produced the units; SpikeInterface measured them; sorter_provenance and unit_qc_provenance are separate metrics for that reason.

A QCFamily is declared once with its conventional thresholds and their published sources, then drives all three metrics. Adding a modality means declaring families, not writing scoring logic.

Thresholds carry their sources (Hill et al. for ISI violations, the Allen Institute pipeline for amplitude cutoff, suite2p's iscell for accepted fraction) so a reader disagrees with the convention explicitly rather than silently. assess_standing reports the thresholds it used in its evidence dict.

Value shapes

_pass_fraction accepts three shapes, because real deposits vary: per-object arrays ({"values": [...]}), precomputed summaries ({"n_pass": k, "n_total": n}), and bare presence (column exists, values unavailable → NOT_ASSESSED, not a low score). Extend this function when new shapes turn up rather than special-casing inside a pack.

When a family is absent, remediation_gaps emits a work item naming the pathway (automated / depositor / needs_raw) and the conventional tools. The runner aggregates these into the passport's remediation queue.

Gaming

Worth being precise about, because the naive framing ("depositors will optimise for the metric") is only half right. Two classes:

  • Optimisation = improvement. README, software versions, sync description, anomaly notes, linked protocols, parameter capture. A depositor who games these has improved the dataset. This is most of the framework and needs no defence.
  • Optimisation = misrepresentation. Raw-signal statistics, unit counts, dropped-frame rates — all improvable by excluding sessions. These are descriptive, ungated, and reported with spread, so silent exclusion shows up as a suspiciously narrow distribution rather than a high score.

Coverage

Every score is reported with a coverage figure. A dimension score of 3.5 from two of eleven metrics is a different object from 3.5 from all eleven, and the consumer must be able to tell them apart. Coverage counts NOT_ASSESSED and ERROR against the denominator; NOT_APPLICABLE is excluded entirely.

Versioning

  • SCHEMA_VERSION in model.py — the JSON contract. Additive changes fine; removals and renames require a bump.
  • Per-metric version — bump on any change to scoring behaviour. Caches key on it, so a bump means recomputation for that metric only.
  • Metric packs are versioned as a unit by the module they live in and should be releasable independently of the core.

Open questions for the team

These are policy, not code, and the code deliberately does not decide them:

  1. Rollout order. Compute silently over the archive first, then expose to depositors only, then publicly? Launching with most of the archive showing low scores is a political event before it is a technical one, and the ordering is the highest-stakes decision in the project.
  2. Legacy datasets. Assess and mark, or mark not assessed wholesale below some deposit date? The engine supports both; nothing has been chosen.
  3. Headline labels. The engine emits none by design. If they are added, non-ordinal descriptors (provenance-complete, sync-verified, benchmark-ready) survive the "not a prestige score" commitment; medal tiers do not.
  4. Weights. Currently hand-set (2.0 for required metadata, validation, readability, timestamps, sync; 1.5 for the provenance core; 0.5 for environment capture and community signal). These are placeholders. The principled version calibrates them against observed reuse outcomes — see below.
  5. Recompute cadence. On version change? On new asset? Nightly? Interacts directly with cache size. Results key on (asset_id, checksum, metric_version), so a metric edit recomputes that metric only.

The calibration path

The framing is that quality is the probability that an external user can successfully reuse the dataset. Nothing in v0.1 is actually probabilistic — it's a hand-weighted ordinal scheme. But the framing is testable, and the usability.reuse_reports metric is the seed: once reuse outcomes are being collected (successful loads, issue reports, starter notebooks that break, reanalysis-grade citations), the weights stop being a committee decision and become a fitted quantity. Keep the outcome data collectable from day one even if nothing consumes it for a year — it's the difference between a taxonomy people argue about and a predictor that can be checked.