DANDI Quality Passport (DQP) — reference implementation

A structured, interpretable description of reuse-readiness for a Dandiset. Not a prestige score, not a judgement of scientific importance: an estimate of whether an external researcher can discover, load, interpret, trust, and reuse the dataset without hidden failure modes.

This repository is a working skeleton, not a prototype to be thrown away. The core engine, aggregation semantics, gate logic, registry, QC-evidence layer, report schema, and test suite are complete and exercised. What is stubbed is deliberately scoped: the NWB/DANDI backend (src/dqp/sources/nwb.py) and the population of metric packs.

pip install -e .
dqp assess examples/dandiset_ecephys.json
dqp assess examples/dandiset_ecephys.json --json > passport.json
dqp metrics --table          # generates the metric table for the RFC

Output:

DANDI Quality Passport -- 000999
  3 session(s), 2 subject(s); packs: behavior, ecephys
  GATES: FAILED
    [GATE_SYNC_UNDOCUMENTED] Multiple streams present with no documented
    synchronisation; alignment cannot be trusted. (1 target(s))
  Dimensions (0-4, coverage in parentheses):
    metadata        3.00  (100% covered)
    structure       2.90  (100% covered)
    provenance      2.71  (100% covered)
    raw_integrity   1.22  (100% covered)
    processed       2.21  (100% covered)
    usability       3.10  (100% covered)
  Flags (17):
    [blocker] No sync signal, shared timebase, or sync description. x1
    [warn] Spike-sorting quality: units present with no quality metrics
           (amplitude_cutoff, isi_violation, isolation_distance, ...). x1
    [warn] Sorted units present but the sorter is not identified. x1
    ...
  Remediation queue (missing QC evidence, and how to close it):
    [automated] Spike-sorting quality: amplitude_cutoff, isi_violation, ...
                via spikeinterface/phy (2 session(s))

Pipeline

Metrics populate a grid; the grid is read three ways. See docs/HOW_IT_WORKS.md.

Documentation

Doc	What it's for	Read it if
Contents	annotated file listing, verification commands	you want to know what's in here
Handoff	the engineering brief	you are building on this — start here
How it works	the pipeline, end to end	you have 5 minutes and want the shape
Architecture	design decisions, open policy questions	you're about to change something structural
Adding a metric	the contributor path + checklist	you're writing a metric or a QC family
RFC	the concept doc: proposal, positions, decisions requested	you want the why, or you're reviewing the proposal

Layout

src/dqp/
  model.py          core types: Dimension, Evidence, MetricResult, Gate, Flag
  qc.py             QC-evidence layer — read before touching a pack
  registry.py       @metric decorator — the extension point
  aggregate.py      session -> subject -> dandiset, with spread and coverage
  runner.py         orchestration, gate evaluation, dimension roll-up
  report.py         Passport object + the JSON contract
  cli.py            assess / metrics / schema
  sources/
    base.py         the protocols a backend must satisfy
    fixture.py      in-memory backend (JSON) — what the tests run against
    nwb.py          STUB: the real backend. Start here.
  metrics/
    shared/         cross-modality metrics
    packs/          modality packs (ecephys, ophys, behavior)
tests/              43 tests, run in <1s, no network
examples/           a fixture Dandiset with realistic problems
docs/               see the Documentation table above
  figures/          dqp-pipeline.svg, qc-three-questions.svg

Six invariants — don't break these

1. DQP assesses QC evidence; it never computes domain QC. Structural and integrity properties (timestamps, shapes, schema validity) are computed here. Unit isolation, motion residual, neuropil contamination, and pose confidence are not — those belong to SpikeInterface, suite2p, CaImAn, DeepLabCut, which do it better and are versioned by the people who own the methods. A test fails CI on any import of those libraries. The payoff: DQP never reads sample data, so there is no expensive pass and no scheduling problem.
2. Gates are never averaged into scores. A gate says a specific reuse barrier exists. It is reported separately, at the top of the passport, and a gated dataset still gets full dimension scores.
3. NOT_APPLICABLE ≠ NOT_ASSESSED ≠ score of 0. Never impute a missing measurement as a failure. This is the single mechanism protecting legacy datasets and unusual experimental designs from being scored down for being unusual, and it is the thing most quality frameworks get wrong.
4. Aggregates carry spread and counts, never just a central value. One clean session must not make a heterogeneous Dandiset look uniform.
5. Every metric declares its evidence type — computed, declared, or community. These are never collapsed. "Quality" here is a mixture of measurement, depositor judgement, and observed reuse; hiding which is which fakes an objectivity the framework does not have.
6. No overall score, no medal tier, is emitted by the engine. If the product later wants headline labels, derive them in the presentation layer from dimensions + gates, where the derivation is visible and arguable.

The scale

Metrics return 0–4, not 0–100: 0 absent/broken · 1 minimal · 2 usable with major caveats · 3 good · 4 excellent. A score of 81.3 implies a precision the underlying signals do not support. Dimension roll-ups are weighted means and are reported to two decimals; that is a display choice, not a claim.

The QC-evidence layer

For each family of derived output — sorted units, ROI segmentation, motion correction, pose estimation, behavioural annotation — DQP asks three separate questions:

Question	For sorted units	Dimension
Coverage — which QC metrics are present?	is isi_violation, snr, presence_ratio … reported?	processed
Provenance — how were they computed?	which tool, version, refractory window?	provenance
Standing — what do the values say?	what fraction of units clear conventional thresholds?	processed

QC provenance is distinct from pipeline provenance: Kilosort produced the units, SpikeInterface measured them, and those are different records. An isi_violation column with no tool recorded is barely more interpretable than no column, because it isn't comparable to the same-named number elsewhere.

When a family is absent, the passport emits a remediation queue item rather than only a low score, naming the pathway (automated — a standard pipeline can compute it from what's deposited; depositor — only the experimenter has what's needed; needs_raw) and the conventional tools. Automated gaps can be batched into a compute job; depositor gaps route into curation correspondence.

What to build next, in order

1. sources/nwb.py — the only real blocker. Everything else runs today. Wrap nwbinspector and dandi validate rather than reimplementing them; stream rather than download; cache on (asset_id, checksum, metric_version).
2. Verify the QC readers against real deposits. qc.py accepts three shapes for QC values (per-object arrays, precomputed n_pass/n_total, bare presence). Real Dandisets will have shapes we haven't seen — extend _pass_fraction rather than special-casing inside packs.
3. The retrospective. Run over ~50 real Dandisets and look at the distribution before anyone argues about thresholds. Expect to move most of the cut points afterwards; that is the point of doing it.
4. Depositor-facing surface before public surface. See docs/ARCHITECTURE.md § Rollout.