Every sample's uniques are merged into a single table, the shared core runs once on it, and the resulting global partition is then projected back into per-sample outputs. Abundance evidence therefore sums across samples: a variant seen once in each of forty samples arrives at the denoiser with abundance forty.
Source: main.rs:1214
flowchart TD
IN["--input sample_1 … sample_n"]
IN --> LOOP
subgraph LOOP["streaming derep + merge, sample index order"]
direction TB
LD["load_derep_for_dada(sample i)"]
FOLD["fold into merged accumulator<br/>seq → merged index · sum counts · sum Phred sums"]
REC["record local_to_merged[i]<br/>+ sample_unique_counts[i]"]
DROP["drop this sample's derep"]
LD --> FOLD --> REC --> DROP
end
LOOP --> RI["build merged RawInput list<br/>(moves the merge vecs, frees intermediates)"]
RI --> PR["mark_priors (only if --prior given)"]
PR --> CORE["dada_uniques — ONCE<br/>THE SHARED CORE"]
CORE --> PROJ
subgraph PROJ["per-sample projection of the global partition"]
direction TB
SUM["sum this sample's reads per global cluster<br/>via local_to_merged + result.map"]
FILT["drop clusters with 0 reads here"]
REN["renumber global → local cluster ids"]
MAP["local unique → local cluster map"]
SUM --> FILT --> REN --> MAP
end
PROJ --> O1["one JSON per sample"]
CORE --> O2["--pooled-record (optional)<br/>global uniques + global map + global ASVs"]
CORE --> O3["--failed-uniques (optional)<br/>global map == None, rows per sample"]
click CORE "../algorithm-core/" "The shared denoising core"
The dada_uniques node is The shared denoising core — the
same seven wrapper steps and the same A–H loop as every other mode, just handed a
much larger table.
--sample-names, if given, must have exactly one entry per input file. The output
directory is created, and a single rayon pool of --threads is built — pooled mode
has one core invocation, so there is no sample-level fan-out to split threads
between.
main.rs:1314 — one loop, one sample at a time, in input order.
Each sample is loaded, folded into the merged accumulator, and then dropped before the next is read. The accumulator holds:
| Structure | Contents |
|---|---|
seq_to_merged |
sequence → merged unique index (dropped after the loop) |
merged_seqs |
the merged unique sequences |
merged_qual_sum |
per-position Phred sums across samples, as u32 |
merged_total |
pooled abundance per merged unique |
local_to_merged[s] |
sample s's local unique index → merged index |
sample_unique_counts[s] |
sample s's local read counts |
!!! note "Why streaming, and why order matters" Holding every sample's derep through the merge used to be the pooled memory peak; loading one at a time removed that (issue 41). Folding must be serial anyway, and derep is a small fraction of pooled wall time, so serializing the load is cheap. Fold order is sample index order, exactly as the old parallel path produced, so the merged table — and every output byte — is unchanged.
The per-position quality sum is the merge-able quantity: a sample's derep
already stores sums, so merging is addition. A checked_add guards the
astronomically unlikely u32 overflow rather than wrapping silently.
The merge vectors are moved into RawInputs, not cloned, so each intermediate
is freed as the list is built instead of sitting resident through the core call
(issue 39). Quals are already sums, so no conversion happens. Empty input across
all files is an error here.
Same mark_priors as the other modes, applied to merged uniques. Pooled runs
usually omit --prior; when present, flagged uniques are tested against
OMEGA_P in the core's step E.
dada_uniques runs a single time on the merged table. Two consequences worth
holding onto:
!!! important "nraw is the merged count"
The Bonferroni divisor for the abundance p-value in the core's step E is the
number of merged uniques, not a per-sample count. Pooling therefore
raises the bar for significance at the same time as it raises abundance —
the sensitivity gain is the net of those two effects, not the first alone.
!!! warning "Per-sample outputs are not independent results" Every per-sample JSON is a projection of one global partition. Adding or removing a sample can change the ASVs reported for the others.
For each sample: walk its local uniques through local_to_merged and
result.map, summing that sample's read counts into per-global-cluster totals;
drop clusters with zero reads in this sample; renumber the surviving global
cluster ids to dense local ids; and build the local unique → local cluster map.
The output shape matches single-sample dada, which is what
lets downstream commands treat both identically.
--pooled-record main.rs:1632 writes one self-contained record with
the merged uniques at their pooled abundance, the global map, and the global
ASVs. It exists because a pool-level analysis must not re-aggregate the per-sample
projections — doing that double-counts sequences shared between samples and
re-derives the failed-singleton split from local counts. This is the input
kdist-calibrate --from-dada-pooled expects. Off by default, and written to an
explicit path rather than --output-dir so it never lands in the per-sample
*.json.gz glob downstream steps rely on; gzip follows the path's own .gz
extension, not --gzip.
--failed-uniques treats failure as a global property (result.map[mu] is
None), emitting one row per sample the failed merged unique appears in, carrying
that sample's read count.
main.rs:1692 reports derep, merge, run_dada and output wall times
with percentages, labelled serial or parallel. Only run_dada is parallel, so the
other three bound achievable speedup — see Results. Peak RSS is
reported after merge and after the core call, the two points that define the
pooled memory profile.