Skip to content

Latest commit

 

History

History
463 lines (372 loc) · 24.2 KB

dada.md

File metadata and controls

463 lines (372 loc) · 24.2 KB

dada

Denoise one or more samples independently — the equivalent of R DADA2's dada(..., pool = FALSE). For pooled and pseudo-pooled inference see dada-pooled and dada-pseudo.

# single sample -> stdout or -o
dada2-rs dada sample.derep.json.gz --error-model err.json -o sample.dada.json

# many samples, processed independently and serially
dada2-rs dada derep/*.json.gz --error-model err.json --output-dir dada/ --threads 24

Each input may be a FASTQ file (dereplicated in memory) or a JSON file produced by derep or sample (.json / .json.gz). Pre-dereplicated input avoids re-reading the FASTQ when iterating on parameters. Output is a JSON object describing the inferred ASVs.

With a single input the result goes to --output / -o (or stdout). With more than one input the samples are processed independently and serially — not pooled — and one {sample}.json per sample is written to --output-dir, which is then required.

Input

--sample-name — sample identifier written to the output JSON. Defaults to the filename stem of the input.

--phred-offset — 33 for Sanger / Illumina 1.8+, 64 for Illumina 1.3–1.7.

--prior — FASTA file of prior sequences (uncompressed or gzipped). Each sequence in the file that matches a dereplicated unique exactly is flagged as a prior, making it immune to the abundance p-value filter. Prior-based splitting uses --omega-p instead of --omega-a.

Error model

--error-model (required) — JSON error model produced by learn-errors, or converted from an R DADA2 learnErrors() .rds with learnerrors_to_dada2rs.R.

--use-err-in — use the model's err_in matrix instead of err_out. err_out (the rates estimated from the final transition counts) is the default and is what R DADA2 uses downstream.

--inherit-err-params — inherit any unspecified algorithm parameters (omega_*, min_*, detect_singletons, band, homo_gap_p, kdist_cutoff, kmer_size, no_kmer_screen) from the error model JSON's params block. Any flag passed explicitly on the CLI still wins.

Without this flag the built-in CLI defaults apply, and a warning is emitted for each CLI value that disagrees with the error model's value. That warning is the point: denoising with parameters that differ from the ones the model was fit under is usually a mistake, and this makes it visible rather than silent.

Note that OMEGA_C is not inherited — see the parameters page.

Denoising

These are R's setDadaOpt() parameters and all default to the R values; the parameters page has the full equivalency table. Omitting a flag means "inherit from the error model if --inherit-err-params is set, otherwise use the R default".

Flag R option Default
--omega-a OMEGA_A 1e-40
--omega-c OMEGA_C 1e-40
--omega-p OMEGA_P 1e-4
--min-fold MIN_FOLD 1
--min-hamming MIN_HAMMING 1
--min-abund MIN_ABUNDANCE 1
--detect-singletons DETECT_SINGLETONS false
--max-clust MAX_CLUST 0 (unlimited)
--greedy GREEDY true
--use-quals USE_QUALS true

--detect-singletons, --greedy, --use-quals and --no-kmer-screen are tri-state: omit to inherit or use the default, or pass true / false to set them explicitly.

Alignment

--band — alignment band radius, R's BAND_SIZE. 16 is the Illumina default; 32 is recommended for PacBio HiFi 16S amplicons (per the DADA2 LRAS manuscript); -1 is unbanded (O(n²), rarely needed).

The platform split is not cosmetic. On MiSeq data the band can drop to 8 safely, but 4 breaks; on PacBio HiFi it cannot drop from 32 at all — 32 → 16 already changes the ASV set. See Band size & platform defaults.

--gap-p — gap penalty, R's GAP_PENALTY (default −8).

--homo-gap-p — homopolymer-run gap penalty, R's HOMOPOLYMER_GAP_PENALTY. When unset it falls back to whatever --gap-p resolves to, reproducing R's NULL default. PacBio pipelines sometimes set this closer to 0 (e.g. -1) on the theory that homopolymer indels dominate — but note that the -1 recommendation originates with 454, not PacBio, and HiFi data does not have that error mode.

--match / --mismatch — R's MATCH (+5) and MISMATCH (−4). R's full 4×4 score matrix is commented out upstream; the active path is these scalars, and dada2-rs matches that path.

--align-backend — nw (default) is Needleman-Wunsch. wfa2 is the experimental WFA backend (wfa2lib-rs): ASV-equivalent on the Illumina and PacBio HiFi data tested, but alignments are not byte-identical. wfa2 requires a build with --features wfa; a default build — and the published crate — errors if it is selected.

Screening

The screen decides which pairs are worth aligning. It does not define the clusters. Set it too aggressively and you lose real comparisons; set it loosely and you pay in alignment time.

--kdist-cutoff — k-mer distance cutoff, R's KDIST_CUTOFF (default 0.42). Pairs above this distance are not aligned. Lower screens more aggressively (faster, more false negatives); raise it for divergent sequences.

Tightening this is not free: it acts through partition and error-model bias, not just by screening out error copies, and n_asv is a misleading metric for judging it — use set identity. The clean lever is to decouple this cutoff from the one learn-errors runs under, leaving error learning at 0.42 while denoising tightens to 0.30; see KDIST cutoff decoupling.

--kmer-size — k-mer size for the screen and the Raw k-mer vectors, R's KMER_SIZE (default 5, range 3–8; 8 is a hard ceiling because k-mer indices must fit in a u16).

For PacBio HiFi, do not leave this at 5: on ~1.4 kb reads the screen becomes a no-op — nearly every pair is aligned, roughly 4–5× slower at scale. Use k=7 for speed or k=6 to cap memory; both give effectively identical ASVs. Memory scales as 4^k per Raw (k=5 → 1 KB, k=6 → 4 KB, k=7 → 16 KB, k=8 → 64 KB). See K-mer screen size.

--no-kmer-screen — disable the screen and align every pair. Much slower; use only when you suspect the screen is filtering valid comparisons.

Performance

Neither flag affects results.

--threads (default 1) — threads for both dereplication and the DADA2 comparison map.

--sample-jobs — multi-input only: how many samples to denoise concurrently, each on its own threads / sample-jobs sub-pool. A single sample's comparison map is often too small to feed many threads, so fanning samples across smaller sub-pools keeps every core fed — roughly 4 threads per sample is the sweet spot — and bounds memory to this many samples in flight. Defaults to round(threads / 4), i.e. serial at ≤4 threads. Dial it down for very large or complex samples if memory is tight.

Output

--output / -o — single input only; write JSON here instead of stdout.

--output-dir — required for more than one input; one {sample}.json per sample, the same convention as dada-pooled. Created if absent.

--compact — minified JSON instead of pretty-printed.

--gzip — gzip the per-sample files as {sample}.json.gz.

ASV lineage (#306). Each record in asvs carries, beside its sequence, abundance and birth statistics:

field meaning
cluster the ASV's cluster id in this denoising result
birth_from the cluster it split from; null for the initial cluster
nearest the closest other ASV, by substitutions plus indel events
nearest_hamming, nearest_indels substitutions and internal indel events to it; end overhangs are not counted

birth_from is the lineage at the time of the split; centres move afterwards, so it need not be the final nearest ASV. Ids are per sample under dada and dada-pseudo, and pool-wide under dada-pooled, where a referenced ASV may be absent from a given sample's file. nearest is found after denoising (a k-mer screen, then one alignment per candidate) and is omitted when there is a single ASV.

Diagnostics

None of these change the ASVs.

--verbose — the run's shape and results to stderr: CPU allocation, alignment backend, active tuning gates, resident footprint, the index decision, a phase-times one-liner, and per-cluster progress. The phase attribution and optimisation counters are not printed; pass --metrics-json for those.

Under --sample-jobs > 1 the per-round progress records are prefixed with the sample that produced them, since concurrent output is otherwise unattributable. With a single sample in flight — or under dada-pooled — the text is left exactly as DADA2's C layer writes it.

--aux-outputs — emit R-DADA2-parity per-cluster diagnostics in the output JSON: cluster_stats (n0/n1/nunq/birth_qave/post-hoc p-value), cluster_quality (mean quality at each reference position), birth_subs (the substitutions that drove each cluster split), and transitions (a 16 × nq transition-by-quality matrix). This costs one extra alignment per Raw against its cluster center, so enable it only when you need the diagnostics.

--cluster-trace — write a single full cluster trace (clusters.json) here: cluster centers, members with hamming distance / λ / p-value, and birth metadata. Useful for ASV-calling QC and one-off plots; see examples/cluster_trace/.

--trace-no-members — omit the per-cluster members array, emitting only centers and birth metadata.

--trace-min-abund (default 1) — only include trace members at or above this abundance.

--metrics-json — write a structured record of the run's phase times, compare attribution, shuffle split, footprint and index decision to this file. Free to leave on: it collects only what the run already pays for.

--metrics-attribution — add the per-comparison timings (busy, map parallel efficiency, the screen / DP / al2subs split). This slows the run, so do not combine it with a timing measurement. Requires --metrics-json.

See Run metrics for the schema and the two levels.

--failed-uniques — write a TSV of uniques that failed to denoise (final p-value < omega_c, map == null). Tidy long format with a header — sequence<TAB>sample<TAB>reads — one row per failed unique per sample it appears in. This answers "what failed to denoise, and how many reads did it cost?" without hand-joining map back to the derep uniques.

--indel-events, --indel-opportunities (experimental) — tabulate the insertions and deletions DADA2's error model ignores, from the same final unique-to-centre alignments that give trans (#291). Neither changes the denoising output.

  • --indel-events: one row per indel event per unique, for every cluster member (correct 1 or 0). Columns: sample, cluster, center_len, unique (input index), reads, correct, p, lambda and cluster_reads (the inputs of p), nsubs, n_internal (non-terminal events in this unique), type (I/D), center_pos, len, bases, hp_len, terminal, qual_min, qual_mean (inserted bases), qual_left, qual_right (flanking unique bases). hp_len is the centre length of the homopolymer run the event lengthens or shortens; 0 when the bases are mixed or match neither flanking base. terminal gaps lie outside the first and last aligned centre positions.
  • --indel-opportunities: read-weighted rate tables over correct members and internal positions, as sample, table, key, opportunities, events. sub_q and ins_q are keyed by quality (aligned unique bases and substitutions; all unique bases and inserted bases); del_hp by centre run length (bases and deleted bases); ins_hp by centre run length (runs and insertions lengthening them); del_fq by the lower quality of the aligned unique bases either side (centre bases and deleted bases); ins_len and del_len by event length (events only). sub_q sums to trans. Further tables split these: ins_q_novel and ins_q_run (insertions matching neither neighbour, or lengthening a run), del_fq_L1…del_fq_L5 (del_fq per centre run length, 5 = 5 or more), del_hp_A…ins_hp_T (per run base), del_cq and del_cq_L<n> (as del_fq, keyed by the centre's own mean quality at the base instead), and del_fqcq_L<n> (both, keyed flanking × 128 + centre).

Qualities are the unique's per-position mean, as in trans. With several inputs, each file holds every sample's rows, sorted by sample.

Both files open with # key: value lines naming the run: dada2_rs_command, dada2_rs_version, error_model (the path given), error_matrix (err_out, or err_in under --use-err-in), and for dada-pseudo also round and reestimated_err_between_rounds. Read them with comment = "#" (pandas) or comment.char = "#" (R). Fuller provenance is tracked in #297.

Experimental

These flags are unstable and off by default. They are not drop-in replacements for the defaults.

--screen-backend — an alternative to the k-mer screen. kmer (default) is the ESPRIT-style 4^k frequency vector that R/C++ DADA2 uses; minimizer is a winnowed-minimizer sketch. Neither defines the clusters.

Expect mostly-concordant but not identical results. On every dataset tested the ASV sets and counts agree closely, but there are real differences: typically a fraction of a percent of reads and a small number of low-abundance ASVs (all under ~15 reads in the datasets measured).

It pays off where a lot of screening happens — diverse pools, where most pairs are dissimilar, so the screen runs on everything and the aligner on almost nothing. There the k-mer screen can reach 44–77% of denoising time and this replaces it with under 10%. On low-diversity data the screen is ~1% of runtime and there is nothing to win.

It requires tuning. The default --kdist-cutoff 0.42 is wrong for this backend — it over-screens by roughly 3×. Use --minimizer-k 8 with a cutoff derived for your data (~0.64 on diverse Illumina pools, ~0.50 on PacBio HiFi) via kdist-calibrate --screen-backend minimizer. See Minimizers as the screen.

--minimizer-k (default 8, range 5–31) — k-mer size for the minimizer sketch. Independent of --kmer-size, which still governs the frequency screen. k=8 was chosen on Illumina via a pair audit (k=11 misses real neighbours) and has not been re-derived per platform the way --kmer-size had to be.

--minimizer-w (default 5, range 1–64) — winnowing window in k-mers. The sketch retains ~2/(w+1) of positions, so larger w is smaller and faster but less sensitive. This was never varied in any published measurement — every result on the findings page is a single w=5 point.

--screen-audit — evaluate both screens on every comparison and align the union, reporting how often they disagree and, for each disagreement, how many substitutions the alignment actually found. Answers whether the minimizer screen passes a superset, a subset, or a different set of the pairs --kdist-cutoff passes. ASVs are unaffected (audit-only alignments are discarded), but the run is substantially slower and its timings are not comparable to a normal run.

--wfa-max-edits (default 50, 0 = unbounded) — WFA edit-budget cap, in edit operations, used with --align-backend wfa2. WFA aborts a pair once it needs more than this many edits and falls back to the NW path for that pair (NW-identical there). The budget is an absolute edit count, not a fraction of read length: denoising only aligns near-identical reads (~99.9% identity), so real error copies stay a few edits apart regardless of read length, while divergent non-error-copy pairs that slip past the screen stay bounded. Ignored by the nw backend.

Internally an edit budget E maps to a WFA cost of E·|gap_p| — e.g. 50·8 = 400 with default scoring. The DADA2RS_WFA_MAX_STEPS environment override is specified in those raw cost units, not edits. DADA2RS_WFA_LINEAR=1 runs WFA with gap-linear penalties instead of affine with gap opening 0: the same scores with one wavefront instead of three, for benchmarking (#51).

--indel-rates, --hp-ceiling, --indel-pair-window, --indel-end-zone — an indel-aware λ (#291). R DADA2's λ scores substitutions only, so a variant that differs from its cluster centre only by indels is absorbed however abundant it is. Given --indel-rates, each indel event between a unique and its centre multiplies λ by its probability and counts toward hamming. The rates are either fitted rates from learn-errors --indel-rates-out, used as given, or the --indel-opportunities table of a prior run on the same kind of data, fitted on load by pooling each quality with its neighbours until it holds 20 events.

event probability
insertion of k bases rate per base at the inserted bases' lowest quality (novel and run-lengthening bases fitted apart) × r^(k−1)
deletion of k bases rate per base at the centre run length (1–4, 5+) and the lower flanking quality × r^(k−1)
homopolymer-length change (largest run involved ≥ 2) --hp-ceiling (default 0.05)
insertion and deletion of equal length within --indel-pair-window (default 5) the substitutions they stand for, from the error model
terminal gaps; indels within --indel-end-zone (default 10) of the 3′ end ignored

The homopolymer ceiling sits above the rate of recurring slippage hotspots (at most ~8% of the parent on PacBio HiFi, ~0.3% from PCR on Illumina), so those stay absorbed while a homopolymer-length variant far above it can split. Lower it for high-fidelity, low-cycle protocols; 1 never splits on homopolymer changes and off scores them at the measured rates. The end zone suits reads with a low-quality 3′ tail; set it to 0 where read ends are reliable, such as HiFi or recent Illumina chemistries. The settings are recorded under params.indel_lambda. The error model is unchanged: learn-errors still learns from R's partition. Evidence for each choice is on #291.

For PacBio HiFi reads trimmed of both primers, use the indel-aware λ with --end-model global --gap-open-p -1 --indel-end-zone 0 (#309):

  • Near-end indels. On spike-ins placed 3–25 bases from either end it recovered 176 of 192 on Zymo and 128 on BEI, against 167 and 119 for ends-free, with no false splits. ends-free misses indels in the first few 5′ bases, which a free leading gap absorbs; the default 3′ end zone misses most indels within 8 bases of the 3′ end.
  • Truncated reads. With linear gaps, global splits reads truncated at one end: a long end gap fragments into interior gaps that λ then scores. The opening cost keeps it one gap, and the read is absorbed.
  • Full run. On 95 PacBio samples (dada-pooled), truncated reads are absorbed, insertion variants such as 16S intervening sequences are kept, and the ASVs match ends-free except for a few low-abundance ones. Pooled denoising takes about 1.5× as long as ends-free.

--end-model ends-free --indel-end-zone 0 remains the conservative choice. Both are experimental.

--end-model (default ends-free) — which alignment end gaps are free (#289). An end gap is a run of gaps at the start or end of an alignment, where one sequence overhangs the other. A free end gap costs nothing; a penalised one costs --gap-p per column, like an interior gap.

value leading (5') gaps trailing (3') gaps use for
ends-free free free anything; R DADA2's model
free-3p penalised free reads that start right after a trimmed primer and end at an arbitrary point
global penalised penalised reads trimmed of both primers, so both ends sit at fixed biological positions

ends-free and free-3p differ only at the 5' end. ends-free lets one read start later than the other at no cost. free-3p assumes primer removal has put every read's first base at the same position, so a leading gap must be a real indel and is charged as one.

Why not global for Illumina or other paired-end data. Each read is anchored only at its own primer. Its 3' end is set by the number of sequencing cycles or by --trunc-len, not by the amplicon, so the last base of two reads falls at different biological positions whenever an indel sits between them. A trailing overhang there is expected, and global would charge it as gaps or force mismatches near the 3' end, which then count as errors in the error model. Forward and reverse reads are denoised separately, before merging, so the same applies to both. Use free-3p for these reads. global fits only when every read spans the whole amplicon and both primers are removed: full-length PacBio 16S, or short amplicons whose reads all run through to the reverse primer. Amplicons of variable length, such as ITS2, mix reads that run through with reads that do not, so their 3' ends are not consistently anchored; use free-3p there.

Ragged 5' ends. A read that keeps part of a primer, or loses a few bases past it, is charged a leading gap under free-3p and global that ends-free leaves free. This does not change the ASVs: a read's error probability (λ) is computed from substitutions only, as in R, so a read that differs from its centre only by a leading gap is absorbed the same way under every model. On the MSA-1003 mock, giving 1–20% of reads a ragged start (±1–3 bases), or leaving the full forward primer on 1–5% of them, gave the same ASVs under all three models (#289).

Both backends honour the setting; chimera detection keeps R's model. The value is recorded in the error model, and dada warns when its own differs. Against truth (MiSeq SOP Mock, PacBio MSA-1003) neither opt-in model changes ASV calls; free-3p gives a byte-identical table on the MiSeq SOP. global on truncated Illumina reads raises reverse-read error rates by up to 24%, which is the 3' mismatch effect described above (#289).

--gap-open-p (default 0) — affine gaps: a gap of length L costs --gap-open-p + L × --gap-p. 0 is R's linear model and leaves every output unchanged; a positive value means the same penalty, as for --gap-p.

With linear gaps, one long gap and the same number of scattered gaps cost the same, so a read truncated at one end, or an inserted block such as a 16S intervening sequence, can align as dozens of scattered gaps that hide its substitutions or, with the indel-aware λ, count as dozens of indels (#309). An opening cost makes the single block the best alignment. Keep it small:

--end-model global learned substitutions vs ends-free: Zymo / BEI / PacBio 95 reads truncated at one end, with --indel-rates
--gap-open-p 0 +0.1% / +0.05% / −0.15% split off as ASVs
--gap-open-p -1 +0.5% / +0.2% / +0.9% absorbed
--gap-open-p -8 +11.5% / +4.7% / +13% absorbed

A large opening cost re-describes nearby insertion–deletion pairs as substitutions, which the error model then learns. On the near-end spike-ins above, global with --gap-open-p -1 and --indel-end-zone 0 recovered 176 of 192 on Zymo and 128 on BEI, against 167 and 119 for ends-free, with no false splits.

It works with every end model and both backends, and cannot be combined with a --homo-gap-p different from --gap-p. The value is recorded in the error model and the output params, and dada warns when its own differs. Each affine alignment costs about 1.3× a linear one (learn-errors on the PacBio mocks took 1.1–1.2× as long).

See also