This document is the reference for the tooling and log output used to evaluate
dada2-rs performance and to benchmark it head-to-head against R DADA2. It is
written in plain Markdown so it can be dropped into MkDocs or Sphinx (via
myst-parser) for a ReadTheDocs site with no changes.
- Where things live:
dev/benchmark/(the head-to-head harness),dev/(validation/error-model scripts),scripts/(plotting and read-tracking helpers). - What you get: per-step wall time, CPU time, effective cores, and peak RSS for both stacks; scaling sweeps; and ASV/error-matrix concordance checks.
A single Python driver runs the full pipeline for each stack, times every step, captures per-process peak memory, and prints a symmetric side-by-side table plus a CSV.
| File | Role |
|---|---|
bench_pooled.py |
Orchestrator (Python). Runs dada2-rs and/or R, captures timing/RSS/CPU, writes summary.csv. |
run_dada2_pooled.R |
R reference pipeline as one process (fair end-to-end wall + one overall RSS). |
bench_step.R |
R reference pipeline as one process per step (gives per-step RSS). |
learn_errors_r.R |
Fits one error model with R's learnErrors() on already-filtered reads — the r-model arm (§2.3). |
Each --pool mode invokes the dada2-rs subcommand that is the direct analog of
the R function — one process each, so the comparison is function vs function,
not harness vs harness:
--pool |
dada2-rs subcommand | R analog |
|---|---|---|
true |
dada-pooled |
dada(..., pool = TRUE) |
false |
dada (multi-input, --sample-jobs) |
dada(..., pool = FALSE, multithread = N) |
pseudo |
dada-pseudo |
dada(..., pool = "pseudo") |
Pipeline stages (each a timed row in the output):
- Illumina (paired):
filter→learn_fwd/learn_rev→dada_fwd/dada_rev→merge→make_table→remove_bimera - PacBio (single-end):
remove_primers(trim + orient + filter in one) →learn→dada→make_table→remove_bimera
All timing/memory comes from os.wait4() on each child process — no
/usr/bin/time, no /proc polling, so it is portable across the cluster and
macOS:
| Metric | Source | Meaning |
|---|---|---|
wall_s |
wall clock around the process | elapsed time |
cpu_s |
ru_utime + ru_stime |
total CPU-seconds consumed |
cores |
cpu_s / wall_s |
effective cores used (≈ threads = fully utilized) |
peak_rss |
ru_maxrss (kB on Linux, bytes on macOS → normalized) |
peak resident set high-water mark |
For fanned steps (filter, remove_primers, which run one subprocess per
sample concurrently), the row aggregates: wall_s = wall clock of the whole
batch, cpu_s = sum across children, peak_rss = max of any single child.
coresis the key diagnostic. It tells you whether a step actually uses the threads you gave it.cores ≈ threads→ saturated;cores ≪ threads→ under-utilized. See §3 for how to read it together withcpu_s.
| Option | Purpose |
|---|---|
platform (positional) |
illumina or pacbio |
input (positional) |
directory of raw FASTQ files |
--dada2rs PATH |
required — path to the binary (no auto-discovery; build target matters, see §6) |
--threads N |
total threads |
--pool {true,false,pseudo} |
denoising mode (see §1.1) |
--sample-jobs N |
samples denoised concurrently for pseudo/false (default round(threads/4)) |
--cache-samples |
pseudo only: hold all samples in memory (default is streaming, which benchmarked faster + lighter) |
--run-r |
also run the R pipeline (off by default → dada2-rs only) |
--r-mode {split,single,both} |
how to run R (see §1.5) |
--no-run-rust |
skip the dada2-rs pipeline |
--nbases N |
bases to subsample for error learning |
--errfun-arm ARM |
error model to run, replacing the per-platform default (§2.3) |
--thread-sweep N,N,... |
thread-scaling study (§2.1) |
--sample-jobs-sweep N,N,... |
samples-in-flight study (§2.2) |
--errfun-sweep ARM,ARM,... |
error-model study (§2.3) |
Platform-specific filter/denoise knobs: --trunc-len, --max-ee, --trunc-q,
--max-n (Illumina); --min-len, --max-len, --band, --homo-gap,
--kmer-size, --primer-fwd, --primer-rev, --max-mismatch (PacBio). Run
python3 dev/benchmark/bench_pooled.py --help for the full list.
The console prints a per-step table for each stack and a HEADLINE block
(end-to-end speedup, pooled/per-sample denoise speedup, peak-RSS ratio). The
machine-readable form is <outdir>/summary.csv:
stack,step,wall_s,cpu_s,cores,maxrss_kb
dada2-rs,dada_fwd,80.53,1562.65,19.40,2219336
...
R-split,dada_fwd,447.67,2164.32,4.84,1470632
R-single,TOTAL,1208.38,5417.63,4.48,2021736maxrss_kb is the exact value; the printed table rounds RSS to MiB. Sweeps
write sweep.csv / sweep_jobs.csv (§2).
R is captured two ways because they trade off differently:
single— the whole R pipeline as one process. Gives a fair end-to-end wall and one overall peak RSS. Per-step wall comes from R'ssystem.time(); per-step RSS/cores are not available (one accumulating process).split— oneRscriptper step (state passed via.rds). Gives per-step RSS, but each step reloads R + dada2 (~150–200 MB, several seconds), so per-step wall is inflated.both(default) reports both; theHEADLINEusessinglefor the fair end-to-end wall andsplitonly where per-step RSS is needed.
Caveat — not perfectly apples-to-apples on RSS: dada2-rs's per-step RSS is per separate process (memory released between steps); R-single's is one accumulating process. The numbers are directionally comparable, not identical in methodology.
# dada2-rs only, Illumina, pooled
python3 dev/benchmark/bench_pooled.py illumina /data/MiSeqSOP \
--dada2rs target/release-native/dada2-rs --threads 24 --pool true
# head-to-head vs R, pseudo-pooling
python3 dev/benchmark/bench_pooled.py illumina /data/MiSeqSOP \
--dada2rs target/release-native/dada2-rs --threads 24 --pool pseudo --run-r
# PacBio HiFi (raw, primered reads), supply primers
python3 dev/benchmark/bench_pooled.py pacbio /data/HiFi \
--dada2rs target/release-native/dada2-rs --threads 24 --pool pseudo \
--primer-fwd AGRGTTYGATYMTGGCTCAG --primer-rev RGYTACCTTGTTACGACTT --run-r
# pseudo in cached mode (default is streaming; --cache-samples opts into all-in-memory)
python3 dev/benchmark/bench_pooled.py illumina /data/MiSeqSOP \
--dada2rs target/release-native/dada2-rs --threads 24 --pool pseudo --cache-samplesbench_table.py turns one or more summary.csv files into a Markdown table —
useful since the large-dataset runs happen on the cluster and the tables are
pasted into the docs (see Benchmark results).
# Scorecard — one row per run, end-to-end head-to-head:
python3 dev/benchmark/bench_table.py \
"pooled=bench_true/summary.csv" \
"per-sample=bench_false/summary.csv" \
"pseudo=bench_pseudo/summary.csv"
# Per-step breakdown for a single run:
python3 dev/benchmark/bench_table.py --per-step bench_pseudo/summary.csvLABEL=path names each run; R columns compare against the R-single rows (fair
end-to-end wall + overall RSS), and rust-only runs simply omit them.
The two scaling sweeps (§2.1, §2.2) prepare inputs once (filter + learn) and then re-run only the denoise step at each setting — isolating the scaling behavior of the inference. The error-model sweep (§2.3) varies the thing that has to be re-learned, so it shares only the filter.
Varies --threads, pinning --sample-jobs 1 so it measures single-sample
thread scaling. Output (sweep.csv): threads, wall_s, cpu_s, cores, speedup, efficiency. efficiency = speedup / (threads / baseline_threads).
Reads the Amdahl story: efficiency rolling off and cpu_s climbing with
threads = the per-sample alignment map is too small to feed the threads, so
rayon workers spin-wait (burning CPU without cutting wall).
At fixed --threads, varies --sample-jobs (samples denoised concurrently,
each on a threads / jobs sub-pool). Applies to --pool pseudo or false.
Output (sweep_jobs.csv): sample_jobs, threads_per_job, wall_s, cpu_s, cores, maxrss_kb, speedup, with the fastest row marked.
How to read it:
- Minimum
wall_sis the optimum number of samples-in-flight. cpu_srising again at highjobs= sub-pools too small, per-map overhead returns (the mirror image of the spin in §2.1).peak_rssrising withjobs= more concurrent working sets — the speed/memory trade for pickingjobs.
The default --sample-jobs = round(threads/4) came from this sweep (the
wall-time curve plateaus around ~4 threads/sample).
Runs the pipeline once per error model on one shared set of filtered reads, so a difference between arms is the error model and nothing else. Filtering happens once; each arm then re-learns on those files and runs the same denoise → merge → table → chimera tail. Unlike §2.1/§2.2 this is not a denoise-only sweep, because the error model changes what denoising is handed.
An arm names one error model. Not every error model is something we fit, so there are three kinds:
| Arm | Kind | Meaning |
|---|---|---|
loess |
learn | the default LOESS errfun, binary's default surface |
loess:interpolate / loess:direct |
learn | LOESS pinned to a surface |
binned-qual:2,11,25,37 |
learn | piecewise-linear over those anchors (see Binned quality scores) |
noqual |
learn | quality-independent |
pacbio |
learn | the PacBio errfun |
external:<command> |
learn | --errfun external --errfun-cmd <command> (see Using an external error model) |
model:<json> |
model | a pre-fitted model, used as-is; nothing is learned. Illumina needs two, model:<fwd>:<rev> |
r-model |
r-model | R DADA2's learnErrors() fits it, on our shared filtered reads |
The same syntax selects a single model with --errfun-arm, which replaces the
hardcoded per-platform default (loess on Illumina, pacbio on PacBio).
Without either flag nothing changes.
Any arm may carry a display label, label=spec, which is what the results
table and the arm's output directory use:
rs-loess=loess:interpolate
r-ref=external:Rscript examples/external_errfun/loess_reference.R
external: and model: arms get a shortened default label
(external:loess_reference.R), with the full spec kept in the CSV's spec
column. Two arms that shorten to the same label are rejected — label them.
Caveats worth knowing before building a sweep string:
- Commas separate arms, so an
external:command containing one needs a wrapper script.--errfun-armdoes not split, so a comma is fine there. external:runs your script once per self-consistency iteration, so a slow script is paid for repeatedly — that arm'slearn_sis not comparable to a built-in's on equal terms.r-modelneedsRscriptwith dada2 and jsonlite. It is the only arm that needs R; ifRscriptis missing the arm is rejected by name before anything runs, so the rest of the sweep is still runnable by dropping it. Point--rscriptat a specific binary if it is not onPATH.model:records no learn step, because nothing was fitted — itslearn_sis0.0and itswall_sis denoising onward. That is the honest number, not a missing measurement.
python3 dev/benchmark/bench_pooled.py illumina /data/MiSeqSOP \
--dada2rs target/release-native/dada2-rs --threads 24 --pool false \
--errfun-sweep loess:interpolate,loess:direct,noqual
# ours vs R's own model vs R's loessErrfun run as an external script
python3 dev/benchmark/bench_pooled.py illumina /data/MiSeqSOP \
--dada2rs target/release-native/dada2-rs --threads 24 --pool false \
--errfun-sweep "loess:interpolate,r-model,r-ext=external:Rscript examples/external_errfun/loess_reference.R"The first arm is the reference; concordance columns are measured against
it, so the arm order is the question being asked. Output
(sweep_errfun.csv): arm, wall_s, learn_s, cpu_s, cores, maxrss_kb, n_asv, jaccard_vs_ref, exact_count_frac, identical.
jaccard_vs_ref— ASV sequence-set overlap. Moves when a model changes which variants are called.exact_count_frac— fraction of shared ASVs whose full per-sample count vector matches. Moves when a model changes abundances without changing the ASV set — the NovaSeq binned-errfun finding is exactly this shape, and reading onlyn_asvwould have missed it.wall_sexcludes the shared filter step;learn_sis the part of it that is the error model itself.
The reference arm is compared against itself, so its row is always
1.0000 / 1.0000 / yes. That is the harness's own null: if a second arm also
reports yes, the two models genuinely agree, not that the comparison is
broken.
--errfun-sweep is dada2-rs only — R's learnErrors has no equivalent knob,
so an R column would be the same model in every row. Passing --run-r with it
is an error rather than a silently constant column. To put R's error model in
the comparison, use the r-model arm, which is R's learnErrors() fitted on
the same filtered reads as every other arm.
On the 20-sample MiSeq SOP set, loess:interpolate, r-model and R's
loessErrfun run as an external: script all produce the same 232 ASVs
with identical per-sample counts, and the r-model and native error
matrices agree to 1e-15 — the end-to-end form of the
R parity ledger. That is a
meaningful null only because the same harness separates loess:direct (3.4%
of count vectors move) and noqual (an ASV lost, half the counts move).
cores (CPU/wall) finds the problem; cpu_s confirms the fix. A step that
is busy but not faster shows up as high cores and high cpu_s and poor
speedup — that signature is spin-waiting, not useful work. When you fix it
(e.g. right-sizing sub-pools), the wall drops and cpu_s drops (reclaimed
spin) — so cores may look unchanged because both its numerator and denominator
shrink. Trust wall_s and cpu_s, not cores alone, when judging a fix.
Rules of thumb:
| Observation | Interpretation |
|---|---|
cores ≈ threads |
step is saturating the machine |
cores ≪ threads, low cpu_s |
genuinely idle (load imbalance, too few samples, serial backbone) |
cores high, cpu_s high, poor speedup |
spin-waiting on an under-fed parallel region |
cpu_s roughly equal between stacks, wall differs |
same work, different parallel efficiency |
Peak RSS caveats: it is the kernel high-water mark of a single process. For fanned/concurrent steps the reported value is the max of one worker, so it understates the true concurrent system memory when many samples run at once.
Run any dada / dada-pooled / dada-pseudo with --verbose for a per-run
breakdown on stderr (also captured in the harness *.log files):
ALIGN: 123456 aligns, 654321 shrouded (1000 raw).
[dada] phase times (serial except compare-map): compare=12.3s (map=11.0s parallel, store=1.3s serial) shuffle=0.4s bud=0.2s p_update=0.3s
[dada] map parallel efficiency: 74% (busy=320s / map=11.0s × 24 threads)
ALIGN— alignments performed vs shrouded (skipped by the k-mer pre-screen). On long reads at smallkthe shroud count is ~0 (screen is a no-op); largerkengages it.phase times— onlycompare-mapis parallel.shuffle,bud,p_update, and thestorehalf of compare are serial. If those dominate, you are Amdahl-bound on the greedy cluster-building backbone, not on the map.map parallel efficiency— how well the parallel region itself uses threads; well below 100% means in-region load imbalance.
Error-model parameter warning. dada* records the alignment params in the
error-model JSON and warns when the denoise params disagree:
[dada] warning: 1 dada parameter(s) differ from error model errors_pacbio.json;
pass --inherit-err-params to adopt the err model's values:
homo_gap_p = -1 (err model: -8)
This means the error model was learned with different params than you are
denoising with. Either pass the same params to learn-errors (preferred — so
the model matches), or --inherit-err-params to adopt the model's values.
Common on PacBio if --homo-gap-p is set for dada but not for learn-errors.
Performance only matters if the results are right. These check correctness against R, independent of timing:
| Tool | What it checks |
|---|---|
dev/compare_asvs.py |
ASV table A/B diff (sequences present, per-sample abundances) between two runs |
dev/run_rust_errors.sh |
runs filter → derep → learn-errors on one sample with R-matched params |
dev/compare_errors.R |
compares the learned error matrices against R's learnErrors() |
dev/run_kmer_sweep.sh |
sweeps --kmer-size, reporting shroud %, ASV count, wall, RSS (issue #15) |
dev/plot_cluster_diag.R |
plots per-iteration cluster diagnostics from --diag-dir output |
plot_errors.R |
plots an error-model JSON |
track_reads.py |
per-stage read-count tracking (the DADA2 "track" table) |
dev/summarize_learn_errors.py |
summarizes a learned error model |
.github/workflows/concordance.yml runs the comparison with --gate, at one of
two levels per arm:
| arm | gate | observed on the fixture |
|---|---|---|
| Illumina, NW | exact: every sample × ASV count equals R's | 0 of 15 cells differ |
| Illumina pseudo-pooling (both arms) | exact | 0 of 389 / 385 cells differ |
| PacBio k=5, NW | exact | 0 of 95 cells differ |
| PacBio k=7, NW | near: same ASVs as R, ≤ max(10 reads, 0.1%) moved | 3 reads moved |
| WFA, Illumina and PacBio k=5 | near | 0 and 1 reads moved |
An exact failure lists the differing cells. It is either a bug or a deliberate departure from R, and either way needs a decision. The near arms differ for known reasons: R's k-mer screen is fixed at k=5, and WFA can choose a different alignment of equal score. The earlier thresholds (recall 0.85, precision 0.80, count correlation 0.95) passed a PacBio run that lost one of R's 93 ASVs and moved 18 reads; both levels fail it (#289).
What the gate does not instrument is per-stage read accounting: the intermediate counts that never reach the sequence table, such as the denoised per-sample totals. See #204.
CI never runs R; the reference tables are committed as static CSVs.
The thresholds above catch regressions. They are not the standard changes are held to. A change that is expected to leave results untouched — a performance change, a refactor, a memory reduction — is checked for byte-identical output before it lands, and the findings pages throughout this site report churn of exactly 0 for that reason. Where a change legitimately does move results, the amount is measured and stated rather than absorbed.
That discipline is the substantive answer to "can I trust these numbers". It also means results should stay stable release to release: the same guardrail that blocks an accidental change blocks an accidental drift, which is a property worth more to a long-running analysis than agreeing with any single external run to the last read.
Typical correctness loop for a benchmark run:
- Run the harness for the stack(s).
- Export both final sequence tables to TSV (
seq-table-to-tsv) or compare the per-sampledadaJSONs withcompare_asvs.py. - Confirm ASV concordance before trusting any speed claim.
--dada2rs is required — the harness does not auto-discover, because the
build target changes the numbers:
target/release— portable, reproducible across nodes. Use for numbers others should be able to reproduce.target/release-native— built with-C target-cpu=native(therelease-nativeprofile). Best-case single-machine performance (AVX2/AVX-512 on x86, full NEON on Apple Silicon), but not portable across microarchitectures.
For a head-to-head where R's C was compiled with the cluster's native gcc, prefer
release-native so the alignment kernel isn't handicapped to SSE2.
Where a change is expected to be output-identical and to move only wall time, the safest A/B runs both arms from the same binary, selected by an environment variable, rather than from two builds:
| variable | effect |
|---|---|
DADA2RS_SHUFFLE_NO_PRUNE=1 |
forces b_shuffle's move pass to scan every cluster, disabling the dirty-cluster pruning (#132) |
dada2-rs dada-pooled ... --verbose # optimised
DADA2RS_SHUFFLE_NO_PRUNE=1 dada2-rs dada-pooled ... --verbose # baselineTwo reasons this is worth the small amount of gating code:
- It removes the wrong-build failure mode. Two A/B runs during #132 were
built from a stale checkout and silently compared
mainagainstmain— caught only because the expected verbose line was missing from the output. Check the version tag in any A/B (grep -o '0\.2\.0-[0-9a-f]*'on an output JSON) even when using this pattern. - The two arms are otherwise bit-identical code, so nothing else — compiler version, LTO, incidental codegen — can differ between them.
The tradeoff: the branch is evaluated at runtime, so it cannot be used for a change whose cost is inside the hot loop being measured. #132's is not — the check is once per move pass, not per raw.
On a multi-domain machine, memory placement is worth 25–30% of run_dada —
larger than most changes this harness exists to measure. bench_pooled.py
therefore takes --numa {none,interleave,bind}.
It applies to both stacks, by construction. The policy is resolved once and
applied in numa_wrap, which every launch routes through, rather than at each
call site. R DADA2 threads the same algorithm — same k-mer screen, same banded
NW — so pinning dada2-rs and not R would inflate every speedup number by up to
~25% for free. Whether R gains as much as we do is an open and interesting
question, and it is answerable only with both arms treated alike.
| policy | what it does | when |
|---|---|---|
none (default) |
no numactl |
matches every historical run in Results |
interleave |
pages round-robin across domains | most reproducible; see dev/numa_pin.sh |
bind |
CPUs and memory on one domain | fastest when the thread count fits inside a domain |
bind is not universally better: at or above a domain's core count it saturates
that domain's controllers and loses to interleaving, which is how
dev/numa_pin.sh came to recommend the opposite from a 64-thread measurement on
a 64-core domain. The harness checks the domain's CPU count and falls back to
interleave with a warning rather than honouring a request that cannot help.
Where numactl is unavailable — or the machine has a single domain — the run
proceeds unpinned with a warning. It never fails, and it never silently records
the policy you asked for:
# numa_effective=none numa_requested=bind threads=48
stack,step,wall_s,cpu_s,cores,maxrss_kb
summary.csv carries both the effective and the requested policy in a header
comment, because a run gathered under one policy is not comparable to one
gathered under another, and a downgrade the operator did not notice is exactly
the failure mode #145 exists to
close.
The default is none so that new runs stay comparable to the archived ones. It
is not a recommendation — see Tuning for your
data.
- PacBio input is raw, primered reads.
remove-primerstrims primers, orients (--orient), and applies the length/quality filters in one pass (mirrors RremovePrimers()+filterAndTrim()). Supply--primer-fwd/--primer-rev(5'→3'; the reverse primer is reverse-complemented internally). - Alignment params must be consistent. Pass
--band,--homo-gap-p, and--kmer-sizeto bothlearn-errorsand the denoise step (the harness does this). Mismatches trigger the §4 warning and subtly change results. The gap penalty is also exposed as--gap-p(default-8);--homo-gap-p, when unset, defaults to--gap-p— mirroring R'sHOMOPOLYMER_GAP_PENALTY = NULL(homopolymer gaps treated as normal gaps). The other result-affectingsetDadaOpt()knobs are also available:--match,--mismatch,--max-clust,--greedy,--use-quals. - k-mer size. On ~1.5 kb HiFi reads
k=5makes the pre-screen a no-op (≈ every pair is fully aligned → slower);k=7is the dada2-rs default. R fixesKMER_SIZE = 5, so a matched-k(k=5) run isolates kernel+threading speed, while ak=7run additionally shows the screen-effectiveness gain. Per issue #15 the final chimera-filtered ASVs are ~unchanged acrossk.
- Build the binary you intend to measure (
releaseorrelease-native). - Single mode, both stacks:
--pool {true|false|pseudo} --run-r→ read theHEADLINEandsummary.csv. - Diagnose under-utilization: look at the
corescolumn; for a deeper view run one denoise with--verboseand read the[dada] phase times/map parallel efficiencylines. - Quantify scaling:
--thread-sweep(thread scaling) and--sample-jobs-sweep(samples-in-flight, withpeak_rss). - Validate results:
compare_asvs.py/compare_errors.Rbefore reporting. - Memory-constrained / huge sample sets:
pseudostreams by default; use--cache-samplesto compare the all-in-memory mode (peak RSS vs wall).