This document describes how Fastmash is built: what each part does, how a command flows through it, and the principles behind the main design choices. It is written for contributors and for anyone evaluating the project. The glossary defines the terms used here.
| Language | Rust (edition 2024), one product binary crate |
| Platform | Linux, macOS 15+, Windows 11 x64 |
| Interface | GNU datamash 1.9 command language, with explicit CSV, health, selection, weighted mean and comparison extensions |
| Numbers | 80-bit extended precision, implemented in software |
| Sorting | Built-in stable sorting with disk spill |
| Runtime dependencies | Static musl Linux download, or system glibc for GNU source builds. System libraries on macOS. Static CRT and system DLLs on Windows |
| Network access | None |
All supported platforms use built-in sorting and private spill storage. ADR 0022 records Windows delivery. ADR 0014: Native sorting in one executable retires the external Sort route. ADR 0010: Native macOS sets its platform scope. ADR 0011: Qualify and deliver the same native archive sets its release delivery boundary.
These choices shape the whole codebase. Each is recorded as a decision in design decisions.
- Speed is the reason to switch. Fastmash exists because people run datamash in pipelines over large files. Performance work is measured on representative jobs against GNU datamash and against the previous Fastmash build. A change that makes an existing job materially slower needs an explicit, recorded justification.
- The datamash command language is the interface. Existing commands, options and field selectors work unchanged. Fastmash reimplements the behavior. It contains no GNU code.
- Portable results. One Fastmash version gives identical results on every supported machine for the same input and explicit settings. Arithmetic does not depend on the CPU's floating-point unit or on the C math library, so a result computed on a laptop matches the one computed in CI.
- Refuse rather than guess. If Fastmash cannot produce a result it can stand behind (an unsupported feature or locale, a checked resource limit, or a numerical result it cannot produce within its fixed rules), it stops with a clear message and exit status 77 instead of printing something plausible.
- Grow as needed, and check. There are no fixed caps on record length,
operation count or sample count. Data storage grows with checked
allocations, and sorting spills to disk. The few remaining fixed
limits (such as the 99-byte
--formatstring) are documented.
graph TD
cli["<b>fastmash</b><br/>command-line program"]
conv["<b>conversion</b><br/>number parsing and formatting"]
pn["<b>portable_numerics</b><br/>software arithmetic"]
nc["<b>numeric_contract</b><br/>80-bit value representation"]
sp["<b>sort_process</b><br/>native OS support"]
cli --> conv
cli --> pn
cli --> nc
cli --> sp
conv --> nc
pn --> nc
| Part | Path | Responsibility |
|---|---|---|
fastmash |
crates/cli |
The program: options, command grammar, records and fields, grouping, sorting, every operation, output and exit status |
conversion |
crates/cli/src/conversion |
Reading numbers from bytes and printing them, missing-value detection, locale number conventions |
portable_numerics |
crates/cli/src/portable_numerics |
Software 80-bit arithmetic for the numerical rules, including fallback log and exp |
numeric_contract |
crates/cli/src/numeric_contract.rs |
The checked 80-bit extended-precision value type and its classification |
sort_process |
crates/cli/src/sort_process |
Native Spill storage and thread policy |
All four modules forbid unsafe code. They have no public Rust library target. ADR 0019 records this boundary.
The unpublished fastmash-corpus-driver crate verifies frozen commands and
transport failures against a selected executable. Source checks and archive
installation checks use the same driver. The
testing guide describes its
fixtures, launchers and native fault mechanisms.
The unpublished fastmash-locale-tools crate generates and verifies numeric and
collation tables against identified native references and independent Unicode
data. Its reports retain the reference inputs and observed differences. See
locale tooling.
The unpublished fastmash-license-notices package selects the product's normal
dependency graph for archive notices. The separately built
release/publication-test-server/ package checks installation and publication.
These tools stay outside the product's normal dependencies.
The separately built tools/icu-exemplars/ development tool queries the installed
ICU 78 reference through safe resource APIs. Its native headers and libclang
requirements stay outside the application workspace. The locale tools combine
its exemplar snapshot with bounded GNU sort case-order probes to generate
collation candidates. These tools do not update product tables themselves.
flowchart TD
A[Arguments and environment] --> B[Parse options]
B --> C[Resolve locale<br/>from built-in rules]
C --> D[Parse the command grammar<br/>mode, grouping keys, operations]
D --> V[Validate controls and prepare<br/>one mode-specific request]
V --> E{Prepared mode}
E -- "table mode or health" --> T[Inspect or transform the table]
E -- "dataset comparison" --> R[Read both datasets<br/>align keys and complete summaries]
E -- "Top-N selection" --> U[Group and select complete records<br/>sort first when requested]
E -- "aggregate, grouping,<br/>crosstab, per-row" --> F[Plan operations<br/>share fields, samples and sums]
F --> G[Resolve named fields<br/>from the input header]
G --> H{"-s with<br/>grouping keys?"}
H -- no --> J[Read records in input order]
H -- yes --> S[Sort route]
S --> J
J --> K[Split fields<br/>detect group boundaries]
K --> L[Convert numbers<br/>skip missing values with --narm]
L --> M[Update operation state]
M --> N{End of group?}
N -- no --> J
N -- yes --> O[Compute and format results]
O --> P[Write output]
T --> P
R --> P
U --> P
P --> Q[Exit status<br/>0, 1, 77 or 70]
Some points worth knowing when reading the code:
- Planning happens before input is read. The whole command is parsed and
validated first, so most usage errors appear before any output. The preparation
module (
preparation.rs) checks control conflicts, selectors, result names and sorting eligibility in their required order, then returns one prepared request for the selected mode. Private payloads keep runners from receiving an unprepared request. Input headers, numerical admission and resource checks still happen at their existing execution points. Preparation does not open inputs. - Operations share work. When several operations read the same field, the
conversion, running sums and retained samples are shared.
median,q1andq3on one field sort one sample set, and the moment statistics share one pass over their inputs. The operation set (operation_set.rs) plans this sharing once the fields are known, reports what the command needs before input is read, and collects records, resets between groups and writes results. - Groups are adjacent records. Without
-s, a group ends when its key changes, and state for the next group starts fresh. That is what makes unsorted grouping a single streaming pass. - Ungrouped text calculations borrow buffered records. Without
--fullor per-row operations, complete records are processed directly from the reader's buffer. Input headers, records spanning buffers and final unterminated records use checked growable storage. Group representatives keep their existing owned storage. - Ordinary results are written progressively. Completed groups go to a small output buffer that is flushed as it fills (line by line on a terminal). A failure part-way through can leave earlier rows written, which is why the exit status must always be checked.
- Formats keep field boundaries. Ordinary text uses its separator rules. Explicit CSV decodes logical records and retains complete fields and original logical-record/physical-line locations. Output encodes each field separately. Record views let calculations borrow shared storage, while retained full rows and admitted selection candidates own what must outlive that view.
- Comparison and health finish before reporting. Dataset comparison keeps complete keys, per-key operation state and required samples, preserving each key's contribution order across each source. It finishes both scans and all changes before writing. Health collects field and width counters and bounded examples before emitting a readable or versioned TSV report. Neither adds spill for retained samples or a total-memory guarantee.
-s sorts records by their grouping keys before grouping. Fastmash chooses a
sort route for each command before it reads input (sort_route.rs: plan,
then run): whether hash grouping comes first, and whether the native sorter
keeps selected fields or complete records:
flowchart TD
A["-s with grouping keys"] --> R{"Input from a file, or piped<br/>in a language locale;<br/>no --vnlog or rand?"}
R -- yes --> Y[Hash grouping<br/>collect each group apart,<br/>sort only the groups]
Y -- "keys repeat, all exact" --> G
Y -- "otherwise: read the<br/>input again" --> C
R -- no --> C[Sort in memory<br/>stable, by projected keys]
C --> D{"Chunk full, and memory<br/>too tight to hold the input?"}
D -- no --> G[Grouping]
D -- yes --> E[Spill sorted runs to<br/>anonymous files in TMPDIR]
E --> F[Merge runs]
F --> G
- Hash grouping comes first, before sorting individual records, when
standard input is a regular file
(
projected_hash.rs). GNU datamash sorts with a stablesort -s, so each of its groups is exactly the records with one key tuple, in input order. Hash grouping collects each tuple's records into that group's own operation state as they arrive, keeping every group's state in one flat array (the operation set's plans are shared. Each group has only its running values, plus a separate store for operations that keep values), then sorts only the tuples, in the sort's order, and writes the groups through the same writer. It gives up before writing anything when the result might differ from the sort's (a missing or NUL key, a key the collation refuses, any operation error), when the groups are estimated to hold fewer than eight records each or, while records of a group rarely come together, to hold more state than the processor's caches (16 MiB), or when they outgrow the sort's memory target. The estimate is judged after 8,192 records and at each doubling: the file's records from its length, and its groups from those seen once and twice (the Chao1 estimate of the groups not yet seen, of which the rest of the file is expected to show a share). When it gives up, the input is read again from its start, header included, through the route the job would otherwise take. Piped input can be read again only if it was kept: in a language locale, whose sort holds whole records anyway, a replay reader holds what hash grouping reads in 1 MiB blocks charged to the same memory budget, and the native sort reads them back, releasing each, and then the rest of the pipe. Its chunk leaves room for what is still held. A pipe has no length, so the rest of it is taken to be as long as what was read. In theClocales piped input sorts, because the held input would take several times the memory of a sort that keeps only the selected fields (FASTMASH_PIPE_GROUPINGchanges either). With-W, whose sort keys keep the blanks before each field that groups ignore, a group is keyed by its fields with those blanks, as the sort compares them, and hash grouping gives up where two keys differ only in the blanks, or where a-zrecord holds a newline.--vnlogandrand(one generator across all groups) always sort.
- The native route sorts in process. It covers every sorted calculation
and sorted
rmdupin every supported locale. Language locales' text ordering comes from pinned Unicode ICU data built into the program rather than the host's locale files. A language sort key (locale.rs) is the ICU collation key, at three levels, of the text with the characters glibc ignores (punctuation, most symbols and spaces, from glibc's iso14651_t1_common incollation_glibc.rs) left out and those some locales weigh first replaced by the lowest weight, followed by glibc's fourth level, written so that its bytes compare as glibc compares it item by item: the ranks of the ignored characters and the weighted characters between them, each after the count of letters without a fourth-level weight (Han, letters a locale adds) skipped before it, with runs of weighted characters written as counts.
- A chunk holds the records being sorted in one arena (each record's
grouping keys, then the fields its operations use, or the whole record with
--full, in the language locales, or for numbers when the field separator could continue one, since a number is read on from its field's start into the rest of the record as GNU datamash reads it) and a 16-byte entry per record: the first key's eight-byte prefix, the key length and the arena offset. A stable radix sort orders the entries by prefix. Records whose prefixes tie on longer keys are sorted the same way by their next eight key bytes, round by round. Complete keys are compared only in small groups and after 64 bytes of a key. Ties keep input order. In the language locales, collation keys are computed in batches on several threads. Selected-field projections for other means, higher moments, normality tests and paired statistics grouped by one key in the C locales are also prepared in parallel. Language projection reuses exact collation keys for texts up to 256 bytes. Each cache holds at most 256 entries and 128 KiB, including allocated capacity. Native sorting reserves cache memory beside the batches' share and divides that allowance among their active parts. CSV sorting accounts actual cache capacity with its other projection scratch. A full cache or a longer key falls back to computing the key directly. The chunk, with the batches' and caches' shares, stays within its memory target, sized for 64 MiB. It takes 16 MiB first. When a chunk fills, it may grow instead of spilling: to hold the whole input of a regular file (an estimate from the bytes read so far, at most the buffer GNUsortwould use for the file), or to twice its size for piped input. If doubling does not fit, it tries a 50% chunk increase. Growth depends on the memory read at that moment. The chunk may take at most a quarter of the available memory, half of what is free under each cgroup memory limit, a share per running Fastmash of what is free under that limit after a reserve and of the host's available memory (so that sorts running together share them), and half of any address-space limit, and only while less than half the swap is in use. Below 64 MiB a chunk asks for at least 64 MiB, and if that growth is refused it still takes 64 MiB unless a cgroup limit refuses that too, so a sort started beside others that have grown spills rather than overrunning their limit. A fact that cannot be read keeps the chunk at 64 MiB.FASTMASH_SORT_MEMORY_BYTESfixes the target instead. The last chunk is read in place, without copying records. Earlier chunks are spilled and merged with it. - Spill writes sorted runs to anonymous temporary files (
O_TMPFILE, or where the filesystem lacks it a private file unlinked as soon as it is open), which the kernel removes automatically when Fastmash exits, however it exits. On Linux, a numerical calculation's runs under a cgroup memory limit are written back to storage every 8 MiB (fdatasync), so that dirty pages do not throttle the writer against the limit. Elsewhere runs use ordinary I/O. - On Linux, a numerical calculation over a large in-memory sort or a final
merge reading one level of runs can overlap emission with calculation
(
projected_calculation_pipe.rs): a merge thread writes each record's selected fields and up to two parsed numbers through two reusable batches, within the final merge's buffer allowance, and the calculation reads them in order. Each batch keeps text bytes, record boundaries and admitted numbers, avoiding a second number decode and delimiter scan. The threads borrow a stack-owned mutex and condition variable. Batch storage uses checked allocation. Long records span batches. Records whose numbers read on across a separator, and other platforms, use the serial path.
- Quoted CSV sorting uses its own complete decoded-record route, without hash replay or a text-format fallback. Record bytes, field ends, original locations and required language comparison segments are packed into shared chunks. Calculations borrow each chunk through group completion, and selection owns records only when they enter its candidates. Actual capacities and reusable decoder/key scratch count against the ordinary sort-memory policy. Checked private spill framing preserves complete ragged fields and locations. Bounded merge heads remain owned. An oversized record and merge heads can exceed the chunk target, and statistical samples retain their separate in-memory policy.
- Every sorted calculation takes the native route, including mixed
calculations.
rand, the other means (geomean,harmmean,ms,rms), skewness, kurtosis, normality tests and paired statistics use its selected-field storage where possible, with the same calculation collector and random state. - Sorted duplicate removal retains complete records, in stable key order, and compares duplicate keys exactly. Case folding changes sort order, not duplicate-key equality. It shares the native sorter and spill storage. A later record whose sort key bytes repeat an earlier record's can never be printed, so it is dropped before sorting. Tracking stops adding keys once they would take a quarter of the sort's memory target, held alongside the sort and released before output, and an exact check after sorting still decides what is printed.
Numerical behavior is where Fastmash differs most from a straightforward port.
- Representation. Values are 80-bit extended precision (64-bit significand), the same range and precision GNU datamash uses on x86-64, so results agree closely with it.
- Software arithmetic. Instead of the x87 floating-point unit and the C math library, Fastmash implements the arithmetic itself. This is what delivers portable results: the output does not depend on the CPU model, the compiler, or the libm version.
logandexp. The geometric mean, the normality tests and similar statistics need transcendental functions. Fastmash aims for the correctly rounded 80-bit result: fast paths prove their result with a certificate and otherwise fall back to arbitrary-precision evaluation. A few boundary cases that cannot be settled within fixed limits are refused (status 77).- Ordered evaluation. Sums are accumulated in input order and formulas follow the same order of operations as GNU datamash, because reordering floating-point arithmetic changes results.
- Versioned numerical rules. Parsing, arithmetic, special values, formatting and refusals are specified together as a numerical profile. Any change to observable numerical output is a deliberate, versioned change listed in the changelog.
- Fast paths. Common cases, such as decimal inputs that can be summed exactly, take specialized paths that give the same result faster.
| Status | Meaning | Examples |
|---|---|---|
| 0 | Success | |
| 1 | Error in the command or its input | Unknown option, non-numeric value, missing field, write failure |
| 77 | Refusal | Unsupported feature or locale, checked allocation failure, numerical capacity reached |
| 70 | Internal failure | A violated numerical invariant or a failed numerical table identity check. Please report it |
Diagnostics are written to standard error as fastmash: message, with a colored
✗ prefix on eligible terminals, sometimes
followed by a hint: line suggesting a fix. When one failure follows another
(for example a failed write of the output after an error), the more serious
status wins: 70 over 77 over 1 (failure::combine). A diagnostic that cannot
be written makes the status 1. Other internal failures are reported with
status 77 (for example an internal conversion failure) or, since panics abort,
end the process with SIGABRT. All of them are bugs.
- Audited startup.
startup.rscaptures originally closed standard streams before Rust can repair them. GNU Linux builds also use its process entry and initial SIGPIPE setup. Musl and macOS use Rust's normal runtime entry and argument initialization. ADR 0017 records this narrowly approved unsafe-code exception. Every unsafe operation explains its soundness locally. Ordinary standard-stream I/O uses safe interfaces instandard_stream_io.rsandstandard_io.rs. - Safe runtime interfaces. Color preferences use the same safe standard
environment reads as other configuration. The macOS support boundary comes
from the deployment target and native checks. Commands do not query the
product version at startup. Normal SIGPIPE restoration uses the safe
sigpipedependency. Signal-mask observation and blocked-SIGPIPE refusals remain. Configuration and argument allocations use standard Rust allocation behavior. - GNU Linux allocator controls.
malloc_arenas.rsretains the existingmallopt(M_ARENA_MAX, arenas)andmalloc_trim(0)calls under a separate, narrow exception in ADR 0020. The binary package denies unsafe code, with allowances on those two blocks and the audited startup module. The conversion, numeric_contract, portable_numerics and sort_process modules forbid unsafe code. Sort run files use checked buffer access, record arena cache hints use a safe dependency API, and decimal parsing uses safe integer division. - Overflow checks stay on in release builds, and panics abort.
- Native entropy uses safe dependency interfaces to
getrandomon Linux andgetentropyon macOS. Invocation seeds and private temporary names fail when required entropy is unavailable. Hash grouping keeps its existing fixed fallback when an internal table seed is unavailable. - No network, shell or child processes. Fastmash performs calculations and sorting inside its single executable.
- Dependencies use committed lockfiles. See
Cargo.toml.
| Dependency | Why |
|---|---|
rustc_apfloat |
Software IEEE arithmetic for 80-bit values |
astro-float-num |
Arbitrary-precision arithmetic for log and exp |
icu_collator, icu_locale_core |
Language-locale text ordering from pinned Unicode data |
memchr |
Fast byte searching for record and field splitting |
unicode-width |
Displayed column widths for aligned terminal results |
num-bigint, num-integer, num-traits |
Exact decimal conversion |
base64, md-5, sha1, sha2 |
The base64 and checksum operations |
libc |
The C entry point and signal setup |
sigpipe |
Safe restoration of normal Unix broken-pipe handling, using the existing libc dependency |
| To understand | Start with |
|---|---|
| The overall flow | crates/cli/src/main.rs |
| Option parsing | crates/cli/src/options.rs |
| The operations: names, spellings, what each needs and shares | crates/cli/src/operation.rs |
| command syntax: operations, selectors and modes | crates/cli/src/grammar.rs |
| Planning and shared work of a command's operations | crates/cli/src/operation_set.rs |
| Binding named fields and grouping keys to the input header | crates/cli/src/binding.rs |
| Reading records: comments, vnlog, the input header, read errors | crates/cli/src/intake.rs |
| Records and field splitting | crates/cli/src/records.rs |
| The sort route | crates/cli/src/sort_route.rs |
| Hash grouping, and holding piped input to read it again | crates/cli/src/projected_hash.rs, replay.rs |
| Sorting and spill | crates/cli/src/projected_sort.rs, projected_spill.rs |
| A statistic, for example quantiles | crates/cli/src/ordered_statistics.rs |
| Output and exit status | crates/cli/src/command_output.rs, buffered_stdout.rs, failure.rs |
| Number parsing and printing | crates/cli/src/conversion/. Numeric fields: crates/cli/src/decimal.rs |
| Arithmetic | crates/cli/src/portable_numerics/mod.rs, crates/cli/src/numerics.rs |
log and exp |
crates/cli/src/mean_math.rs, guarded_log.rs, boundary_exp.rs |
Architecture decisions are recorded in design decisions. Read them before proposing a change to one of the principles above.