Skip to content

preflight.md

Latest commit

 

History

History
417 lines (293 loc) · 21.2 KB

preflight.md

File metadata and controls

417 lines (293 loc) · 21.2 KB
Raw

Preflight

preflight is Primus' cluster diagnostic tool. It produces:

  • A fast info report (host / GPU / network configuration), and
  • A configurable suite of performance tests (GEMM TFLOPS, intra-node and inter-node communication bandwidth, P2P, ring P2P).

Use it to spot misconfiguration, hardware degradation, or perf outliers before committing a large distributed training run to a global rendezvous.

  • User-facing entry: primus-cli ... -- preflight [args]
  • No-container launcher: runner/primus-cli direct -- preflight ... — see preflight-without-container.md.
  • Implementation entrypoint: primus/cli/subcommands/preflight.py → primus/tools/preflight/preflight_perf_test.py.

Looking for a faster, distributed-rendezvous-free per-node screen? See node-smoke-test-instruction.md. The recommended workflow is smoke first, preflight second — see §10 Comparison with node-smoke.

1. Two run modes (and how preflight picks one)

Preflight has two report types, controlled by a single precedence rule:

Mode Triggered by What it does
Info-only --host, --gpu, --network (in any combination) Lightweight host / GPU / network introspection. Emits a per-node report without requiring a rendezvous; multi-node aggregation then uses a timeout-bounded rendezvous (--dist-timeout-sec), so it never hangs indefinitely on network misconfig.
Perf-only --perf-test, --tests ..., or --quick Runs the configured perf tests under a global rendezvous. Implied by --tests and --quick.
Default (info + perf) No flags at all Runs the info report first, then every perf test.

Mode precedence

  1. Any of --perf-test / --tests / --quick is set → perf-only mode. If info selectors (--host/--gpu/--network) are also present, they are dropped and a WARN is emitted (also written as a > Note: at the top of the perf report). To get both reports, run two invocations.
  2. Otherwise, any of --host/--gpu/--network is set → info-only mode. Perf-only tuning knobs (e.g. --comm-sizes-mb) are inert in this mode and trigger a single WARN listing them.
  3. Otherwise (no flags) → default: info report first (no rendezvous), then perf tests.

The default order ensures you always get a report even if torch.distributed initialization later hangs.

2. Quick start

Info report only (fast)

primus-cli direct -- preflight --host --gpu --network

Full preflight (info + every perf test)

primus-cli direct -- preflight

Perf tests only

primus-cli direct -- preflight --perf-test

Fast pre-launch sanity check

primus-cli direct -- preflight --quick

Equivalent on SLURM via primus-cli slurm:

primus-cli slurm srun -N 4 -- preflight --quick

Without a container, see preflight-without-container.md for the equivalent runner/primus-cli direct -- preflight ... invocations.

3. Test selection (--tests)

--tests takes a comma-separated list of canonical tokens (or all). Implies --perf-test.

Token What it runs
gemm Single-GPU square GEMM TFLOPS sweep.
intra-allreduce Intra-node all_reduce bandwidth at every selected --intra-group-sizes x --intra-comm-sizes-mb.
intra-alltoall Intra-node all_to_all bandwidth, same configuration matrix.
inter-allreduce Inter-node all_reduce bandwidth at every selected --inter-group-sizes x --inter-comm-sizes-mb.
inter-alltoall Inter-node all_to_all bandwidth, same configuration matrix.
inter-p2p Inter-node point-to-point send/recv between fixed adjacent 2-node pairs (does not use --inter-group-sizes). Sized by --inter-comm-sizes-mb, falling back to --comm-sizes-mb.
inter-ring-p2p Inter-node ring-pattern P2P, sized by --ring-p2p-sizes-mb.
all Every token above. Default when --tests is omitted.

Examples:

# GEMM only
primus-cli direct -- preflight --tests gemm

# Just the inter-node bandwidth tests
primus-cli direct -- preflight --tests inter-allreduce,inter-alltoall

# Combine with size overrides
primus-cli direct -- preflight \
    --tests gemm,inter-allreduce \
    --comm-sizes-mb 64,1024 \
    --inter-group-sizes all

Unknown tokens fail fast (before any rendezvous):

[Primus:Preflight] ERROR: invalid perf config: --tests: unknown token 'gem'.
Valid tokens: gemm, intra-allreduce, intra-alltoall, inter-allreduce,
inter-alltoall, inter-p2p, inter-ring-p2p, all

4. Quick preset (--quick)

--quick is the recommended pre-launch sanity preset. Implies --perf-test. It substitutes:

Knob --quick value
--tests gemm,intra-allreduce,inter-allreduce
--comm-sizes-mb 64,1024
--intra-group-sizes LOCAL_WORLD_SIZE (full intra-node group only)
--inter-group-sizes all (full N-node group only)
warmup 5
iteration 20

User-supplied flags override the preset. For example:

# Quick preset, but with a custom size set
primus-cli direct -- preflight --quick --comm-sizes-mb 32,256

A full perf run with default knobs takes minutes; --quick typically finishes in <60s on healthy hardware.

5. Tuning the perf tests

All perf tuning knobs default to None so preflight can tell whether you set them. When unset, the documented defaults below apply.

5.1 Message sizes (collective + P2P)

Flag Default Applies to
--comm-sizes-mb CSV 2,4,8,16,32,64,128,256,512,1024 Default for both intra- and inter-node allreduce / alltoall and inter-p2p when no specific override is given.
--intra-comm-sizes-mb CSV falls back to --comm-sizes-mb Override for intra-node allreduce / alltoall.
--inter-comm-sizes-mb CSV falls back to --comm-sizes-mb Override for inter-node allreduce / alltoall / inter-p2p. 
# Smaller, focused sweep
primus-cli direct -- preflight --comm-sizes-mb 8,128

# Different sizes for intra vs inter
primus-cli direct -- preflight \
    --tests intra-allreduce,inter-allreduce \
    --comm-sizes-mb 8,128 \
    --intra-comm-sizes-mb 4,32

5.2 Group sizes

Flag Default Notes
--intra-group-sizes CSV 2,4,8 Each value must divide LOCAL_WORLD_SIZE.
--inter-group-sizes CSV 2,4,all all means the full N-node group. Other values are subgroup sizes. Only inter-allreduce and inter-alltoall consult this flag — for inter-alltoall, every requested per-group node count is internally capped at 16 before deduping (see "Inter-node alltoall is capped at 16 nodes" below), while inter-allreduce uses the requested sizes unchanged. inter-p2p and inter-ring-p2p ignore this flag (fixed adjacent 2-node pairs and a full-cluster ring, respectively). 
# All-GPU intra + full N-node inter only
primus-cli direct -- preflight \
    --tests intra-allreduce,inter-allreduce \
    --intra-group-sizes 8 \
    --inter-group-sizes all

Validation is gated by which tests are actually selected. For example, --tests gemm --intra-group-sizes 3 does not abort on a host with LOCAL_WORLD_SIZE=8; the intra-group constraint is only checked when an intra test is enabled.

Inter-node alltoall is capped at 16 nodes

Regardless of the cluster size or what --inter-group-sizes requests, the inter-alltoall test always runs on per-group node counts of at most 16. Concretely, every requested value G is replaced with min(G, 16), and the resulting list is deduped. Examples:

Cluster --inter-group-sizes Requested (resolved) inter-alltoall actually runs
8 N all [8] [8] (no change)
64 N 2,4,all [2, 4, 64] [2, 4, 16]
128 N 2,4,16,32,all [2, 4, 16, 32, 128] [2, 4, 16]
128 N 64 [64] [16]

When the cap actually changes the list, preflight emits a single one-line WARN to stdout so the row labels in the report (e.g. alltoall-16nodes instead of alltoall-128nodes) are not surprising.

Why the cap, and why 16:

  • It matches real-world usage. Production MoE training rarely dispatches tokens across more than ~8 nodes (for example, DeepSeek-V3's largest published configuration uses EP=64 over 8 nodes with per-token dispatch capped at 4 nodes). A 16-node ceiling covers every published configuration with comfortable headroom.
  • It keeps the test from exhausting per-node network resources. A large inter-alltoall sub-group opens a near-full mesh of connections per rank during communicator setup. Capping it at 16 keeps that well within a node's ephemeral-port budget and avoids spurious Address already in use failures at scale (see §7).
  • Other inter-node tests are unaffected. The cap applies only to inter-alltoall: inter-allreduce uses --inter-group-sizes unchanged, while inter-p2p and inter-ring-p2p don't consult it at all. All three also open far fewer simultaneous connections than alltoall.
  • It is intentionally not configurable. This is a known-safe ceiling for the communication shapes preflight characterizes, not a tuning knob.

5.3 Ring P2P sizes

Flag Default Applies to
--ring-p2p-sizes-mb CSV 10,20,40,80,160 inter-ring-p2p only. 
primus-cli direct -- preflight \
    --tests inter-ring-p2p \
    --ring-p2p-sizes-mb 5,20,80

5.4 Plotting

Flag Effect
--plot After each perf test, write per-size bandwidth bar charts under <dump-path>/<test>/ and reference them in the markdown report.

6. Reliability knobs

Two knobs that are inert under happy-path conditions but matter at scale or on flaky networks.

6.1 --comm-cleanup-delay-sec FLOAT (default 2.0)

Delay (seconds) inserted between destroying NCCL/RCCL process groups and creating new ones. It provides cross-rank synchronization across the destroy → setup transition, so a rank doesn't try to connect to a peer whose listener hasn't finished closing.

  • Default 2.0 is essentially free and worth keeping at every cluster size.
  • Set to 0 to disable the sleep entirely (barrier only).
  • Bump to e.g. 5 only on very flaky networks.
# Small/medium clusters: the default is fine. Override only if you
# see port-reuse races on a very flaky network.
primus-cli slurm srun -N 8 -- preflight --quick --comm-cleanup-delay-sec 5

See §7 for guidance on running at very large scale.

6.2 --dist-timeout-sec INT (default 120)

Timeout (seconds) for torch.distributed.init_process_group. If init does not complete within this many seconds, preflight writes the info report (when applicable) plus a Distributed Init failure section to the markdown report, prints a clear error, and exits 2 — instead of hanging indefinitely.

# Fail fast if rendezvous does not work
primus-cli direct -- preflight --perf-test --dist-timeout-sec 30

7. Running on very large clusters (≥ 64 nodes)

At very large scale there are a few practical considerations beyond what smaller runs encounter. A default preflight invocation still runs correctly at every scale we test (up to 128 nodes) without special flags — the points below are limitations to be aware of, plus recommendations that make large-cluster runs faster and easier to interpret.

7.1 Limitations

  • inter-alltoall is measured on at most 16 nodes per sub-group. Regardless of cluster size or --inter-group-sizes, the alltoall test is capped at 16-node sub-groups (see §5.2). This is intentional — it matches real-world MoE dispatch patterns and keeps the test from exhausting per-node network resources during communicator setup — but it does mean preflight will not report alltoall bandwidth for a larger topology. The cap applies only to inter-alltoall: inter-allreduce honors --inter-group-sizes unchanged, while inter-p2p and inter-ring-p2p don't use it at all.
  • preflight briefly builds many communicators. Unlike a real training job — which creates its communicators once at startup and reuses them — preflight repeatedly builds and tears down large communicators in a short window. On a cluster with an unusually narrow ephemeral-port range this can occasionally surface as Address already in use during setup. It is a preflight-specific artifact rather than a training failure mode; the one-line OS fix is in §7.3.

7.2 Recommended: split large runs by test family

For clusters at or beyond ~128 nodes, run one test family per invocation instead of one large run. Each invocation stays short, and it becomes easy to see which specific communication shape is degraded if a number looks off.

# 1) GPU + intra-node fabric first (cheap, no inter-node OOB churn).
primus-cli slurm srun -N 128 -- preflight \
    --tests gemm,intra-allreduce,intra-alltoall

# 2) Inter-node DP-style collectives, all-nodes group only.
primus-cli slurm srun -N 128 -- preflight \
    --tests inter-allreduce \
    --inter-group-sizes all
# Note: --inter-group-sizes all is honored here for inter-allreduce.
# For inter-alltoall it would be capped at 16 (see §5.2).
primus-cli slurm srun -N 128 -- preflight \
    --tests inter-alltoall \
    --inter-group-sizes all

# 3) Inter-node PP-style ring P2P (the test that benefits most from
#    isolation — it's the closest match to what real pipeline-parallel
#    training actually exercises).
primus-cli slurm srun -N 128 -- preflight \
    --tests inter-ring-p2p

# 4) Optional: pairwise inter-node P2P scan (useful for finding a
#    single bad link, slower because it walks many pairs).
primus-cli slurm srun -N 128 -- preflight \
    --tests inter-p2p

Each invocation tears down its own WORLD on exit and touches only one --tests value, so you get a per-test wall clock, can re-run a single phase in isolation, and get a separate report per run via --report-file-name.

7.3 Optional OS tuning

preflight runs fine with default OS settings at every scale we test. If you do hit Address already in use on a cluster with a narrow ephemeral-port range, widen the range — this is good general practice for any RDMA host regardless of preflight:

# Widen the per-node ephemeral port range (default ~28k → ~64k ports).
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

# Persist across reboots:
echo 'net.ipv4.ip_local_port_range = 1024 65535' | sudo tee /etc/sysctl.d/99-large-cluster.conf
sudo sysctl --system

8. Reporting

Flag Default Effect
--dump-path DIR output/preflight Output directory for reports + plots.
--report-file-name NAME auto-generated preflight-${NNODES}N-YYYYMMDD-HHMMSS Base name for report files. Omit to let preflight auto-generate a unique timestamped name (prevents stale leftovers from prior runs being mistaken for fresh output). Pass an explicit value when you want a stable / well-known filename.
--disable-pdf enabled Skip PDF generation (Markdown only). Useful when weasyprint/markdown2 aren't installed.

Output files:

File Produced when Notes
<name>.md / <name>.pdf Info-only mode, or default mode Info report.
<name>_perf.md / <name>_perf.pdf Perf-only mode, or default mode Perf report (GEMM + comm).

Only rank 0 writes the report.

Perf report layout

A <name>_perf.md produced by a default run contains, in order:

  1. (Optional) > Note: line listing dropped info selectors.
  2. # Nodes legend — Node N → Hostname table, used by every subsequent table to keep host columns compact.
  3. =======IB Bandwidth roofline (GB/s)======= — bandwidth of the first IB device on Node 0.
  4. Per enabled test, in this order: gemm, intra-comm, inter-comm, inter-p2p, inter-ring-p2p. Each section has a configuration line, a results table (Node / Rank / hostname / per-size GB/s), optional plots, and a per-rank wall-clock summary.
  5. [Primus:Preflight] <test> done in <T>s lines on stdout for at-a-glance progress on the launching shell.

9. Backward-compat aliases

Flag Equivalent Notes
--check-host, --check-gpu, --check-network --host, --gpu, --network Same behavior. Keep working for older scripts.
--no-split-nodes-subgroup --inter-group-sizes all and drops inter-p2p Pre---tests/--inter-group-sizes alias. Use the new flags in new scripts.

10. Comparison with node-smoke

Aspect node-smoke preflight
Rendezvous None — every node independent Global torch.distributed
Wall clock ~30–60 s for 6 nodes (Tier 1+2) Minutes; scales with N for inter-node tests
Granularity Per-node PASS/FAIL Per-rank measurements (no auto-fail by default)
Inter-node bandwidth matrix Not tested (intentionally) Yes (allreduce/alltoall/p2p/ring-p2p)
Drift detection Yes (versions, NIC firmware, port count) No
Host limits / RDMA roll-call Yes (hard fail) Reported via collect_*_info only
Output format Per-node JSON + cluster md + SLURM-ready txt Markdown + PDF

Recommended workflow: run node-smoke first to exclude broken nodes, then run preflight on the surviving set to get cross-node bandwidth measurements. See node-smoke-test-instruction.md §4 ("Quick start") for the integration commands.

11. Validation & error handling

Preflight resolves the perf config before any distributed rendezvous. This means typos and bad sizes/group-sizes fail in seconds, not after a 120s NCCL init:

[Primus:Preflight] ERROR: invalid perf config: --tests: unknown token 'gem'.
[Primus:Preflight] ERROR: invalid perf config:
    --intra-group-sizes: [3] do not divide LOCAL_WORLD_SIZE=8
[Primus:Preflight] ERROR: invalid perf config: --comm-sizes-mb: values must be positive (got 0)

In info-only mode, perf-only tuning knobs trigger a single warning so you notice them but they don't abort:

[Primus:Preflight] WARN: --comm-sizes-mb,--intra-group-sizes have no effect
in info-only mode (no --perf-test/--tests/--quick).

In default mode where info selectors are dropped because perf intent was set, the preserved warning is also written into the perf report header:

> Note: info selectors --host were dropped because perf mode
> (--perf-test/--tests/--quick) takes precedence. Run them in a separate
> invocation if you want both reports.

12. Operational tips

  • For multi-node runs, always use primus-cli slurm or primus-cli direct under srun so distributed environment variables (NNODES / NODE_RANK / MASTER_ADDR) are set correctly.
  • Make sure slurm requests GPU resources. For some clusters, you may need to explicitly request GPU resources with srun -N <nodes> --gpus-per-node=<gpus-per-node>.
  • Insufficient CPU cores cause >30x perf slowdowns — pass srun -c <cores-per-node> so RCCL's network proxy threads have CPU to spawn on. Verify with srun -N 1 --gpus-per-node=8 bash -c 'nproc'.
  • For a quick environment snapshot, prefer --host --gpu --network — you always get a local per-node report even on a broken network, and any multi-node aggregation is timeout-bounded (--dist-timeout-sec), so the command never hangs.
  • Between each communication test phase, preflight performs a global barrier + --comm-cleanup-delay-sec sleep (default 2 s) for cross-rank sync across the destroy → setup transition. The default works at every cluster size we test up to 128 nodes. See §7 for large-cluster guidance.
  • For pre-launch screening of a large cluster, the recommended sequence is:
    1. node-smoke to prune broken nodes (failing_nodes.txt).
    2. preflight --quick on the surviving nodes for the perf sanity numbers.
    3. preflight (full) on the same set if the --quick numbers raise a flag.

13. Running preflight without a container

If you cannot (or prefer not to) use a container, see preflight-without-container.md for the step-by-step runner/primus-cli direct -- preflight ... walkthrough — Python virtual-environment setup, SLURM invocation patterns, NCCL configuration for Broadcom and Pensando (AINIC) clusters, and many configurable-knob examples.

14. See also