Performance-aware topology planner + placement simulator (merged planner, cost model, scenario corpus)

[i386]

Status: design proposal · Date: 2026-08-26 · Owner: TBD

Goal

Replace capacity-only split placement with one performance-aware planner that consumes the complete input set — memory, sustained node performance, directed link latency/bandwidth, model-family legality, workload intent, and operational stability — plus a simulator to evaluate placement decisions under synthetic conditions without a cluster.

Origin: Buzz skippy-topology channel discussion, 2026-08-26 (James). Full design: docs/design/PERFORMANCE_AWARE_TOPOLOGY_PLANNER.md (PR #1454).

Problem

Today's planner (crates/skippy-coordinator/src/topology.rs) places contiguous layer ranges by memory arithmetic only; its sole network term is stage_count × max RTT. The fleet already gossips gpu_mem_bandwidth_gbps and gpu_compute_tflops_fp16/fp32, and skippy-topology already models directed edges (StageEdgeSignal) — both are dropped before the planner sees them. Measured cost: 68 tok/s solo → 21 at 2-way → 12-13 at 3-way on documented Wi-Fi hardware (docs/BENCHMARKS.md).

Non-goals

• No stage-runtime / wire-protocol / llama.cpp changes (placement only)
• Contiguous layer pipelines only — no tensor/pipeline-parallel hybrid graphs
• No live adaptive replanning until the cost model is calibrated (phase 5)

Workstreams

1. Phase 0 — plumbing: thread gossiped perf metrics through SplitTopologyPlanInput → TopologyNode; instrument observed stage timings. No behavior change.
2. Phase 1 — merged scoring: cost model (mem-bw/compute + directed edge time; pipeline TPOT = max over stages) in skippy-coordinator, legality from skippy-topology; absent-signal fallback must be bit-identical to current placement.
3. Phase 2 — placement sim: deterministic scenario → planner → assert, in CI. Property tests: slower link moves the boundary; half-bandwidth node gets fewer layers; absent signals reproduce current placement.
4. Phase 3 — execution sim: discrete-event pipeline, workload traces, TTFT/TPOT/throughput curves. Calibration gate: reproduce docs/BENCHMARKS.md ratios within tolerance.
5. Phase 4 — default-on after staging A/B vs capacity-only.
6. Phase 5 — adaptive replanning with hysteresis + migration budgets.

Scenario corpus

Node tiers from consumer laptops (CPU/iGPU) through A100/H100/MI300X; link tiers from loopback through intercontinental WAN, directed/asymmetric edges first-class; (hardware, backend, quant) triples; spec priors upgraded to mesh-llm-gpu-bench measurements over time. In-repo TOML shared by CI, planner tests, and sims.

Acceptance gates

• Parity: signal-less inputs produce identical plans to the current planner
• Calibration: execution sim reproduces BENCHMARKS.md anchors within tolerance
• A/B: no regression vs capacity-only on staging meshes before default-on

Cross-team: simulator data realism (consumer → datacenter, multi-backend) flagged for @mic's agents once this issue lands.

[i386]

Design doc PR: #1454

[michaelneale]

🤖 Deep review of PR #1454 at head 5f3bedaba (design doc + planner + simulator), requested in the Buzz skippy-topology channel. Reviewed the full diff against merge-base eb209afea, plus where main (d4239e4cb) has moved since. Overall: the architecture is right — real planner under test, capacity-only fallback discipline, integer µs math, property-style scenario assertions — but there are four clusters of problems that between them decide whether this planner can be trusted on a heterogeneous fleet. Findings below with file:line references.

1. Ranking integrity: unmeasured subsets systematically win ties (the biggest correctness cluster)

The 5f3bedaba target-ordering fix is correct as far as it goes, but the surrounding comparison logic still rewards missing data:

• Asymmetric target_met scoring. Perf-aware plans compute decode_tpot_target_met from the full modeled TPOT (topology.rs:547), while capacity-only plans compute it from the network-only estimate (topology.rs:613). target_met is the first ordering key in latency_candidate_ordering, so a subset that merely lacks a bandwidth signal takes a strictly easier target test and can outrank a faster fully-signaled plan. Withholding perf signals improves a subset's ranking — the exact inverse of the incentive we want.
• decode_tpot_target_met.unwrap_or(true) (topology.rs:686): a candidate whose target status is unknown (no RTT data at all) is treated as meeting the target. Unknown should rank below Some(false) or at minimum below Some(true).
• Mixed-signal same-shape comparison (candidate_better_for_same_shape, topology.rs:644-668): the comment claims same-shape candidates always both carry modeled TPOT, but best_for_count compares candidates from different node subsets of the same count — one can be fully signaled and one not. In that case both fall through to estimated_decode_network_ms_per_token.unwrap_or_default(), where None becomes 0 ms and an unmeasured candidate beats a measured one.
• Capacity-path ms flooring (topology.rs:807, :613): the modeled path got µs precision in 5f3bedaba, but the capacity path still floors µs→ms before the target comparison, so a plan up to 1 ms over target can be marked met.

Suggested direction: make target-met a three-state ordering (Some(true) > Some(false) > None or measured-above-unmeasured), never compare target flags computed from different metrics, and score the capacity path in µs.

• DP objective mismatch (topology.rs:1046-1140): commit 5ea0f031 deliberately moved the ranking metric to the serial model (Σ stage times + Σ hops) because bottleneck-form under-priced multi-stage plans — but perf_balanced_spans still picks the span split minimizing max stage time, with Σ only as tie-break. Under the serial model the optimum loads more layers onto faster nodes; the DP is optimizing a different objective than the one candidates are ranked on, so it can return provably non-optimal spans for the chosen subset. Either flip the DP primary key to the sum (max as tie-break) or document why bottleneck stays primary.

2. Trust boundary: gossiped performance signals are only sanitized on the sending side

• sanitize_model_throughput_hints (metrics.rs:239, clamp to 10 s/layer) runs only in snapshot_local_announcement_data (gossip.rs:1074). Inbound announcements are stored verbatim (gossip.rs:365, :809) and consumed unclamped (local_package.rs:435-441). A malicious peer can advertise a near-u32::MAX bandwidth CSV and the planner will attract the bulk of the layers to it — a mesh-wide decode DoS by attraction. (Lying low on µs/layer is harmless since observed timing is a max floor — that part of the design is good.) Fix: re-run sanitization + plausibility clamps on receipt in apply_announcement.
• Batched decode inflates the measured floor (stage_performance.rs:37-56 + connection.rs:438-455): record_stage_decode_timing divides by layer count but never by the executed token count, so a batch of 8 records ~8× the true per-token per-layer time, and the planner uses this as a floor — pessimizing or rejecting good plans on exactly the nodes doing the most work.
• Stale-observation resurrection (stage_performance.rs:61-88): entries older than the 30-min window are filtered from hints but never evicted/reset in the map; after idle, new samples blend at 1/8 (EWMA) or 1/(n+1) weight against arbitrarily old data. Reset to (sample, 1) when the prior observation exceeds MAX_OBSERVATION_AGE, and evict aged entries (the map also grows unboundedly keyed by coordinator-supplied model_id).
• No hysteresis on the timing signal: the exact observed µs value participates in split_participant_set_hash (local_package.rs:945-961), so every gossip round's jitter churns the settle hash and can ratchet span boundaries back and forth. Quantize the advertised value into buckets and/or require a minimum modeled-TPOT improvement before adopting a re-balanced plan. This shouldn't wait for phase 5 — the churn exposure exists as soon as observed timing feeds placement.

3. Staleness vs main (rebase needed, and it changes numbers)

The PR is CONFLICTING with main but a test merge shows only 2 conflicting files (mesh/gossip.rs, console_print_allowlist.json) — the rebase itself is cheap. What the rebase invalidates is not:

• Wire-byte pricing is 2× understated. split_planning.rs:414 prices frames at activation_width × 2 (f16). Main moved to raw f32 transport in fix(skippy): use raw f32 activation transport #1482 (7369cfd46, ancestor of main, not of this branch), so the correct figure is ×4 — ~2 ms/hop error on a 5 MiB/s uplink, exactly the fixture class this PR tests. Per the prior channel discussion: price the wire contract analytically (f32 + sideband multipliers) rather than binding to the post-evaluation boundary ABI, which isn't available at planning time.
• Calibrated overheads are stale. CALIBRATED_PER_HOP_OVERHEAD_US = 13_000 / PER_STAGE = 1_300 were back-solved from BENCHMARKS.md anchors measured before perf(skippy): cut per-token split decode overhead (~10%) #1575 (16bd1db8a) removed a ~5 ms/token poll-quantization toll from the direct-return path. The anchors need re-measurement post-rebase; ideally the constants become clearly-versioned against a benchmark date.
• Boundary metadata sourcing. Main now negotiates real activation boundaries from the built graph (Make Skippy negotiate real activation boundaries #1585, a3a82f7bd). The design doc's cost model should state that frame bytes ultimately come from artifact-derived boundary contracts (issue direction already agreed in-channel), with analytical f32 pricing as the interim.
• Synthesized pair-edge claim is wrong (split_planning.rs:459-467): the comment says taking max of the two coordinator RTTs "can never underestimate a real hop". Triangle inequality gives sum, not max, as the bound — two nodes each 5 ms from the coordinator can be 100+ ms apart. Use the sum (or a calibrated multiplier) and treat these edges as estimates, not bounds.

4. Simulator: right architecture, but currently a fit — not a validation

Strengths first: the sim drives the production plan_topology (no reimplementation), asserts behavioral properties rather than golden numbers, is deterministic/hermetic/CI-wired, and has a planner↔sim consistency test. That's the right skeleton. But:

• 4 free knobs vs 3 same-family anchors. active_weight_fraction=0.34, per_stage_overhead_ms=1.3, per_hop_overhead_ms=13.0, plus an "effective" M4 Max bandwidth of 417000 MiB/s (while heterogeneous_pair.toml uses 546000 for the same hardware class). All three anchors are GLM-4.7-Flash Q4_K_M on Apple Silicon over one Wi-Fi network. The system is underdetermined: essentially any monotone additive model passes the ±15% tolerance. The Sydney↔Queensland 20 ms measurement already in BENCHMARKS.md is an unused out-of-family holdout — use it. Then adopt a holdout policy: new measured benchmarks land as prediction tests with frozen coefficients before they may be used for re-fitting.
• No prefill model. sustained_compute_gflop_per_s is parsed into the schema but never used by the execution layer. Prefill is compute-bound and its ranking can differ from decode ranking (a 4090 vs M-series gap is ~5-8× at prefill vs ~2.3× at decode). This is a ranking-relevant omission, not just an absolute-number one.
• active_weight_fraction applied to all layer bytes including dense attention/norm weights — it's a fitted lump of expert sparsity + attention density + quant efficiency and won't transfer across MoE architectures. (Also execution.rs:23-24 says ~7.2 GB/token while the anchor TOML says ~5.8 — the comments contradict each other.)
• Nested schema is not strict. Top-level unknown-key and malformed-link-key rejection works and is tested, but no struct carries #[serde(deny_unknown_fields)] — a typo like active_weight_fractoin silently defaults to dense (≈3× stage-time error), and a typoed bandwidth key silently flips a node into capacity-only fallback. For a calibration tool this is the biggest correctness trap in the crate; it's a one-line fix per struct.
• Duplicated cost model. execution.rs::estimate_execution and topology.rs::modeled_serial_decode_tpot_us implement the same formula twice with independently sourced coefficients, pinned by a 1% parity test on one scenario. And the sim ignores observed_decode_us_per_layer entirely while the planner takes max(analytical, observed) — set that field in any scenario and planner and sim disagree immediately. Run the parity test across every scenario and over randomized span assignments, and single-source the coefficients.
• Corpus gaps (schema is the binding constraint): no asymmetric-link scenario (and execution.rs:60-64 silently falls back to the reverse direction when a directed link is missing — should error), max 3 nodes, no memory-tight fit, no missing-signal TOML scenario, no dense-vs-MoE placement contrast, no mid-run straggler/replan scenario, no concurrency/prompt-length/objective vocabulary in the workload schema.

Suggested priority order

1. Ranking fixes (target-met three-state, mixed-signal comparisons, µs precision on capacity path, DP objective) — these decide whether plans are correct at all in mixed fleets.
2. Receipt-side sanitization + batch normalization + stale-reset — required before the observed-timing floor is trustworthy, and the bandwidth clamp is a security issue.
3. Rebase + f32 frame pricing + re-measured anchors — cheap merge, but the numbers must follow fix(skippy): use raw f32 activation transport #1482/perf(skippy): cut per-token split decode overhead (~10%) #1575.
4. Sim rigor: deny_unknown_fields, out-of-family holdout anchor, prefill term, sim honors observed floor, parity test across the corpus.
5. Corpus/schema growth + hysteresis — feeds directly into the phase-4 A/B gate and phase-5 replanning.

Also worth an explicit note in the doc/plan: minimal quantized hysteresis on observed signals should ship with the observed-timing floor, not be deferred wholesale to phase 5, because the churn exposure starts now.

Done well and worth keeping exactly as-is: the all-or-nothing bandwidth gating with bit-identical capacity fallback (tested), the kill-switch preserving pre-existing RTT/TPOT fields, RTT settle-window corroboration before trusting perf signals, observed timing as a max floor (under-reporting is a no-op), u128 µs arithmetic, and behavioral scenario assertions.