doctor split reports split_capable: false / unknown_model_size for a layer package that then splits and serves

[michaelneale]

Summary

mesh-llm doctor split reports verdict unknown_model_size with split_capable: false for a layer package that then forms a locked two-node split and serves successfully. An operator reading that verdict would reasonably abandon a topology that works.

Observed on a clean worktree at fb58bf73ba375fba1f1921c432bcb0d4ac02ac2b (0.76.0-rc4), Metal release bundle, two Macs on a LAN.

Reproduce

# node A (coordinator)
mesh-llm serve --model 'meshllm/Qwen3.5-35B-A3B-UD-Q4_K_XL-layers@69cbd9fc1f928305e8e86e1148a1db6a6f1de8e9' \
  --split --max-vram 24 --ctx-size 8192 --port 9557 --console 3342 --bind-port 7852 \
  --split-topology-lock lock.json
# node B joins with the printed token, same package ref, --max-vram 11

mesh-llm doctor split \
  --model-ref 'meshllm/Qwen3.5-35B-A3B-UD-Q4_K_XL-layers@69cbd9fc1f928305e8e86e1148a1db6a6f1de8e9' \
  --port 3342 --output-dir ./split-doctor

Output before load:

🩺 Split readiness: unknown_model_size
Eligible participants: 2
Excluded peers: 0

split-readiness.json:

{
  "verdict": "unknown_model_size",
  "participant_count": 2,
  "exclusion_count": 0,
  "blockers": [],
  "capacity_advice": {
    "state": "unknown_model_size",
    "reason": "model_size_unknown",
    "split_capable": false,
    "eligible_node_count": 2,
    "missing_capacity_node_count": 0,
    "aggregate_capacity_bytes": 35000000000,
    "best_single_node_capacity_bytes": 24000000000
  }
}

The split then formed on the locked ranges 0..27 / 27..41, both stages reached ready, and inference returned a correct answer.

Mechanism

Two distinct problems, both rooted in the same missing catalog field.

1. split_capable: false is not "not split capable", it is "unknown".

evaluate_model_target_capacity derives it from the size hint:

// crates/mesh-llm-host-runtime/src/api/model_target_capacity.rs:158
let split_capable = size_hint.map(|hint| hint.split_capable).unwrap_or(false);

When the lookup misses, unwrap_or(false) collapses unknown into a definite no. The state field correctly says unknown_model_size, but split_capable next to it asserts a negative that was never determined. Any consumer reading the boolean rather than the state gets the wrong answer.

2. The lookup misses because layer-package aliases require total_bytes.

ModelTargetSizeLookup::from_entries only registers a layer-package alias when the catalog supplies total_bytes:

// model_target_capacity.rs:77-89
for package in variant.packages.iter().filter(|p| p.package_type == "layer-package") {
    if let Some(model_bytes) = package.total_bytes {
        lookup.insert_package_alias(&package.repo, ModelSizeHint { model_bytes, split_capable: true });
    }
}

The cached catalog entry for this package (~/.cache/meshllm/catalog/entries/unsloth/Qwen3.5-35B-A3B-MTP-GGUF.json) has no total_bytes:

"packages": [
  { "type": "layer-package", "repo": "meshllm/Qwen3.5-35B-A3B-UD-Q4_K_XL-layers", "layer_count": 41 }
]

The variant-level fallback cannot rescue it either: curated.size is the string "41 layers", and parse_size_label_bytes rejects it (unit "layers" is not in the multiplier table), so insert_model_aliases is skipped too. Result: size_hint = None -> required_bytes = None -> UnknownModelSize, and split_capable defaults false.

Note the doctor already holds enough information to size the package — the manifest it validates lists every layer file — but the capacity path only consults the catalog.

Suggested fix

Two separable changes:

• Do not report a boolean where the answer is unknown. When the state is UnknownModelSize / UnknownCapacity, split_capable should be absent or tri-state rather than false. The illegal state here is "definitely not split capable, for a reason that is definitionally about not knowing".
• Size layer packages from the manifest when the catalog omits total_bytes, or have catalog generation always populate it for layer packages. Either removes the miss entirely.

A narrower interim fix: since a layer-package entry is by construction split capable, split_capable: true could be set from the package type alone, independent of whether the byte count resolved.

Also observed

After the split was serving, the same command returns verdict ready / already_serving but still split_capable: false — the AlreadyServing branch returns early (split_readiness.rs:324-326) carrying the same defaulted boolean.

Evidence

Full doctor bundle, both /api/runtime/stages responses, the topology lock, per-node fetched-file inventories, native load lines, and the inference response are captured. Happy to attach on request.

[mikemikimike]

Implemented in PR #1558.

• Changed: split capability is tracked independently from catalog size. Layer-package aliases remain recognized as split-capable when total_bytes or curated size metadata is missing, including source, revision-qualified, variant-name, and package references. Truly unknown capability is omitted instead of reported as false.
• Preserved behavior: capacity advice remains unknown_model_size until model bytes are known, and known split/non-split values continue to drive the existing capacity calculations.
• Tests: focused model_target_capacity tests pass (5/5); formatting, package check, and clippy pass with existing warnings. The full Windows workspace run is limited by the repository's missing patched llama.cpp quant archives, and the host-runtime full run has unrelated Windows/path/snapshot failures in untouched modules.