rmlx baseline reads the prompt fixture with a plain std::fs::read_to_string and tokenizes the whole file (JSON envelope + syntax), not the extracted message content (crates/rmlx-cli/src/commands/baseline.rs).
Repro
rmlx baseline --model <bonsai> --prompt-tokens 65536 --kv-quant none --max-ctx 65536 --record against a chat-JSON fixture (longctx_64k.json, real content ~63–65k tokens) tokenizes to 69667 tokens — over both --max-ctx 65536 and the default --max-prompt-tokens cap. Post-#223 ("error loudly on GPU prompt truncation") this is now a hard error, not the old silent-halve, so the KV-MB capture aborts.
Impact
The bench KV-MB path (docs/models/bonsai/8B/rMLX.md protocol) breaks at 64k unless --max-prompt-tokens 65528 is passed to force the old truncate-to-cap behavior. The HTTP chat-completions path (run_one) is unaffected — it sends messages and applies the real chat template.
Fix
baseline should extract message content from a chat-JSON fixture (apply the chat template / pull messages[].content) before tokenizing, matching what the serve path does — so --prompt-tokens N means N content tokens, not N file-text tokens. Found during the post-0.3.0 Bonsai KV-matrix re-validation.
🤖 Generated with Claude Code
rmlx baselinereads the prompt fixture with a plainstd::fs::read_to_stringand tokenizes the whole file (JSON envelope + syntax), not the extracted message content (crates/rmlx-cli/src/commands/baseline.rs).Repro
rmlx baseline --model <bonsai> --prompt-tokens 65536 --kv-quant none --max-ctx 65536 --recordagainst a chat-JSON fixture (longctx_64k.json, real content ~63–65k tokens) tokenizes to 69667 tokens — over both--max-ctx 65536and the default--max-prompt-tokenscap. Post-#223 ("error loudly on GPU prompt truncation") this is now a hard error, not the old silent-halve, so the KV-MB capture aborts.Impact
The bench KV-MB path (
docs/models/bonsai/8B/rMLX.mdprotocol) breaks at 64k unless--max-prompt-tokens 65528is passed to force the old truncate-to-cap behavior. The HTTP chat-completions path (run_one) is unaffected — it sendsmessagesand applies the real chat template.Fix
baselineshould extract message content from a chat-JSON fixture (apply the chat template / pullmessages[].content) before tokenizing, matching what the serve path does — so--prompt-tokens Nmeans N content tokens, not N file-text tokens. Found during the post-0.3.0 Bonsai KV-matrix re-validation.🤖 Generated with Claude Code