Accept a code-span-wrapped route line from the manager - #58
Open
TON14 wants to merge 3 commits into
Open
Conversation
The manager prompt displays every route inside backticks -- "then exactly one
route: `Next: gui`, `Next: cli`, ..." -- and models, most reliably the smaller
ones, copy that formatting into their answer. The parser stripped `*`, so a
bold `**Next: cli**` was accepted, but a code span "`Next: cli`" fell through
to `invalid`: the model then re-plans against feedback that repeats the same
backticked notation, and entire runs burn their round budget on formatting the
harness's own instruction taught the model.
Observed live: a deepseek-v4-flash manager produced a full, valid plan ending
in "`Next: cli`" three rounds in a row; every round was scored invalid and the
run died as max_rounds_exhausted with nothing executed.
Backticks now strip exactly where asterisks already did, including after the
rationale delimiter is cut ("`Next: cli` — reason" leaves a closing backtick
on the route half). Prose mentioning a route mid-sentence stays invalid, as
does an undelimited suffix.
The PR's own history is the argument: the branch was verified green on each platform by hand, and each round of hand-verification still found something the other platform could not see (a POSIX-only test guard, a cmd.exe-only command-line limit). A matrix of ubuntu + windows at both ends of requires-python (3.10 and 3.14) makes that check automatic for every push and pull request. The suite needs no Node toolchain -- the Web bundle is a packaging artifact -- so the job is checkout, setup-python, `pip install -e ".[test]"`, pytest. The Windows symlink fixtures skip themselves on runners without SeCreateSymbolicLinkPrivilege, which is expected and green.
…ands The windows-latest lanes exercise platform support this branch does not carry: it is based on a main whose supervisor still calls os.killpg and whose agent stubs are #!/bin/sh scripts, so those lanes fail on known pre-existing breakage rather than on anything in this change. AMAP-ML#57 brings the Windows support together with the full two-platform matrix; when it merges, its version of this workflow supersedes this one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Accepts a manager route line wrapped in markdown code spans — "
Next: cli" — everywhere the bold form**Next: cli**was already accepted.The bug
The manager prompt itself displays every route inside backticks:
Models — most reliably the smaller ones — copy that formatting into their answer.
parse_role_manager_next_stepstrips*before matching, so bold wrapping is fine, but a code span falls through toinvalid. The round's feedback then repeats the same backticked notation, the model reproduces it again, and the run burns its whole budget on formatting the harness's own instruction taught the model.Observed live (Linux,
deepseek_harnessbackend,deepseek-v4-flashas manager): three rounds in a row produced a full, valid plan ending in "Next: cli"; every round scoredinvalid; the run died asmax_rounds_exhaustedwith nothing executed.The fix
Backticks now strip exactly where asterisks already did — including after the rationale delimiter is cut, since in "
Next: cli— reason" the closing backtick sits before the dash and survives the first strip. Deliberately unchanged: prose mentioning a route mid-sentence ("routeNext: climentioned mid-prose") and undelimited suffixes ("Next: done later") still scoreinvalid.Testing
Next: cli", "Next: cli", and "Next: cli— rationale"; existing tests for delimited rationale and undelimited-suffix rejection pass unchanged.