Repository navigation
feat(specialist): operator-defined custom specialist type - #1804
Conversation
Add a custom_specialist domain whose focus and description come from --custom-specialist-prompt-file / --custom-specialist-description. - System prompt keeps the shared identity, output protocol and iron rules; the operator text replaces the domain focus verbatim. - User prompt carries mandate, hardware, PD, budget and gap, plus the source hint in patch mode. - Orchestration sees the domain in the specialist EMIT list with a one-line selection hint only when configured; PolicyGate denies it otherwise. - The definition persists in SharedState; flags on --resume-from replace it. Conflicting launches (--orch-prompt, FRAMEWORK_AGENT disabled, --research-lane-capacity 0, --reset-state) fail fast. - The Coordinator dispatches it at least once in FRAMEWORK_AGENT, at most once per macro cycle, falling back to research mode when source patches cannot be authored.
- Track the guarantee attempt per macro cycle in memory; admission denials create no task row, so a registry lookup re-dispatched every tick. - Check custom specialist conflicts before any state save: in main() for new sessions and right after state load on resume. - Reset custom_specialist_dispatched when resume supplies a new definition. - Pin custom system/user prompts with exact-equality tests.
43f6446 to
875f873
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
|
|
||
| def test_fresh_launch_conflict_exits_before_the_session_starts(tmp_path, monkeypatch): | ||
| import hyperloom.inference_optimizer.cli as cli |
|
|
||
| @pytest.mark.parametrize("pruned", [True, False]) | ||
| async def test_guarantee_falls_back_to_research_when_patches_are_impossible(guarantee_coord, pruned, monkeypatch): | ||
| import hyperloom.orchestrator.specialists.runner as runner_mod |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
|
|
||
| def test_fresh_launch_conflict_exits_before_the_session_starts(tmp_path, monkeypatch): | ||
| import hyperloom.inference_optimizer.cli as cli |
|
|
||
| @pytest.mark.parametrize("pruned", [True, False]) | ||
| async def test_guarantee_falls_back_to_research_when_patches_are_impossible(guarantee_coord, pruned, monkeypatch): | ||
| import hyperloom.orchestrator.specialists.runner as runner_mod |
PR #1804 -- feat(specialist): operator-defined custom specialist typeWhat it does: Operators could not point a specialist at a focus outside the fixed specialist catalogue. This PR adds Blocking issues: 1
Checked: all 13 changed files; cli/init.py |
List the operator-defined specialist flags in the SKILL caller table, the IR-4 specialist contract, the workload configuration reference and the FRAMEWORK_AGENT concept section.
Recognise the custom specialist only by its own name and keep the operator-defined layer out of the knowledge-domain tags. A generic 'custom' tag or domain_hint no longer makes stalled-domain forcing pick a denied custom dispatch every tick, and a built-in dispatch tagged 'custom' is admitted as before.
PR #1804 -- feat(specialist): operator-defined custom specialist typeWhat it does: Operators had no way to point a specialist at a focus outside the fixed specialist catalogue. This PR adds Blocking issues: none Checked: all 16 changed files; unconfigured parity base dc46f8d vs head (15 Orchestration prompts, 260 built-in specialist system/user prompt pairs across 13 tag variants including |
…nd allow a bare reset The custom user prompt left out the Experience KB section while the FRAMEWORK_AGENT warmup still recorded the block as injected; render it. A stored definition made --resume-from ... --reset-state exit 2 with a remedy that could not run; only a definition passed on this launch now conflicts with --reset-state, and a bare reset clears the stored one.
…d router Enqueue the guarantee on a real Coordinator and check that it creates one research task and holds back while it is queued; also skip on a queued or running domain-form custom task.
…g branches Fold the resume conflict into the existing resume-conflict check, print the re-export from the helper that applies it, and set the spawn flag without a branch, so _run_optimize and _spawn_fitting_queued keep their merge-base complexity.
PR #1804 -- feat(specialist): operator-defined custom specialist typeWhat it does: Operators had no way to give a specialist a focus of their own, because the specialist catalogue is fixed. This PR adds Blocking issues: none Checked: all 16 changed files; head delta a6f876f..e798aba (cli/init.py LGTM |
PR #1804 -- feat(specialist): operator-defined custom specialist typeWhat it does: an operator could not add a specialist domain of their own -- the catalogue in Blocking issues: none Checked: the full consumer set of the new catalogue entry ( LGTM |
xiaofei-zheng
left a comment
xiaofei-zheng
left a comment
There was a problem hiding this comment.
Approving on the review posted above: no blocking issues.
3 participants
Summary
Adds one operator-defined specialist type,
custom_specialist(KB anchorcustom_specialist). The operator supplies its focus and a one-line description at launch:python -m hyperloom optimize ... \ --custom-specialist-prompt-file ./my_focus.md \ --custom-specialist-description "Tunes the MoE all-to-all path for EP>1 deployments"custom_specialistjoins the specialist EMIT domain list, followed byOPERATOR-DEFINED DOMAIN: custom_specialist — <description>. Both the initial build and the phase-change rebuild carry it.custom_specialisttag is denied withspecialist_custom_not_configuredwhen no definition exists.SharedState. On--resume-from, passing the flags replaces it. A new definition also re-arms the at-least-once guarantee, while the same or no definition keeps it.gap.custom.sessionwhen there are none. It runs in patch mode by default and falls back to research mode, with an observation, when source patches cannot be authored. Admission denials create no task row, so attempts are tracked per macro cycle in memory: at most one attempt per cycle.KNOWLEDGE_DOMAIN_TAGS, so its round counter is set to 0 on dispatch and never incremented, and it is never force-dispatched. Its runs come from Orchestration and the guarantee only.Validation and conflicts
Argument checks run in
main():The launch fails fast when the type is configured, either by flags or by the resumed state, together with any of:
--orch-prompt(the replaced Orchestration prompt would never offer it)--no-framework-agent, or a resumed state with FRAMEWORK_AGENT disabled--research-lane-capacity 0--reset-statetogether with the flags on this launch (the reset would wipe the new definition)A bare
--reset-stateon resume is not a conflict: the stored definition is cleared with the rest of the state, which is how an operator removes it.The check runs before any state is written: in
main()for a new session, and right after the state is loaded on resume.The prompt text is written in plain text to
state.jsonand to each task'sprompt.md, so--helpwarns not to put credentials in it.Unconfigured behaviour is unchanged
All new parameters default to empty, and every new branch is gated on a non-empty definition. The type is recognised only by its own name,
custom_specialist, which no prompt shows unless it is configured; the generic wordcustom(for example a tag an LLM writes in a--framework customsession) resolves to nothing, as before.KNOWLEDGE_DOMAIN_TAGSis identical tomain, so round counters, stalled-domain forcing and the gate's tag vocabulary are unchanged.customtag is still admitted.mainand on this branch, then hashed: every built-in specialist domain × {patch, research} × extra tags (includingcustomandcustom_specialist), both system and user prompts, plus the Orchestration prompt for 5 phases. All 225 hashes are identical.Tests
test_custom_specialist.py(57 tests):_run_optimize; a stored definition with--reset-state(also with--orch-prompt) is not a conflict, while the flags with--reset-stateare, with or without a stored definition.KNOWLEDGE_DOMAIN_TAGStuple, acustom-tagged built-in dispatch admitted, and stalled-domain forcing picking the serving gap over acustom-hinted one.acd741263; the Experience KB test and the two--reset-staterows fail onbcb506aca.ruff check --select C901, head against the merge base, per the style guide).test_cli_bootstrap*,test_cli_resume_launch_shape,test_experience_kb_integration,test_per_domain_prompts,test_policy_gate,test_pump_decouple,test_shared_state_*,test_specialist_dispatch_params,test_specialist_lifecycle,test_specialist_prompt_builder_coverage_unit.End-to-end runs
Spur MI355X, Qwen3-8B, sglang, GLM backend,
--no-kernel, framework phase only. The Experience service was not configured in these workspaces.At
bcb506aca, 1 h budget each; the run without the flags was stopped at about 53 min, in FRAMEWORK_AGENT:custom_specialist, nospecialist_custom_not_configureddenial, round counters are keyed by the eight built-in anchors only, and the custom state fields stay at their defaults.domain=custom_specialist(no tags) in the first FRAMEWORK_AGENT batch. The task started in the same tick, so the guarantee returned oncustom_specialist_dispatchedand never dispatched. The system prompt carried the operator focus verbatim underLayer: operator-defined. KB anchor: custom_specialist.. The task succeeded and was the only custom task of the session.On the first head, 2 h: Orchestration also selected
custom_specialiston its own, the task succeeded with a sourced "no actionable overhead" finding, and the stalled-domain force re-dispatched it on its own research hint;bcb506acaremoved that path.The follow-up fix (Experience KB section,
--reset-state) is covered by unit tests, not by an end-to-end run.Docs
SKILL.md: a row in the caller-responsibility table and an IR-4 paragraph.docs/reference/environment-variables.md: the flags under workload configuration.docs/conceptual/optimization-loop.md: one sentence in the FRAMEWORK_AGENT section.Not covered
cli/__init__.py:1874, before the first resumestate.saveat line 1905.