fix(cli): let a resumed session's recorded workload outrank the shell

docs/how-to/optimize-custom-workload.md

@@ -140,13 +140,23 @@ The CLI exports each pin under its own name, persists them in `state.json`, and
 also serializes them into `INFERENCE_OPTIMIZER_EXTRA_ENV` as a JSON object so a
 resume can tell which names the previous launch set.
 
-A pin and an `export` of the same name are equivalent: no name is withheld. For
-a knob Hyperloom resolves itself — `ISL`, `OSL`, `CONC`, `TP`, `EP`,
-`PRECISION`, `MAX_MODEL_LEN`, `FRAMEWORK` — the environment is one rung of that
-knob's ladder: **an explicit flag wins, then the environment (where your pin
-lands), then the resumed session's recorded value, then the default**. So
-`--extra-env ISL=4096` sets the ISL, and `--isl 2048 --extra-env ISL=4096` runs
-at 2048. Everything else a pin names is set directly and is not interpreted.
+On a fresh launch a pin and an `export` of the same name are equivalent: no name
+is withheld. For a knob Hyperloom resolves itself — `ISL`, `OSL`, `CONC`, `TP`,
+`EP`, `PRECISION`, `MAX_MODEL_LEN`, `FRAMEWORK` — the environment is one rung of
+that knob's ladder: **an explicit flag wins, then the environment (where your
+pin lands), then the default**. So `--extra-env ISL=4096` sets the ISL, and
+`--isl 2048 --extra-env ISL=4096` runs at 2048. On a resume the two stop being
+equivalent for `ISL`, `OSL`, `CONC`, `TP`, `EP` and — when the session recorded
+one — `MAX_MODEL_LEN`: the pin is still honoured, but a bare `export` is
+overwritten by the session's recorded value, so the ladder reads **flag, then the
+pins in effect for this resume (its own `--extra-env`, or the launch's persisted
+pins when it passes none), then the recorded value, then the environment, then
+the default**. `PRECISION` keeps the fresh-launch ladder on a resume too, with
+the recorded value and the checkpoint's dtype below the environment. `FRAMEWORK`
+belongs to the session: a resume re-exports the recorded framework and reads
+`$FRAMEWORK` only when the session recorded none, so neither a pin nor a flag
+changes it.
+Everything else a pin names is set directly and is not interpreted.
 **Forward every pin as its own `--extra-env` flag** — a dropped pin is lost
 silently. On `--resume` the persisted pins are re-exported, so they only need
 re-passing when you want to change them.

docs/reference/environment-variables.md

@@ -77,9 +77,18 @@ The following variables configure filesystem paths for Hyperloom's runtime depen
 
 ## Workload configuration
 
-Set with CLI flags, not env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` /
-`TP` / `EP` env vars are ignored and overwritten (`GPU_TYPE` is a fallback when
-`--gpu-type` is omitted).
+Set with CLI flags. `ISL` / `OSL` / `CONC` / `TP` / `EP` resolve on a ladder,
+and the ladder differs by branch. A fresh launch takes **an explicit flag, then
+the environment (where an `--extra-env NAME=VALUE` pin lands), then the
+default**. A resume takes **an explicit flag, then the pins in effect for this
+resume — its own `--extra-env`, or the launch's persisted pins when it passes
+none — then the session's recorded value, then the shell environment, then the
+default** — so a value left over in the shell cannot displace the shape the
+session was measured at. `PRECISION` is not on that ladder: on both branches it
+is **`--precision`, then `$PRECISION`, then the session's recorded value, then
+the checkpoint's own dtype, then the default**, so on a resume a `$PRECISION`
+left in the shell still outranks the recorded value. `GPU_TYPE` is a fallback
+when `--gpu-type` is omitted.
 
 - **Model / workload shape:** `--model`, `--model-class`, `--framework`,
   `--framework-version`, `--precision`, `--tp`, `--ep`, `--isl`, `--osl`,

docs/reference/multi-node.md

@@ -151,8 +151,8 @@ The following flags are used in multi-node workload skill blocks.
 | `--gpus-per-node` | GPUs per pod (default 8); feeds the TP feasibility check |
 | `--tp` | Tensor parallelism degree |
 | `--ep` | Expert parallelism degree (MoE; often `ep == tp`) |
-| `--isl`, `--osl`, `--conc` | Benchmark input length, output length, and concurrency. These must be flags — the corresponding env vars are ignored and overwritten |
-| `--gpu-type`, `--precision` | Target GPU (for example `mi325x`) and dtype (`bf16`). Flags are authoritative; env vars are not |
+| `--isl`, `--osl`, `--conc` | Benchmark input length, output length, and concurrency. The flag wins; on a fresh launch the corresponding env var is next, then the default. On a resume it is the pins in effect for this resume (its own `--extra-env`, or the launch's persisted pins when it passes none), then the session's recorded value, then the env var, then the default |
+| `--gpu-type`, `--precision` | Target GPU (for example `mi325x`) and dtype (`bf16`). `--precision` has its own ladder on both branches: the flag, then `$PRECISION`, then the session's recorded value, then the checkpoint's dtype, then the default. `--gpu-type` is a hint: on a fresh launch the hardware probe wins and a disagreeing flag only warns; a resume takes the GPU type from the session and ignores the flag |
 | `--framework` | `sglang` |
 | `--target-gain`, `--max-hours` | Optimization goal and time budget |
 | `--pd-mode disaggregated` | Enable PD disaggregation (infera and rayjob) |

examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md

@@ -278,7 +278,7 @@ and the stop reason. Never print API keys, tokens, or custom header values.
 3. Keep `PYTHONPATH="$REPO_ROOT:${PYTHONPATH:-}"` in the launch shell so
    critic subprocesses can import `hyperloom.agents` after changing cwd.
 4. Run it detached the way the harness understands: if `$CLAW_SESSION_ID` is set and your bash tool takes a `run_in_background` parameter, hand the optimizer command to it with `run_in_background=true`, without shell-level detachment (`setsid`, `nohup`, or a trailing `&`); otherwise use `setsid nohup ... &`. See the Launch section of the packaged `hyperloom/inference_optimizer/SKILL.md` for why — a hand-detached run is invisible to Claw and its sandbox is reclaimed about fifteen minutes after the turn ends.
-5. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload.
+5. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`: the flag is the top of the ladder, while a value reaching the environment from `.env` is indistinguishable from one a previous run left there, and on `--resume-from` the session's recorded workload outranks it.
 6. Include `--phase-budget-framework-pct 0.43` and `--phase-budget-kernel-pct 0.42`
    in the optimize command. Do **not** pass `--no-framework-agent` or `--no-kernel` —
    this demo runs the full OPTIMIZE phase (FRAMEWORK_AGENT + KERNEL_AGENT), and

examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md

@@ -230,7 +230,7 @@ and the stop reason. Never print API keys, tokens, or custom header values.
 2. Keep `PYTHONPATH="$REPO_ROOT:${PYTHONPATH:-}"` in the launch shell so
    critic subprocesses can import `hyperloom.agents` after changing cwd.
 3. Run it detached the way the harness understands: if `$CLAW_SESSION_ID` is set and your bash tool takes a `run_in_background` parameter, hand the optimizer command to it with `run_in_background=true`, without shell-level detachment (`setsid`, `nohup`, or a trailing `&`); otherwise use `setsid nohup ... &`. See the Launch section of the packaged `hyperloom/inference_optimizer/SKILL.md` for why — a hand-detached run is invisible to Claw and its sandbox is reclaimed about fifteen minutes after the turn ends. In Docker mode that launch still runs inside one attached `docker exec … bash -lc`; never `docker exec -d`.
-4. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload.
+4. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`: the flag is the top of the ladder, while a value reaching the environment from `.env` is indistinguishable from one a previous run left there, and on `--resume-from` the session's recorded workload outranks it.
 5. Include `--phase-budget-framework-pct 0.43` and `--phase-budget-kernel-pct 0.42`
    in the optimize command. Do **not** pass `--no-framework-agent` or `--no-kernel` —
    this demo runs the full OPTIMIZE phase (FRAMEWORK_AGENT + KERNEL_AGENT).

examples/hyperloom-qwen3-8b-3h/SKILL.md

@@ -224,7 +224,7 @@ and the stop reason. Never print API keys, tokens, or custom header values.
 2. Keep `PYTHONPATH="$REPO_ROOT:${PYTHONPATH:-}"` in the launch shell so
    critic subprocesses can import `hyperloom.agents` after changing cwd.
 3. Run it detached the way the harness understands: if `$CLAW_SESSION_ID` is set and your bash tool takes a `run_in_background` parameter, hand the optimizer command to it with `run_in_background=true`, without shell-level detachment (`setsid`, `nohup`, or a trailing `&`); otherwise use `setsid nohup ... &`. See the Launch section of the packaged `hyperloom/inference_optimizer/SKILL.md` for why — a hand-detached run is invisible to Claw and its sandbox is reclaimed about fifteen minutes after the turn ends. In Docker mode that launch still runs inside one attached `docker exec … bash -lc`; never `docker exec -d`.
-4. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload.
+4. Pass all required optimize CLI flags in the `python -m hyperloom optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`: the flag is the top of the ladder, while a value reaching the environment from `.env` is indistinguishable from one a previous run left there, and on `--resume-from` the session's recorded workload outranks it.
 5. Include `--phase-budget-framework-pct 0.50` and `--phase-budget-sweep-pct 0.01`
    in the optimize command. These are the value *before* redistribution: with
    `--no-kernel`, KERNEL_AGENT is disabled and its freed share is added on top,

src/hyperloom/inference_optimizer/cli/__init__.py

@@ -1235,14 +1235,29 @@ def _positive_env_int(name: str) -> int:
     return value if value > 0 else 0
 
 
+def _seed_env_from_resumed_state(state: Any, pins: dict[str, str]) -> None:
+    """Put the session's recorded workload in the environment, so a resume outranks whatever the shell exported.
+
+    Hyperloom's own repro scripts ``export TP``/``CONC``/``ISL``/``OSL``, and sourcing one before a resume must not
+    silently re-measure the session at another shape. ``pins`` -- this resume's ``--extra-env``, or, when it
+    passes none, the pins persisted from the launch -- is skipped, so changing the workload stays an explicit act.
+    """
+    for name in ("ISL", "OSL", "CONC", "TP", "EP", "MAX_MODEL_LEN"):
+        recorded = getattr(state, name.lower(), 0)
+        if recorded and name not in pins:
+            os.environ[name] = str(recorded)
+
+
 def _resolve_workload_knobs(
     args: argparse.Namespace,
     state: Any | None = None,
 ) -> None:
     """Fill the unset numeric workload knobs on ``args``: flag, environment, resumed state, default.
 
-    The environment rung carries an ``--extra-env`` pin, which :func:`_export_operator_launch_shape` has already
-    written. Precision has its own function because it reads the checkpoint, so it cannot run this early.
+    The environment rung carries the ``--extra-env`` pins :func:`_export_operator_launch_shape` has already written,
+    and on a resume also the recorded workload :func:`_seed_env_from_resumed_state` writes over any shell export -- a
+    resumed value therefore arrives through that rung rather than through ``state``. Precision has its own function
+    because it reads the checkpoint, so it cannot run this early.
     """
     int_knobs = (
         ("isl", DEFAULT_ISL),
@@ -1264,7 +1279,8 @@ def _resolve_precision(args: argparse.Namespace, state: Any | None = None) -> No
     """Fill ``args.precision``: flag, environment, resumed state, checkpoint detection, default.
 
     Must run after the model is resolved: the detection rung reads the checkpoint's ``config.json``, and finding
-    nothing there settles ``args.precision`` on the default with no later call able to correct it.
+    nothing there settles ``args.precision`` on the default with no later call able to correct it. A ``PRECISION``
+    reaching this function through the environment is treated as an explicit flag, mismatch warning included.
     """
     precision = getattr(args, "precision", None)
     if not precision:
@@ -1999,9 +2015,10 @@ async def _run_optimize(args: argparse.Namespace) -> int:
             model=str(state.model_path or ""),
             launch_info_file=getattr(args, "launch_info_file", None),
         )
-        # Resolve workload knobs with the resumed state as the fallback source (explicit --isl/--conc/... on this
-        # resume still win), then project the resolved values into env so resume sees the same workload contract (not
-        # YAML defaults).
+        _seed_env_from_resumed_state(state, _resume_extra_env)
+        # The seeding above puts the recorded workload on the environment rung, so it outranks a shell export while an
+        # explicit --isl/--conc/... on this resume still wins. The resolved values are then projected back into env so
+        # resume sees the same workload contract (not YAML defaults).
         _resolve_workload_knobs(args, state)
         _resolve_precision(args, state)
         # Same ladder as the knobs above; the environment rung is what lets a re-passed pin change it.

src/hyperloom/inference_optimizer/tests/test_cli_resume_launch_shape.py

@@ -122,12 +122,16 @@ def test_the_fresh_launch_anchors_stay_in_order():
     from hyperloom.inference_optimizer.cli import _run_optimize
 
     tree = ast.parse(textwrap.dedent(inspect.getsource(_run_optimize)))
-    first: dict[str, int] = {}
+    calls: dict[str, list[int]] = {}
     for node in ast.walk(tree):
         if isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
-            name = node.func.id
-            if name not in first or node.lineno < first[name]:
-                first[name] = node.lineno
+            calls.setdefault(node.func.id, []).append(node.lineno)
+    first = {name: min(lines) for name, lines in calls.items()}
+    # The ladder runs on both branches; the resume anchor needs that branch's call, not the fresh one. -1 when the
+    # seeding has drifted past every ladder call, so the comparison below fails rather than skipping.
+    if "_seed_env_from_resumed_state" in first:
+        after = [ln for ln in calls.get("_resolve_workload_knobs", []) if ln > first["_seed_env_from_resumed_state"]]
+        first["_resolve_workload_knobs_resume"] = min(after) if after else -1
 
     # (earlier, later, why the order matters)
     anchors = [
@@ -141,6 +145,11 @@ def test_the_fresh_launch_anchors_stay_in_order():
             "_export_workload_envs_for_optimize",
             "projecting first publishes the flag-derived default over a pinned TP/CONC/EP",
         ),
+        (
+            "_seed_env_from_resumed_state",
+            "_resolve_workload_knobs_resume",
+            "seeding after the resume ladder leaves a stale shell export as the session's workload",
+        ),
         (
             "_export_workload_envs_for_optimize",
             "_preflight",
@@ -184,6 +193,57 @@ def test_a_malformed_pinned_knob_does_not_take_the_run_down(monkeypatch):
     assert args.isl == DEFAULT_ISL
 
 
+def test_a_stale_shell_export_does_not_re_measure_a_resumed_session(monkeypatch):
+    """The recorded workload outranks the shell; a name this resume re-pins still changes it.
+
+    Hyperloom's own repro scripts ``export TP``/``CONC``/``ISL``/``OSL``, so sourcing one before a resume would
+    otherwise re-measure the session at a different shape and record the shell's numbers as its own.
+    """
+    from hyperloom.inference_optimizer.cli import _resolve_workload_knobs, _seed_env_from_resumed_state
+
+    state = _ns(isl=3333, osl=777, conc=99, tp=8, ep=2, precision="int4", max_model_len=32768)
+    # What sourcing a generated repro script leaves behind, plus the one knob this resume re-pins.
+    for name, value in (("TP", "4"), ("CONC", "256"), ("ISL", "4096"), ("OSL", ""), ("EP", ""), ("PRECISION", "")):
+        monkeypatch.setenv(name, value)
+    monkeypatch.setenv("MAX_MODEL_LEN", "")
+    monkeypatch.setenv("INFERENCE_OPTIMIZER_EXTRA_ENV", "")
+    repinned = {"CONC": "16"}
+    _export_operator_launch_shape(server_args="", extra_env=repinned)
+
+    _seed_env_from_resumed_state(state, repinned)
+    args = _knob_args()
+    _resolve_workload_knobs(args, state)
+
+    assert (args.tp, args.isl) == (8, 3333), "a stale shell export replaced the recorded workload"
+    assert args.conc == 16, "a re-passed pin must still change the workload"
+    assert os.environ["MAX_MODEL_LEN"] == "32768"
+
+
+def test_a_plain_resume_does_not_warn_about_the_recorded_precision(tmp_path, monkeypatch, capsys):
+    """The recorded precision reaches ``args`` without taking the flag branch, whose warning no operator asked for.
+
+    A seeded ``PRECISION`` is indistinguishable from ``--precision``, so a resume of an fp8 session whose checkpoint
+    reads bf16 would warn on every restart while changing nothing.
+    """
+    from hyperloom.inference_optimizer.cli import _resolve_precision, _seed_env_from_resumed_state
+
+    model = tmp_path / "model"
+    model.mkdir()
+    (model / "config.json").write_text('{"torch_dtype": "bfloat16"}', encoding="utf-8")
+    # Every name the seeding writes, so monkeypatch restores it: an unscoped EP reaches the next test in the shard.
+    for name in ("ISL", "OSL", "CONC", "TP", "EP", "MAX_MODEL_LEN", "PRECISION"):
+        monkeypatch.setenv(name, "")
+    monkeypatch.setenv("INFERENCE_OPTIMIZER_EXTRA_ENV", "")
+    state = _ns(isl=3333, osl=777, conc=99, tp=8, ep=2, precision="fp8", max_model_len=0)
+
+    _seed_env_from_resumed_state(state, {})
+    args = _knob_args(model=str(model))
+    _resolve_precision(args, state)
+
+    assert args.precision == "fp8"
+    assert capsys.readouterr().err == ""
+
+
 def test_resume_restores_pins_before_the_agentx_staleness_guard(tmp_path, monkeypatch, request):
     """A session whose AgentX backend came from a pin resumes; the guard must not read an unpinned environment.