diff --git a/docs/compatibility.rst b/docs/compatibility.rst index 023d2c3483..605df23e0a 100644 --- a/docs/compatibility.rst +++ b/docs/compatibility.rst @@ -93,6 +93,26 @@ The following table lists the validated Hyperloom version and component combinat .. _AgentKernelArena 0.2.0: https://rocm.docs.amd.com/projects/agent-kernel-arena/en/docs-0.2.0/ .. _Magpie 0.2.0: https://rocm.docs.amd.com/projects/magpie/en/docs-0.2.0/ +.. note:: + + The base benchmark path remains compatible with Magpie 0.2.0. The pinned + native AgentX pair is `Magpie v0.3.0 + `_ plus native launch overrides, custom-model replay, and client/eval + compatibility fixes at commit ``d80eb4d3dad7fabe01ce81d049e2983adf2c86dd`` and InferenceX commit + ``408c015be4b22d14c69518643609669405507077``. Pass the Magpie YAML with + ``--benchmark-config``; its ``benchmark.agentx: enable`` source switch + automatically selects Hyperloom's persisted AgentX session and grading mode. + Fresh ``HYPERLOOM_AGENTX=1`` launches also select native AgentX and resolve + a recipe from normal workload inputs. Saved epoch-1 sessions keep the legacy + client; saved epoch-2 native sessions keep their measurement-only contract, + and epoch-3 sessions keep their upstream launcher contract. + New native sessions use epoch 4 and the Magpie-managed launch-overrides contract. ``benchmark.docker_image`` overrides/pins the effective + recipe image and is included in its fingerprint; an existing + ``HYPERLOOM_IMAGE`` is only a strict consistency assertion. Neither value + starts or attests the outer container. + ``install.sh`` upgrades an importable pre-AgentX Magpie + build instead of treating import success as sufficient. + .. note:: TraceLens does not have hard requirements for the GPU, ROCm version, or the OS; it has scripts to verify whether a trace is valid/parseable. TraceLens is: diff --git a/docs/components/magpie.md b/docs/components/magpie.md index d0f5e407d7..863e3d2f3a 100644 --- a/docs/components/magpie.md +++ b/docs/components/magpie.md @@ -39,12 +39,59 @@ cmd = [ ``` Each run produces a `benchmark_report.json` that Hyperloom parses to extract -throughput/measurements and pick winners. Hyperloom pins Magpie to a ref that -already copies benchmark scripts atomically upstream; `_magpie_patcher.py` applies -install-time compatibility patches (SGLang trust, eval concurrency, client -tokenizer hook) to the pinned scripts. See +throughput/measurements and pick winners. Hyperloom pins the Magpie +[v0.3.0 release](https://github.com/AMD-AGI/Magpie/releases/tag/v0.3.0) plus native launch overrides, custom-model replay, and the +generic client/eval compatibility fixes at immutable commit +`d80eb4d3dad7fabe01ce81d049e2983adf2c86dd`. The fixes keep evaluation scripts +bound to their intended InferenceX source after directory changes and run the +packaged benchmark client with Transformers 5 in a separate Python environment. +The serving framework environment is unchanged. The pinned scripts also support +Hyperloom's client tokenizer and trust settings directly, so compatibility checks +leave the audited Magpie package unchanged. These capabilities are pinned to a merged Magpie +commit beyond v0.3.0. Earlier GPU validation used the v0.3.0 release commit and does not +validate these extensions; their source and wheel share an audited execution tree. +The pin includes [Magpie #117](https://github.com/AMD-AGI/Magpie/pull/117): SGLang detailed +profiling accepts runtimes without shape discovery and enables each shape-discovery +option only when supported. Graph-mode captures still require graph profiling support +and retain per-batch graph trace export. +The installer and CLI apply native capability, commit, and execution-tree checks +only to native AgentX sessions. Ordinary benchmarks retain the importability +contract, including compatible custom packages and existing offline installations. +With `benchmark.agentx: enable`, Magpie resolves the serving specification, +owns server startup and cleanup, and runs the maintained InferenceX/AIPerf client. +The integration validates the package, recipe/client sources, and effective +server launch evidence. Both the repository root and the `inferencex-e2e/` +project layout are supported; no Slurm launcher extension is required. + +Fresh `HYPERLOOM_AGENTX=1` sessions use Magpie native AgentX and retain +Hyperloom's optimization loop. Magpie resolves the recipe and launcher from +the model, framework, GPU, precision, and concurrency; ambiguous inputs fail +with a selector error. New sessions record backend `native` and epoch 4. +With no registered recipe match, an explicit image, TP, EP, concurrency, and +canonical model identify a custom SGLang/vLLM workload; Magpie validates its +model context and resolves a formal generic launcher. Registered ambiguities +remain errors. Candidate KEEPs retain the exact native launch snapshot, so +resume and subsequent candidates do not apply the same launch arguments twice. +Persisted epoch-1 sessions resume the legacy client, and epoch-2 native sessions +retain their measurement-only contract. Saved epoch-3 sessions retain their +upstream-launcher contract. No baseline or KEEP record is migrated. + +Native profiling derives a diagnostic run from the accepted candidate and uses +Magpie's phase-gated, step-bounded torch profiler. Magpie owns the first-capture +delay, repetition interval, count limit, and framework-specific controls. +Hyperloom selects a complete capture from the manifest for kernel/roofline +analysis. Successful standalone `profile` tasks also publish their trace to the +Coordinator for analysis, without requiring benchmark-valid throughput. +Diagnostic success never makes the run a publishable KEEP result. +Detailed annotations require a compatible instrumented serving runtime; an +ordinary GPU trace alone does not establish complete kernel shape metadata. See +[Run an InferenceX AgentX workload](../how-to/optimize.md#run-an-inferencex-agentx-workload) +for the example and current optimization limitations, and [Hyperloom optimization loop](../conceptual/optimization-loop.md) for more information. ## Magpie documentation For detailed documentation on Magpie, see [Magpie on ROCm Docs](https://rocm.docs.amd.com/projects/magpie/en/latest/). + +The pinned generic serving path also supplies InferenceX evaluation defaults (`EVAL_ONLY=false`, +`IS_AGENTIC=0`, and a local placeholder API key), preserving explicitly configured values. diff --git a/docs/conceptual/optimization-loop.md b/docs/conceptual/optimization-loop.md index e8c293e94a..f32bd897b0 100644 --- a/docs/conceptual/optimization-loop.md +++ b/docs/conceptual/optimization-loop.md @@ -102,8 +102,10 @@ PRELUDE establishes the session baseline: `competitor_target.json`. With no external GPU configured, it writes a no-target marker and clears the competitor target. Advisory and final-report comparisons read the same target file; missing targets remain unavailable. - AgentX uses accepted `current_best` metrics, normalized by `state.tp` and - matched at `state.conc`, without rereading benchmark artifacts. + AgentX uses accepted `current_best` metrics, normalizing total throughput by + the trusted, fingerprint-bound recipe's recorded TP×PP×PCP GPU count (with + `state.tp` only as a legacy-artifact fallback) and matching at `state.conc`, + without rereading benchmark artifacts. `--no-target-advisory` disables prompt hints, not the final comparison. External references never change Objective, scoring, or KEEP/REVERT. 2. `baseline` measures the starting throughput and records the benchmark @@ -112,6 +114,27 @@ PRELUDE establishes the session baseline: `roofline` is the preferred composite path when enabled; it wraps profiling, trace analysis, and `analysis.md` snapshot publication. +For native AgentX, Hyperloom pre-resolves every recipe point through the same +interpreter that runs Magpie. It fills an omitted outer `--tp` from resolved +TP×PP×PCP and treats an explicit value as an exact assertion. Topology-changing +rounds fail before launch. In fresh epoch-4 sessions, Magpie owns server +startup and records the effective candidate arguments, environment, and source +overlays. Hyperloom binds that evidence before accepting a result. Profile +rounds derive from the accepted configuration and use Magpie's phase-gated +torch capture. Each capture remains separate and diagnostic; it cannot anchor +KEEP comparisons. GEAK proposals require review and canonical AgentX +remeasurement before adoption. Saved epoch-2 sessions retain their earlier +measurement-only restrictions, and epoch-3 sessions retain the upstream +launcher and compatibility-profiler contract. + +Session mode is selected before PRELUDE: `--benchmark-config ` reads the +source config, and `benchmark.agentx: enable` automatically stamps AgentX mode +for grading and persisted state. Fresh `HYPERLOOM_AGENTX=1` sessions also select +native AgentX; no additional opt-in is required. Backend identity survives +resume through the accepted/source +config and measurement epoch; native-only refusals do not apply to legacy +AgentX sessions. + `model_class` is supplied by the launcher or derived once from model metadata at boot. There is no separate live `classify` action. @@ -320,6 +343,11 @@ gates. What the phase actually does depends on the kernel backend, and the branches look very different from Orchestration's side: +- **Native AgentX**: optimization sessions dispatch diagnostic proposals and + require verified candidate launch evidence plus canonical AgentX + remeasurement before KEEP. Source changes also require the Critic review + and transactional integration. Only persisted epoch-2 measurement-only + sessions retain `unsupported_upstream_launcher_hook` and skip this phase. - **Default (`geak`)**: entering the phase enqueues one Coordinator-owned `kernel_agent` task, which holds `server_lifecycle`, `workspace_mutation` and `benchmark_lane` for the whole pipeline. Under GEAK it runs a single @@ -360,18 +388,21 @@ be attempted or explicitly rejected before report can close the run. ## SWEEP -SWEEP measures the optimized stack against the baseline across a -concurrency ladder, one arm each, and produces the throughput-vs- -interactivity curve. The ladder is sized for the workload: powers of two -down from 256 for a synthetic run, `1,4,8,10,14,20,28` for an agentic one, -where a request carries orders of magnitude more prompt and the same card -saturates far lower. Override either with `--conc-sweep-concs`. Under -AgentX the sweep is off unless `--enable-conc-sweep` is passed, since each -rung is a 3600 s window and the session grades at a fixed CONC; SWEEP then -records a disabled skip. +SWEEP measures the optimized stack against the baseline across a concurrency +ladder and produces a throughput-vs-interactivity curve. Synthetic workloads +default to powers of two down from 256; legacy AgentX and the MLPerf backend +use `1,4,8,10,14,20,28`. Override the ladder with `--conc-sweep-concs`. +All AgentX sessions default the sweep off. Legacy AgentX and MLPerf may opt in +with `--enable-conc-sweep`. + +Native AgentX measures the fixed recipe concurrency supplied by `--conc`. +Changing it would change the accepted workload identity, so explicit +`--enable-conc-sweep` is rejected during preflight; `--conc-sweep-concs` does +not enable it. Results update `last_conc_sweep` and feed the final report and breakdown. -The phase exits on `sweep_done` (or `sweep_failed`). +When enabled, the phase exits on `sweep_done` (or `sweep_failed`); a disabled +sweep records its terminal skip and proceeds to CLOSE. ## CLOSE diff --git a/docs/how-to/optimize.md b/docs/how-to/optimize.md index 0f33bee923..511a201279 100644 --- a/docs/how-to/optimize.md +++ b/docs/how-to/optimize.md @@ -55,22 +55,216 @@ Requirements: | Field | Meaning | How to choose | |-------|---------|---------------| | `TP` | Tensor-parallel size — number of GPUs the model is sharded across | Must match the number of GPUs in your server node (for example, `8` for a single 8-GPU MI300X node) | -| `CONC` | Concurrent requests — baseline benchmark concurrency (`--conc`, default `64`) | Set to your target concurrency. The SWEEP phase separately measures a ladder around it: `256,128,64,32,16,8,4,2` for a synthetic workload, `1,4,8,10,14,20,28` under `HYPERLOOM_AGENTX`, where it runs only with `--enable-conc-sweep`. Override the ladder with `--conc-sweep-concs`. | +| `CONC` | Concurrent requests — baseline benchmark concurrency (`--conc`, default `64`) | Set to your target concurrency. Synthetic SWEEP defaults to `256,128,64,32,16,8,4,2`; legacy AgentX and MLPerf can opt into `1,4,8,10,14,20,28` with `--enable-conc-sweep`. Override either ladder with `--conc-sweep-concs`. Native AgentX measures this fixed recipe point and rejects explicit sweep enablement. | | `ISL` | Input sequence length — tokens in each request's prompt | Match your production workload; `1024` is a common starting point | | `OSL` | Output sequence length — tokens generated per response | Match your production workload; `1024` is a common starting point | ```{note} -`ISL` / `OSL` describe the synthetic request shape. Under the opt-in agentic -trace-replay mode (`HYPERLOOM_AGENTX=1`) request lengths come from the recorded -trace corpus instead, so these two values do not affect what is measured — the -server's context window is sized from the model's own configuration rather than -from `ISL+OSL`. +`ISL` / `OSL` describe the synthetic request shape. In a source config with +`benchmark.agentx: enable`, request lengths come from the recorded trace corpus +instead, so these two values do not affect what is measured — the server's +context window is sized from the model's own configuration rather than from +`ISL+OSL`. ``` See [`src/hyperloom/inference_optimizer/SKILL.md`](https://github.com/AMD-AGI/Hyperloom/blob/main/src/hyperloom/inference_optimizer/SKILL.md) for the full prompt field reference (every field maps to a CLI flag defined in `cli/parser.py`). +## Run an InferenceX AgentX workload + +AgentX measurement uses Magpie's native InferenceX integration. Pass a source +Magpie YAML with `--benchmark-config`; its `benchmark.agentx: enable` switch +automatically selects Hyperloom's AgentX session and grading mode. Alternatively, +set `HYPERLOOM_AGENTX=1` and pass the usual model, framework, GPU, precision, and +concurrency arguments; Magpie resolves the native recipe and launcher. This +integration is pinned to Magpie +[v0.3.0](https://github.com/AMD-AGI/Magpie/releases/tag/v0.3.0) plus native launch overrides, custom-model replay, and generic +client/eval compatibility fixes at commit `d80eb4d3dad7fabe01ce81d049e2983adf2c86dd` and InferenceX commit +`408c015be4b22d14c69518643609669405507077`. + +New AgentX sessions record native backend epoch 4 and retain Hyperloom's +optimization loop. Recipe or matrix ambiguity is an error: use the existing +`AGENTX_RECIPE`, `AGENTX_CONFIG_FILE`, or `AGENTX_SELECTOR` inputs to select the +intended workload. A local checkpoint requires a canonical `AGENTX_MODEL_ID` +or source `benchmark.model`; Hyperloom does not infer identity from a basename. +When no registered recipe matches, Magpie can resolve a custom SGLang or vLLM +workload on MI300X, MI325X, or MI355X from an explicit image, TP, EP, and +concurrency. Supply `HYPERLOOM_IMAGE`, `--tp`, `--ep`, and `--conc`, or the +corresponding source `docker_image` and `envs.TP`/`EP_SIZE`/`CONC` fields. +Magpie verifies model context from the checkpoint's `config.json` or Hugging +Face metadata; `--max-model-len` may cap it. An ambiguous registered recipe +still requires a selector and never falls through to a custom workload. +Model architecture, modality, and quantization support still depend on the +chosen framework image; the generic launcher does not make every model runnable. +Persisted epoch-1 sessions resume their legacy client, and epoch-2 native +sessions retain the measurement-only contract; epoch-3 sessions keep their +upstream launch contract. Resume never upgrades an epoch +or reuses a baseline or KEEP record from another backend. + +For the pinned GLM-5.2 TP4 recipe, create a source YAML. It carries the public +model identity, framework, launcher, image pin, and fixed concurrency; recipe +internals remain in InferenceX: + +```yaml +benchmark: + framework: sglang + model: amd/GLM-5.2-MXFP4 + precision: fp4 + runner_type: mi355x + run_mode: local + agentx: enable + docker_image: lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260924 + gpu_selection: + auto: false + envs: + CONC: 8 + ROCR_VISIBLE_DEVICES: 0,1,2,3 +``` + +Then run Hyperloom inside the exact image named by the YAML: + +```bash +python -m hyperloom.inference_optimizer.cli optimize \ + --benchmark-config ./agentx-glm52.yaml \ + --max-hours 3 +``` + +The explicit three-hour budget is for one canonical baseline. The normal +two-hour default is commonly shorter than model load, warmup, drain, and the +3600-second measurement together. Size it upward if more rounds are intended. + +`--benchmark-config` is for a fresh launch. A resume rejects it. Once a baseline +has been accepted, resume restores its materialized config and runtime pins; +before that point it restores the session's snapshotted source YAML and pins. +The installer defaults must still resolve to the pinned Magpie and InferenceX +revisions above; changing either pin is a coordinated compatibility change. + +The source and resolved fields have different owners: + +| Owner | Fields | +|---|---| +| Source YAML / operator | canonical `model`, `framework`, `precision`, `runner_type`, `run_mode: local`, `benchmark_script`, effective `docker_image`, fixed `envs.CONC`, and optional `agentx.recipe` / `agentx.selector` controls | +| InferenceX recipe | logical TP/PP/PCP/EP, `MODEL_PREFIX`, KV-offload settings, CPU DRAM allocation, corpus, duration, and AIPerf protocol | +| Hyperloom | physical GPU reservation `TP×PP×PCP`, the zero-based ROCR mask, `gpu_selection.auto=false`, session pins, and result validation | + +`benchmark.model` is the exact model id in InferenceX's `amd-master.yaml`; +the optional CLI `--model` is the local mounted checkpoint. Magpie resolves the +recipe's TP/PP/PCP/EP, model prefix, KV-offload settings, CPU DRAM allocation, +and AIPerf protocol. Before acquiring GPUs, Hyperloom resolves the recipe with +the same benchmark interpreter that will execute Magpie. It runs the result in +the already selected serving container (`run_mode: local`) and does not start a +nested Docker container. `benchmark.docker_image` overrides and pins the +effective recipe image, and Magpie includes that value in the recipe +fingerprint. An existing `HYPERLOOM_IMAGE` is an optional consistency assertion +and must match it exactly. Neither value starts a container or proves the image +of the process already running. + +The example omits `benchmark.inferencex_path`. Preflight reuses a writable +checkout at the tested commit or clones one into the dependency cache. An +optional source path only nominates a preferred checkout: a missing or +wrong-revision path falls back to the pinned clone, while an explicit checkout +at the right revision that is not writable fails preflight. Managed AgentX +accepts either the repository root or its `inferencex-e2e/` project directory, +including paths containing spaces. Saved shell-launcher sessions retain their +original path restrictions. + +If the checkpoint should come directly from Hugging Face, omit CLI `--model`; +Hyperloom uses `benchmark.model` from the source YAML and removes +`MODEL_PATH` from the native InferenceX subprocess so it uses its normal +Hugging Face cache. For a local checkpoint, pass an absolute path or an +explicit relative path such as `./models/GLM-5.2`. + +For native AgentX, omitted `--tp` and `--ep` are filled from the selected +recipe. An explicit `--tp` is an exact outer resource assertion and must equal +the resolved `TP×PP×PCP` physical GPU count; an explicit `--ep` must equal the +resolved EP. The example therefore omits both and resolves to TP4/EP4 on +`4×1×1` physical GPUs. Topology-changing AgentX sweeps are rejected. The pinned +launchers overwrite logical `HIP_VISIBLE_DEVICES` with physical ROCR values, +so Hyperloom derives `gpu_selection.auto=false` and the zero-based mask +`ROCR_VISIBLE_DEVICES=0,...,N-1` when omitted. Explicit source values are +assertions; a nonzero or reordered mask is rejected. + +The effective concurrency comes from CLI `--conc` or `benchmark.envs.CONC` and +must exist in the selected recipe. Native AgentX concurrency sweeps default +off; explicitly passing `--enable-conc-sweep` fails preflight because the +pinned launcher has no distinct optimized arm to compare. `CONC=8` uniquely +selects the example's TP4/EP4 DRAM+HiCache arm; `envs.TP` is not a selector. If +a concurrency belongs to multiple arms, use the YAML-native object form: + +```yaml +agentx: + enabled: true + selector: + tp: 4 + kv_offloading: dram + kv_offload_backend: hicache +``` + +Add `agentx.recipe` only when multiple recipe names match. For an enabled +native configuration, `AGENTX_RECIPE` and JSON `AGENTX_SELECTOR` environment +variables provide the same selectors when the YAML does not specify them; +they do not select the native backend by themselves. Magpie fails closed +rather than guessing. Hyperloom writes the fixed measurement concurrency to +`benchmark.envs.CONC` and removes a stale `benchmark.agentx.concurrency` from +the input YAML. The resolved recipe owns +the corpus; native `AGENTX_DATASET` and `WEKA_LOADER_OVERRIDE` overrides are +rejected. Canonical mode runs for 3600 seconds and configures a 393-trace +dataset-entry cap. That value is a loader ceiling, not a guarantee that 393 +traces, sessions, or requests survive availability and context-length filters. +For a 1200-second diagnostic directly in Magpie, set +`benchmark.agentx.mode: fast` in an enabled AgentX configuration, or pass +`--agentx --agentx-mode fast` to its `benchmark` command. Fast results are +non-publishable, so a full Hyperloom `optimize` rejects them as its baseline, +including when selected through Hyperloom's `AGENTX_MODE=fast` override. + +This Hyperloom integration supports Magpie AgentX v1 in local, single-node +SGLang or vLLM mode. It bypasses Hyperloom's outer Ray actor automatically; +leave `INFERENCE_OPTIMIZER_RAY_EXEC` unset or set it to `0`, because explicitly +setting it to `1` is rejected. Multi-node/disaggregated execution, +`server_lifecycle`, and Atom are also unsupported. + +New native sessions use the Magpie-managed launch-overrides contract. +Server arguments, environment changes, and source overlays must be represented +in launch evidence before a canonical result can be accepted. Hyperloom keeps +the workload fingerprint fixed while each candidate has its own execution +identity. Unsupported launch controls fail closed. Saved epoch-2 sessions keep +their earlier measurement-only restrictions. + +If PRELUDE schedules profiling, epoch-4 sessions derive a diagnostic run from +the accepted native configuration and candidate launch. Magpie waits for +AIPerf's measured phase, then captures the configured server steps. It owns +`torch_profiler.start_seconds` (default 0), `num_profiles` (default 1), and +`interval_seconds` (default 200, measured after the previous flush). Oversized +counts are reduced to the available measurement window; complete captures can +succeed even when fewer than requested. Capture, flush, or cancellation errors +remain failures. Hyperloom selects a complete capture by its manifest rather +than mixing trace files across rounds. `detailed_annotations: true` requests +the framework instrumentation required for full shape-aware analysis; Magpie +checks support and configures the framework. Basic traces do not by themselves +prove that kernel shape or annotation requirements are satisfied. + +Profiled runs are diagnostic (`benchmark_valid=false`, `publishable=false`); +their throughput never replaces a baseline or KEEP. Canonical measurements +retain the recipe's replay and context policy; `ISL` and `OSL` do not reshape +AgentX. Saved epoch-3 sessions retain their earlier compatibility profiler. +GEAK uses diagnostic evidence to propose changes. Its proxy scores cannot +promote a candidate: accepted source patches +require a real Critic review and transactional integration followed by native +AgentX measurement. A rejected review, missing artifact, or failed canonical +run leaves the accepted configuration unchanged. Epoch-2 sessions keep their +earlier GEAK skip. + +Accepted native results require `benchmark_valid=true`, `publishable=true`, an +`agentic-coding` scenario, matching strict recipe/launch/raw fingerprints, and +a trusted fingerprint-bound GPU topology. `publishable` attests Magpie's +canonical protocol; Hyperloom separately binds the selected recipe, resolved +server specification, client sources, and pinned checkout. It still cannot cryptographically +prove the actual outer image. Its execution identity covers the resolved +`BenchmarkConfig` plus the effective, scrubbed launcher environment for an +audited set of server/framework/runtime controls; credentials, cache routing, +output paths, and unrelated login-shell variables remain outside that hash. + ## Monitor the run The agent reports a session ID, log path, and PID, then reads persisted state diff --git a/docs/reference/authentication.md b/docs/reference/authentication.md index 40dd012ec9..2452b81b17 100644 --- a/docs/reference/authentication.md +++ b/docs/reference/authentication.md @@ -240,8 +240,14 @@ side is a second provider rather than part of the same gateway credential. ## Path environment -These are *not* secrets. You normally do not hand-export -`INFERENCEX_PATH` or `TRACELENS_ROOT` — `install.sh` and its chained +These are *not* secrets. For a native AgentX launch, normally omit both +`benchmark.inferencex_path` and `INFERENCEX_PATH`; preflight reuses or clones the +tested pin. A source `benchmark.inferencex_path` only nominates a preferred +writable checkout for that fresh session. Preflight replaces a missing or +wrong-revision checkout with its pinned clone; a correct but non-writable +explicit checkout fails. A simultaneously supplied, different ambient +`INFERENCEX_PATH` conflicts with the source before preflight. You likewise do +not normally hand-export `TRACELENS_ROOT` — `install.sh` and its chained kernel-agent installer clone and pin the open-source checkouts when missing. Runtime paths are persisted into `.env` by the installer; the CLI preflight loads them and derives `PATH` / `LD_LIBRARY_PATH` from `ROCM_PATH` / @@ -262,16 +268,32 @@ Open-source dependencies default under `HYPERLOOM_CACHE_DIR` (`$REPO_ROOT/.cache`), cloned per revision as `@`. The cache is repo-local and writable, so open-source runs need no privileged `/opt` mount. -Leave these variables unset unless you maintain your own checkouts. An -explicit path pointing at a missing directory fails preflight. +Leave these variables unset unless you maintain your own checkouts. Most +explicit missing dependency paths fail preflight; native AgentX is the +exception described above because InferenceX preflight repairs it with the +pinned clone. | Variable | Set by operator? | Default / auto-clone target | Description | |------------------------------|------------------|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| | `HYPERLOOM_CACHE_DIR` | rarely | `$REPO_ROOT/.cache` | Writable, repo-local root for open-source deps (TraceLens, InferenceX, GEAK), cloned per revision as `@`. | -| `INFERENCEX_PATH` | optional override | `${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/InferenceX@` | [SemiAnalysisAI/InferenceX](https://github.com/SemiAnalysisAI/InferenceX) for baseline and target analysis; the inference_optimizer installer (`src/hyperloom/inference_optimizer/assets/install.sh`) clones it when unset. | +| `INFERENCEX_PATH` | optional legacy override | `${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/InferenceX@` | [SemiAnalysisAI/InferenceX](https://github.com/SemiAnalysisAI/InferenceX) for baseline, native AgentX recipe resolution, and target analysis. Omit it to reuse/clone the tested pin. Source `benchmark.inferencex_path` may nominate a checkout; a conflicting ambient value fails before preflight, a missing or wrong-revision candidate is replaced by the pinned clone, and a correct but non-writable explicit checkout fails. | | `TRACELENS_ROOT` | optional override | `${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/TraceLens@` | [AMD-AGI/TraceLens](https://github.com/AMD-AGI/TraceLens) for profiling and kernel detection; the kernel-agent installer clones and pins it when unset. | | `TRACELENS_INTERNAL_ROOT` | optional | unset (MAF measured on-device) | Optional internal TraceLens extension that backfills MAF without an on-device benchmark. When unset, Hyperloom measures MAF on an idle GPU (microbenchmark) — roofline gap / MI355+ MAF analysis is still produced, just measured locally. Hyperloom never clones it. | -| `MAGPIE_PATH` | optional override | Resolved from installed `Magpie` package | Magpie package root for benchmark wrappers and patch inspection. `install.sh` pip-installs Magpie from `MAGPIE_PACKAGE_SPEC` when it is not importable. | +| `MAGPIE_PATH` | optional override | Resolved from installed `Magpie` package | Magpie package root for benchmark wrappers and native AgentX resolution. `install.sh` installs `MAGPIE_PACKAGE_SPEC` when Magpie is missing **or** the importable build lacks the pinned native AgentX capability. | + +Native AgentX defaults to `MAGPIE_REF=d80eb4d3dad7fabe01ce81d049e2983adf2c86dd` +(Magpie v0.3.0 plus native launch overrides, custom-model replay, and client/eval compatibility fixes) and +`INFERENCEX_REF=408c015be4b22d14c69518643609669405507077`. Treat changing either +pin as a coordinated compatibility change, not as an authentication override. +That strict 40-character SHA, native capability, and audited Magpie execution-tree +check applies only when native AgentX is selected. Generic Magpie benchmarks keep +the established importability contract and may use an operator-provided compatible +tag, branch, alternate SHA, or `MAGPIE_PACKAGE_SPEC`. +Native `benchmark.docker_image` overrides/pins the effective recipe image and is +part of its fingerprint; it is not a credential. An existing +`HYPERLOOM_IMAGE` is checked for exact agreement with that effective value, but +neither starts a container or cryptographically proves which outer image is +running. --- diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md index 7c114f3ffd..fbf2fd3dcd 100644 --- a/docs/reference/environment-variables.md +++ b/docs/reference/environment-variables.md @@ -77,11 +77,12 @@ The following variables configure filesystem paths for Hyperloom's runtime depen ## Workload configuration -Set with CLI flags, not env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` / -`TP` / `EP` env vars are ignored and overwritten (`GPU_TYPE` is a fallback when -`--gpu-type` is omitted). +Set with CLI flags or an explicit `--benchmark-config`, not ambient workload +env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` / `TP` / `EP` env vars +are ignored and overwritten (`GPU_TYPE` is a fallback when `--gpu-type` is +omitted). -- **Model / workload shape:** `--model`, `--model-class`, `--framework`, +- **Model / workload shape:** `--benchmark-config`, `--model`, `--model-class`, `--framework`, `--framework-version`, `--precision`, `--tp`, `--ep`, `--isl`, `--osl`, `--conc`, `--max-model-len`, `--profile-osl`. - **Goal / budget:** `--target-gain`, `--target-roofline`, `--max-hours`, @@ -327,7 +328,7 @@ consumers and triage read a contract rather than candidate internals. | Variable | Default | Description | |----------|---------|-------------| -| `INFERENCE_OPTIMIZER_RAY_EXEC` | Unset (`on` for single-node) | Controls whether single-node serving benchmarks and `needs_gpu` specialists run through Ray actors. When unset, single-node runs are routed through Ray-managed leases while multi-node stays on the multi-node backend. Set to `0` / `false` / `no` / `off` to force the local subprocess path, or `1` / `true` / `yes` / `on` to force Ray. | +| `INFERENCE_OPTIMIZER_RAY_EXEC` | Unset (`on` for ordinary single-node runs) | Controls whether single-node serving benchmarks and `needs_gpu` specialists run through Ray actors. When unset, ordinary single-node runs are routed through Ray-managed leases while multi-node stays on the multi-node backend. Set to `0` / `false` / `no` / `off` to force the local subprocess path, or `1` / `true` / `yes` / `on` to force Ray. Native AgentX is the exception: it always bypasses the outer Ray actor, and explicitly forcing Ray is rejected. | --- @@ -869,14 +870,105 @@ output tokens per request, so output-only grading optimises about 1% of the token budget, and a variant can lift decode tok/s while degrading user-perceived latency with no visible cost. -`HYPERLOOM_AGENTX=1` adopts the 2-D grading shape InferenceX uses. Upstream -sweeps a concurrency ladder, keeps TTFT / ITL / TPOT percentiles separately, and -publishes a **Pareto frontier** with E2E normalised interactivity as the x-axis +An input YAML with `benchmark.agentx: enable`, passed through +`--benchmark-config`, automatically adopts the 2-D grading shape InferenceX +uses and persists AgentX session mode. Upstream InferenceX submissions sweep a +concurrency ladder, keep TTFT / ITL / TPOT percentiles separately, and +publish a **Pareto frontier** with E2E normalised interactivity as the x-axis and token throughput per chip as the y-axis. It never collapses the axes into one weighted number, and it has **no fixed interactivity target** — interactivity is a frontier coordinate, not a constraint, so a point that trades interactivity for throughput moves along the frontier rather than violating a rule. +Native AgentX measurements are materialized from that source YAML as Magpie +`agentx: enable` runs. +The canonical identity and launcher are intentionally separate from the local +checkpoint path. The pinned pair is Magpie v0.3.0 plus native launch overrides, custom-model replay, and client/eval +compatibility fixes at commit `d80eb4d3dad7fabe01ce81d049e2983adf2c86dd` and InferenceX commit +`408c015be4b22d14c69518643609669405507077`: + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `HYPERLOOM_BENCHMARK_CONFIG` | No | None | Legacy environment alias for CLI `--benchmark-config`. Prefer the CLI flag. Fresh launches only. Resume restores the accepted materialized config, or the snapshotted source config when baseline was not accepted yet. | +| `HYPERLOOM_AGENTX` | No | Off | Enables Magpie native AgentX for fresh sessions, including Hyperloom optimization. Persisted epoch-1 sessions keep their legacy client and epoch-2 sessions keep their native measurement-only contract. Native `benchmark.agentx: enable` sets AgentX session mode automatically; an explicitly false environment value conflicts and is rejected. | +| `HYPERLOOM_IMAGE` | No | Effective resolved image | Optional strict consistency assertion for native local mode. `benchmark.docker_image` overrides/pins the effective recipe image; otherwise the recipe default is used. If this variable already exists, it must match exactly. It does not attest the running container. | +| `AGENTX_MODEL_ID` | Native: only without `benchmark.model` | None | For native AgentX, fallback for the exact model id from the selected InferenceX recipe (for example `amd/GLM-5.2-MXFP4`). A separate CLI `--model` may name the local checkpoint and is emitted as `MODEL_PATH`; omitting it uses the source model id remotely. | +| `AGENTX_SERVER_SCRIPT` | No | Resolved by Magpie | For managed AgentX, the resolved client is `srt_agentic.sh`; serving comes from Magpie's resolved specification. Persisted shell-launcher sessions retain their original launcher identity and restrictions. | +| `INFERENCEX_PATH` | No | Installer pin | Legacy fallback for source `benchmark.inferencex_path`. The source path nominates a preferred writable checkout. Preflight replaces a missing or wrong-revision path with a pinned clone; an explicit correct but non-writable checkout fails. A simultaneously supplied, different ambient path conflicts with the source before preflight. Managed AgentX accepts the repository root or `inferencex-e2e/` project directory, including spaces. Saved shell-launcher sessions retain their original path restrictions. | +| `AGENTX_MODE` | No | `canonical` | Environment equivalent for enabled native AgentX of `benchmark.agentx.mode`. `canonical` uses the 3600-second native protocol and is required by `optimize`. `fast` uses 1200 seconds and is a direct-Magpie diagnostic; its non-publishable result is rejected as a Hyperloom baseline. | +| `AGENTX_RECIPE` | When inference is ambiguous | Inferred by Magpie | Environment equivalent for enabled native AgentX of `benchmark.agentx.recipe`: an exact recipe key from the pinned InferenceX config. | +| `AGENTX_CONFIG_FILE` | No | InferenceX `configs/amd-master.yaml` or `nvidia-master.yaml` | Environment equivalent for enabled native AgentX of `benchmark.agentx.config_file`: an explicit recipe-config path understood by Magpie. | +| `AGENTX_SELECTOR` | When one recipe/concurrency has multiple arms | `{}` | Environment equivalent for enabled native AgentX of `benchmark.agentx.selector`, encoded as a JSON object, for example `{"tp":4,"kv_offloading":"dram","kv_offload_backend":"hicache"}`. Prefer the YAML object with `--benchmark-config`. | +| `AGENTX_FAILED_REQUEST_THRESHOLD` | No | `0.10` | Environment equivalent for enabled native AgentX of `benchmark.agentx.failed_request_threshold`, from `0` through `1`. | + +The source YAML can provide `benchmark.model`, `benchmark.precision`, +`benchmark.framework`, `benchmark.runner_type`, `benchmark.run_mode`, +`benchmark.benchmark_script`, optional `benchmark.inferencex_path`, effective +`benchmark.docker_image`, `benchmark.envs.CONC`, optional topology assertions, +and either scalar `benchmark.agentx: enable` or the object form: + +```yaml +agentx: + enabled: true + recipe: glm5.2-fp4-mi355x-sglang-agentic-mtp # only if inference is ambiguous + selector: # only if this CONC has multiple arms + tp: 4 + kv_offloading: dram + kv_offload_backend: hicache +``` + +Hyperloom checks those source fields before session state is seeded. The image +field overrides/pins the effective recipe image and is included in its +fingerprint; Hyperloom does not inspect the runtime to prove the actual +container. The selected recipe owns logical TP/PP/PCP/EP, `MODEL_PREFIX`, +`KV_OFFLOADING`, `KV_OFFLOAD_BACKEND`, and `TOTAL_CPU_DRAM_GB`; users do not copy +those fields into `benchmark.envs`. In particular, `envs.TP` is not an arm +selector. A concurrency not present in the recipe, or an ambiguous point +without `agentx.selector`, fails before launch. Hyperloom pre-resolves each +point through the same benchmark interpreter that will run Magpie. Native +InferenceX owns the duration (3600 seconds in canonical mode, 1200 in fast +mode) and configures a 393-trace dataset-entry cap. That value is a loader +ceiling, not a guarantee that 393 traces, sessions, or requests survive +availability and context-length filters. Native launches reject non-empty +`AIPERF_*`, `AGENTIC_*`, `AGENTX_DATASET`, `AGENTX_WARMUP_REQUESTS_PER_LANE`, +and `WEKA_LOADER_OVERRIDE` in the shell or `benchmark.envs` during CLI preflight, +before dependency installation or benchmarking. Remove those legacy overrides; +configure replay through supported `benchmark.agentx` options and the resolved +InferenceX recipe. Legacy sessions and the MLPerf backend retain their own controls. +A pre-existing `HYPERLOOM_IMAGE` must exactly match the effective +resolved image. In contrast, +`AGENTX_DURATION` and `AGENTX_NUM_ENTRIES` still control legacy measurements +and saved compatibility-profiler sessions. They do not change managed AgentX measurement +duration or corpus size. Legacy `AGENTX_DATASET` and `WEKA_LOADER_OVERRIDE` +remain supported on the legacy backend. + +Hyperloom owns the fixed concurrency of native measurement rounds through +CLI `--conc` or source/materialized `benchmark.envs.CONC`. It removes +`benchmark.agentx.concurrency` from the source YAML because Magpie gives that +field precedence over `envs.CONC`. New native sessions retain the optimizer's +concurrency sweep. Saved epoch-2 sessions keep sweep disabled; explicitly +enabling it on those sessions fails preflight. + +When omitted, Hyperloom fills outer `--tp` with the resolved `TP×PP×PCP` +physical GPU count and `--ep` with recipe EP. Explicit values are exact +consistency assertions. Topology-changing AgentX sweeps are rejected. The +pinned launchers copy physical ROCR values back into logical +`HIP_VISIBLE_DEVICES`, so Hyperloom derives `gpu_selection.auto=false` and +`ROCR_VISIBLE_DEVICES=0,...,N-1` when omitted. Explicit masks are assertions; +nonzero, reordered, short, or long masks are rejected rather than remapped. + +Hyperloom's native bridge currently supports local, single-node SGLang and +vLLM. It automatically bypasses Hyperloom's outer Ray actor; leave +`INFERENCE_OPTIMIZER_RAY_EXEC` unset or set it to `0`. An explicit true value is +rejected. Multi-node/disaggregated AgentX, `server_lifecycle`, and Atom are not +supported. + +Fresh native sessions use the Magpie-managed launch-overrides contract. +Candidate arguments, environment controls, and source overlays have distinct +execution identities, while the canonical workload fingerprint stays fixed. +Unsupported controls fail closed. Existing epoch-2 sessions preserve the old +measurement-only contract and immutable materialized execution fingerprint. + Hyperloom's local KEEP rule (fixed concurrency, no ladder) approximates the per-concurrency arm selection maintainers apply before submitting: @@ -892,32 +984,78 @@ into the session a round lands, so the decaying session threshold does not apply to it. Total token throughput no longer participates in the verdict — on this corpus it is almost entirely prefill, so it cannot see an output collapse. -Hyperloom reads `e2e_norm_intvty_p90` from the accepted `current_best` for both -grading and advisory comparison. This is aiperf's summary **P10** of the -per-request rate `OSL / E2EL_s`, representing the slow tail for a -`LARGER_IS_BETTER` metric. Comparison does not recompute request-level metrics. -The external reference uses `1 / P90(E2EL_s / OSL)`; finite-sample linear -interpolation means the estimators need not be numerically identical. -The comparison is advisory and does not change KEEP/REVERT. +Hyperloom reads `e2e_norm_intvty_p90` from native Magpie's +`agentx_metrics.latency_seconds`. InferenceX computes it as +`1 / P90(E2EL_s / OSL)`, so it is already the slow-tail rate for a +`LARGER_IS_BETTER` metric; Hyperloom does not multiply it by 1000 or invert it +again. Older profiler-compatibility artifacts carry AIPerf's P10 of the +per-request `OSL / E2EL_s` rate. The two finite-sample estimators can differ; +profile output is diagnostic and does not replace the native measurement used +for KEEP/REVERT. TTFT is included in E2EL, unlike per-user `1/ITL`; on a ~114k-prompt replay TTFT is most of what a user waits for so grading on `1/ITL` would miss it. Default-on for AgentX runs, explicit opt-in via `HYPERLOOM_PERF_METRIC=intvty_v1` -otherwise. Either AgentX signal turns it on: the ambient `HYPERLOOM_AGENTX=1`, or -`benchmark_mode=agentx` stamped at seed — so a round in a subprocess that never -inherited the env var still grades on the agentic axis. Serving frameworks only; -scriptable frameworks (xDiT, custom) keep output-throughput grading. +otherwise. Source `benchmark.agentx: enable` stamps `benchmark_mode=agentx` at +seed; the fresh-launch `HYPERLOOM_AGENTX=1` switch does the same — so a round in a +subprocess that never inherited the env var still grades on the agentic axis. +Serving frameworks only; scriptable frameworks (xDiT, custom) keep +output-throughput grading. | Variable | Default | Description | |--------------------------------|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `HYPERLOOM_PERF_METRIC` | `intvty_v1` under `HYPERLOOM_AGENTX=1`, else output tput | `intvty_v1` grades E2E normalised interactivity P50 (median) as the primary objective, with the P90 slow tail and output throughput each held as a guard. Reported in the final summary as `grading mode`. | +| `HYPERLOOM_PERF_METRIC` | `intvty_v1` in AgentX session mode, else output tput | `intvty_v1` grades E2E normalised interactivity P50 (median) as the primary objective, with the P90 slow tail and output throughput each held as a guard. Reported in the final summary as `grading mode`. | | `HYPERLOOM_PERF_NOISE_PCT` | `5.0` | Noise band in percent applied to the guards. A candidate whose slow tail or output throughput sits within this band of the anchor is not considered worse on that axis; the median has its own fixed bar and is not subject to the band. The default is the top of the 1–5% run-to-run noise upstream records for this workload. An unparseable value raises `EnvValueError` naming the variable, rather than grading against a band nobody chose. | | `HYPERLOOM_ALLOW_UNVERIFIED_SUBMISSION` | Unset (fail closed) | Truthy accepts a measurement whose submission verdict is absent or undetermined (`submission_valid=None`). A measurement the scenario explicitly judged invalid (`submission_valid=False`) is always rejected regardless of this flag. Applies to every measurement the run accepts (baseline, explore, kernel, sweep), not only the baseline — an unverified measurement makes every gain derived from it unverifiable. | | `INFERENCE_OPTIMIZER_BASELINE_SERVER_READY_SEC` | `7200` | Initial server-boot budget written by the persistent-server lifecycle configuration helper. Actual benchmark launches synchronize this field to `INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC`; this variable is not an additional benchmark deadline or a way to extend one. Non-benchmark lifecycle callers that do not perform that synchronization retain their own boot budget. | +Epoch-4 native profiling derives a diagnostic Magpie run from the accepted +candidate configuration. Magpie owns server lifecycle, measured-phase gating, +step capture, and framework profiler arguments. The diagnostic torch settings +are `num_steps`, `num_profiles`, `start_seconds`, `interval_seconds`, +`capture_timeout_seconds`, `flush_timeout_seconds`, and `detailed_annotations`. +The first delay is measured from AIPerf measurement start; each later interval +starts after the previous trace flush. Magpie computes the capture upper bound +from the remaining measurement duration and reduces oversized requests. + +Hyperloom consumes `agentx_metrics.profile_capture` and chooses one complete +capture for kernel/roofline analysis. It preserves all per-round artifacts and +the requested, planned, effective, and completed counts. Normal count reduction +is not a capture failure. A failed run, cancellation, or incomplete trace is +still a failure. Detailed annotations require a compatible instrumented +framework and are configured by Magpie; an accepted runtime/source overlay is +preserved, and installed framework files are not silently patched. + +Diagnostic runs deliberately set `benchmark_valid=false` and +`publishable=false`; successful capture does not promote their performance +numbers to baseline or KEEP. System profiling and gap analysis remain unsupported +for native AgentX. Saved epoch-3 sessions keep their earlier compatibility +profiler and its controls. Epoch-2 sessions keep their measurement-only contract. +GEAK proposals still require Critic-reviewed source integration and canonical +native AgentX remeasurement before acceptance. + +For native results, Hyperloom requires both `benchmark_valid=true` and +`publishable=true`, an `agentic-coding` scenario, and a valid recipe fingerprint, +including exact recipe-to-launch and recipe-to-raw fingerprint matches. It +accepts GPU count only from a trusted, fingerprint-bound report, Magpie config +snapshot, materialized workload record, or InferenceX raw aggregate. The raw +aggregate retains measured token-length distributions and cache metrics. +Magpie's `publishable` bit attests the canonical AgentX protocol and recipe +fingerprint. Hyperloom separately binds the selected recipe, resolved server +specification, client sources, and pinned checkout. It does not cryptographically attest the +actual outer image. The execution identity covers the resolved +`BenchmarkConfig` and the effective, scrubbed launcher environment for the +audited server/framework/runtime controls. It intentionally excludes +credentials, cache routing, output paths, and unrelated login-shell variables. + ### MLPerf agentic backend +The explicit MLPerf backend keeps its existing client path. Fresh AIPerf +sessions use native Magpie AgentX. Combining a native AgentX source YAML with +`HYPERLOOM_AGENTIC_BACKEND=mlperf` is rejected because they select different +workloads. + `HYPERLOOM_AGENTIC_BACKEND=mlperf` (with `HYPERLOOM_AGENTX=1`) keeps Magpie's server lifecycle and swaps the AgentX client for the MLCommons `inference-endpoint` harness (`utility/run_agentic.sh`), replaying @@ -959,7 +1097,9 @@ defaults to `trajectories × 26 s × 2 + 1800 s` (26 s per trajectory measured o 8×MI355X at concurrency 16, doubled for a cold first round, plus boot), and never below the stock 7800 s: about 9600 s for smoke and 33,700 s for `full`. -AgentX profiling starts when AIPerf reports its measured phase. +### Saved compatibility-profiler sessions + +For saved epoch-3 AIPerf sessions, profiling starts when AIPerf reports its measured phase. `AGENTX_PROFILE_WINDOW_S` controls the capture window and defaults to 20 seconds; phase waiting is bounded by the materialized benchmark timeout. Capture lifecycle status is written to a per-invocation `capture-status.json`; the adjacent diff --git a/docs/reference/session-breakdown.md b/docs/reference/session-breakdown.md index a10495b0cd..93d1a6068e 100644 --- a/docs/reference/session-breakdown.md +++ b/docs/reference/session-breakdown.md @@ -280,9 +280,17 @@ downstream consumers: | `invocation` | Same shape as `baseline.invocation`; lets a consumer replay the final benchmark. | | `closing_phase_entered` | True iff Coordinator entered the closing phase cleanly (vs SIGTERM exit). | -> **Scope of `throughput_tok_s_per_gpu`: whole-server total in `throughput_unit`, not per-GPU.** -> The key name is a misnomer held fixed by this contract; do not divide it by a GPU count. -> `cumulative_gain_pct_validated` is a ratio of two such numbers and is unaffected. +> **Legacy scope of `throughput_tok_s_per_gpu`: whole-server total in `throughput_unit`.** +> Existing synthetic and legacy AgentX records retain this historical contract; +> the key name alone does not establish per-GPU normalization. +> +> Native AgentX records additionally retain the aggregate in `throughput_tok_s`. +> When the measurement records `agentx_gpu_count`, `throughput_tok_s_per_gpu` +> contains the aggregate divided by that physical count, with unit `tok/s/GPU`. +> Without a recorded count, the per-GPU field is null and `throughput_unit` is +> `tok/s (aggregate)`. Do not divide an already normalized native display value +> again or infer the count from TP×EP. Grading keeps the original measurement +> axes, and `graded_on` identifies the metric used for validated gain. > Consumer best practice: index on > `(session.session_id, final.throughput_tok_s_per_gpu, diff --git a/docs/release-notes.md b/docs/release-notes.md index e8a07f9d73..c3c53e2a00 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -93,6 +93,77 @@ is a patch release on top of 1.1.2. The per-change history is on the ### Added +- **Use native Magpie/InferenceX AgentX throughout new optimization sessions.** + Fresh `HYPERLOOM_AGENTX=1` launches resolve the native recipe and launcher from + ordinary workload inputs and retain Hyperloom's optimization loop. New sessions + persist backend `native`, epoch 4, and the Magpie-managed launch contract. + Models without a registered recipe can use the formal custom workload path + with an explicit image, TP, EP, and concurrency; Magpie validates model + context instead of borrowing an unrelated model's launcher. + Existing epoch-1 sessions resume the legacy client; epoch-2 native sessions + retain their measurement-only contract. No historical baseline or KEEP is + migrated. Unsupported or ambiguous recipes fail before measurement. The + pinned pair is Magpie v0.3.0 plus native launch overrides, custom-model replay, and client/eval compatibility fixes at commit + `d80eb4d3dad7fabe01ce81d049e2983adf2c86dd` and InferenceX commit + `408c015be4b22d14c69518643609669405507077`. Both dependencies are pinned by + immutable commit for reproducible AgentX measurements. SGLang detailed profiling + no longer requires shape discovery; unsupported shape-discovery options are omitted, + while graph profiling support remains required for graph-mode captures. + The upstream hotfix + keeps generic GSM8K evaluation and its probe files reachable after benchmark + directory changes. The packaged benchmark client uses Transformers 5 in its + own Python environment, without upgrading the serving framework. Client tokenizer + and trust settings are supported directly by the pinned scripts, preserving the + audited package during compatibility checks. Install preserves the + audited package and launcher trees; native preflight repairs Magpie files changed by an + earlier compatibility patch. Generic installs retain the importability contract + for custom or already-installed packages. Native CLI preflight rejects non-empty + legacy `AIPERF_*`, `AGENTIC_*`, dataset and warmup-per-lane overrides in the + shell or source YAML, before installation or benchmarking. Fixed-sequence `ISL`, `OSL`, and + `RANDOM_RANGE_RATIO` controls are removed by Magpie's AgentX configuration. + A new + `--benchmark-config ` option accepts the Magpie source config; + `benchmark.agentx: enable` automatically selects Hyperloom's AgentX session + and grading mode. `HYPERLOOM_AGENTX=1` is an alternative fresh-launch input, + not a second requirement. The YAML can provide `benchmark.model`, + `benchmark.precision`, `benchmark.framework`, `benchmark.runner_type`, + `benchmark.run_mode`, `benchmark.benchmark_script`, optional + `benchmark.inferencex_path`, effective `benchmark.docker_image`, fixed + `benchmark.envs.CONC`, and YAML-native `benchmark.agentx.selector`. Hyperloom + pre-resolves each recipe through the benchmark interpreter, + separates a canonical model id from local `MODEL_PATH`, resolves the + Magpie server specification and upstream client, and validates strict recipe/launch/raw + fingerprints plus trusted topology. This addresses + [#1601](https://github.com/AMD-AGI/Hyperloom/issues/1601). + **Upgrade note:** native local mode takes its effective image pin from + `benchmark.docker_image` (an override included in the fingerprint) or the + resolved recipe default; an existing `HYPERLOOM_IMAGE` must match it. Omitted + `--tp`/`--ep` are filled from resolved recipe topology; explicit values are + consistency assertions. Topology changes are rejected, and Hyperloom derives + the zero-based `ROCR_VISIBLE_DEVICES` mask with `gpu_selection.auto=false`. + Native AgentX + bypasses outer Ray; explicitly enabling `INFERENCE_OPTIMIZER_RAY_EXEC` fails. + New sessions use the Magpie-managed launch-overrides contract for candidate + arguments, environments, and source overlays. Canonical revalidation checks + launch evidence and a fixed workload fingerprint; candidate execution + identities remain distinct. Epoch-2 resumes retain their earlier restrictions. + Epoch-4 profiling uses the accepted native candidate and Magpie's measured-phase + torch profiler. It supports a first-capture delay, step count, repeated captures, + and automatic count reduction. Successful standalone `profile` tasks publish + their trace to the Coordinator without becoming valid performance measurements. + Complete captures are consumed separately; + cancelled or failed captures remain failures. Detailed annotations require a + compatible instrumented framework. Diagnostic results are never baseline or + KEEP evidence. Saved epoch-3 sessions keep their upstream launcher contract. + GEAK proposals require canonical AgentX revalidation before acceptance. + Finally, `publishable` + is only the Magpie protocol attestation; Hyperloom separately binds the + selected recipe to the resolved server specification, client sources, and pinned checkout. It + does not cryptographically attest the actual outer image. The execution + identity additionally hashes the effective, scrubbed launcher environment + for audited server/framework/runtime controls, while excluding credentials, + cache routing, output paths, and unrelated login-shell variables. + - **`--repo-scope` / `--no-repo-scope` for `forge-fuse`.** Give discovery and authoring the whole framework repository instead of one resolved file. Requires `--discover llm` or `--discover anchored`. Use when the fusion chain @@ -1169,7 +1240,9 @@ are marked below; the full per-change list is in the `HYPERLOOM_AGENTX`); `--conc-sweep-concs` still overrides both. The `sweep` action is gone from the LLM catalogue, the executor registry and the phase contract, and `conc_sweep_done` / `conc_sweep_failed` collapse into - `sweep_done` / `sweep_failed` with no alias for the old spelling. + `sweep_done` / `sweep_failed` with no alias for the old spelling. This records + the 1.1.0 behavior; the Unreleased native AgentX bridge above supersedes its + AgentX default by disabling the sweep and rejecting explicit enablement. - **The post-KEEP confirmation round is removed** *(breaking change — session record)*: an `explore` variant and an `integrate_patch` candidate were each re-benched diff --git a/examples/hyperloom-agentx-glm-5.2-mi355x/README.md b/examples/hyperloom-agentx-glm-5.2-mi355x/README.md new file mode 100644 index 0000000000..7dd24aa285 --- /dev/null +++ b/examples/hyperloom-agentx-glm-5.2-mi355x/README.md @@ -0,0 +1,56 @@ +# Native Magpie AgentX: GLM-5.2 on MI355X + +This example pins Magpie v0.3.0 plus native launch overrides, custom-model replay, and client/eval compatibility fixes to commit +d80eb4d3dad7fabe01ce81d049e2983adf2c86dd and InferenceX commit +408c015be4b22d14c69518643609669405507077. The YAML omits +`inferencex_path`, so preflight reuses or clones that tested pin. An optional +`benchmark.inferencex_path` only nominates a preferred writable checkout: +preflight replaces a missing or wrong-revision checkout with its pinned clone. +Both repository-root and `inferencex-e2e/` project paths are supported for +managed AgentX, including paths containing spaces. + +Run Hyperloom from inside the exact recipe image: + + python -m hyperloom.inference_optimizer.cli optimize \ + --benchmark-config examples/hyperloom-agentx-glm-5.2-mi355x/benchmark.yaml \ + --max-hours 3 + +The explicit three-hour budget is sized for one canonical baseline. The normal +two-hour default is commonly shorter than model load, warmup, drain, and the +3600-second measurement together; increase it if additional rounds are intended. + +As written, the source model id is fetched through the normal Hugging Face +path, and the native InferenceX subprocess receives no local `MODEL_PATH` +override. To use a mounted checkpoint, keep the canonical recipe id in +`benchmark.model` and add `--model /absolute/path/to/checkpoint`. + +The YAML source switch automatically enables the session-wide AgentX contract; +fresh `HYPERLOOM_AGENTX=1` launches also select native AgentX. New sessions +use epoch 4 and keep Hyperloom optimization. Persisted epoch-1 and epoch-2 +sessions keep their original backend and measurement contract. +At `CONC=8`, the recipe has one GLM-5.2 arm: TP4/EP4 with DRAM+HiCache. +`envs.TP` is not an arm selector and is intentionally absent. If a concurrency +belongs to multiple arms, change `agentx: enable` into the object form shown in +the YAML comments and select the arm under `agentx.selector`; keep +`envs.CONC` as the fixed measurement point. Magpie resolves and persists the +recipe-owned TP/PP/PCP/EP, model prefix, HiCache, DRAM, CPU-memory, and +fingerprint fields. + +The YAML `docker_image` overrides/pins the effective recipe image and is folded +into its fingerprint; an optional pre-existing `HYPERLOOM_IMAGE` must match it. +Neither field starts a container or proves which outer image is actually +running, so launch Hyperloom inside that image before invoking the command. + +New sessions use Magpie-managed serving and launch evidence for optimization. +Candidates require a canonical AgentX result; proxy and profiler scores cannot +be accepted as gains. Saved epoch-2 and epoch-3 sessions keep their original +measurement and launch contracts. + +Native diagnostic profiling retains the accepted candidate configuration and +uses Magpie's measured-phase torch capture. It supports a first-capture delay, +multiple captures, and automatic count reduction to fit the measurement window. +Hyperloom selects a complete capture for kernel/roofline analysis and keeps +other captures separately. Full shape-aware analysis requires a compatible +instrumented framework with detailed annotations; ordinary GPU traces do not +by themselves satisfy that requirement. Diagnostics never replace canonical +baseline or KEEP measurements. diff --git a/examples/hyperloom-agentx-glm-5.2-mi355x/benchmark.yaml b/examples/hyperloom-agentx-glm-5.2-mi355x/benchmark.yaml new file mode 100644 index 0000000000..89bb969b31 --- /dev/null +++ b/examples/hyperloom-agentx-glm-5.2-mi355x/benchmark.yaml @@ -0,0 +1,46 @@ +# Native Magpie AgentX example for the pinned InferenceX revision +# 408c015be4b22d14c69518643609669405507077. inferencex_path is intentionally +# omitted: Hyperloom preflight reuses or clones the tested pin. A custom +# benchmark.inferencex_path only nominates a preferred writable checkout. +benchmark: + framework: sglang + model: amd/GLM-5.2-MXFP4 + precision: fp4 + runner_type: mi355x + run_mode: local + + # This switch automatically selects Hyperloom's AgentX session contract when + # the file is passed with --benchmark-config. CONC=8 uniquely selects the + # TP4/EP4 DRAM+HiCache arm below. At a concurrency shared by multiple arms, + # replace this scalar with an object such as: + # agentx: + # enabled: true + # selector: + # tp: 4 + # kv_offloading: dram + # kv_offload_backend: hicache + agentx: enable + + # Magpie resolves serving from the pinned recipe and runs the upstream client. + # docker_image pins the effective runtime and participates in its fingerprint. + docker_image: lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260924 + + envs: + # This is the fixed recipe point to measure. TP/PP/PCP/EP, MODEL_PREFIX, + # KV offload, and CPU DRAM settings are resolved from the recipe. + CONC: 8 + # Optional explicit topology assertion. Hyperloom derives the same + # zero-based mask when it is omitted. + ROCR_VISIBLE_DEVICES: 0,1,2,3 + + # Also derived by Hyperloom for native AgentX; kept explicit as an assertion. + gpu_selection: + auto: false + + profiler: + torch_profiler: + enabled: false + system_profiler: + enabled: false + tracelens: + enabled: false diff --git a/scripts/refresh_inferencex_anchor_contract.py b/scripts/refresh_inferencex_anchor_contract.py index 768b117f44..d74ed5b900 100644 --- a/scripts/refresh_inferencex_anchor_contract.py +++ b/scripts/refresh_inferencex_anchor_contract.py @@ -15,12 +15,14 @@ It refuses to record a broken contract. If an anchor no longer matches, re-anchor it in ``_inferencex_patcher.py`` first, then run this again. -Requires ``gh`` authenticated against the (private) InferenceX repo. +Uses immutable Git objects from ``--checkout`` when supplied, otherwise requires +``gh`` authenticated against the InferenceX repo. Usage:: python scripts/refresh_inferencex_anchor_contract.py python scripts/refresh_inferencex_anchor_contract.py --ref + python scripts/refresh_inferencex_anchor_contract.py --checkout /path/to/InferenceX """ from __future__ import annotations @@ -48,10 +50,11 @@ def main() -> int: default=_INFERENCEX_REF_DEFAULT, help="commit to verify against (default: the pin the code installs)", ) + parser.add_argument("--checkout", type=Path, help="read pinned Git objects from a local repository") args = parser.parse_args() try: - record = build_record(args.ref) + record = build_record(args.ref, checkout=args.checkout) except RuntimeError as exc: print(f"refresh failed: {exc}", file=sys.stderr) return 1 diff --git a/src/hyperloom/common/agentx_accounting.py b/src/hyperloom/common/agentx_accounting.py new file mode 100644 index 0000000000..765472235a --- /dev/null +++ b/src/hyperloom/common/agentx_accounting.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Measured request errors from InferenceX's phase-overlapping accounting.""" + +from __future__ import annotations + +from typing import Any + + +def measured_request_errors(accounting: dict[str, Any]) -> int | float | None: + """Exclude warmup errors when the dropped union is available. + + InferenceX counts errors across both phases, while dropped rows are the + union of warmup and error rows. Older artifacts lack that union, so retain + their conservative all-phase error count instead of guessing an overlap. + """ + errors = accounting.get("records_error_dropped") + if isinstance(errors, bool) or not isinstance(errors, (int, float)) or not errors >= 0: + return None + if "records_dropped_total" not in accounting: + return errors + + dropped = accounting["records_dropped_total"] + warmup = accounting.get("records_warmup_dropped") + if not all(type(value) is int for value in (dropped, warmup, errors)): + return None + if not (0 <= warmup <= dropped and 0 <= dropped - warmup <= errors <= dropped): + return None + return dropped - warmup diff --git a/src/hyperloom/common/agentx_mode.py b/src/hyperloom/common/agentx_mode.py new file mode 100644 index 0000000000..223490ee19 --- /dev/null +++ b/src/hyperloom/common/agentx_mode.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Resolve AgentX routing from saved session identity or fresh launch inputs.""" + +from __future__ import annotations + +import json +import os +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import yaml + + +def native_agentx_enabled(value: Any) -> bool: + """Return whether a serialized Magpie ``benchmark.agentx`` enables AgentX.""" + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "enable", "enabled"} + if isinstance(value, dict): + raw = value.get("enabled", True) + if isinstance(raw, str): + return raw.strip().lower() in {"1", "true", "yes", "enable", "enabled"} + return bool(raw) + return False + + +def config_enables_native_agentx(path: str | Path) -> bool: + """Read the explicit native AgentX selector from a benchmark YAML.""" + try: + parsed = yaml.safe_load(Path(path).expanduser().read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError, TypeError, ValueError): + return False + benchmark = parsed.get("benchmark") if isinstance(parsed, Mapping) else None + return isinstance(benchmark, Mapping) and native_agentx_enabled(benchmark.get("agentx")) + + +def _session_state(state: Any, env: Mapping[str, str]) -> Any: + if state is not None: + return state + session = str(env.get("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", "") or "").strip() + if not session: + return None + path = Path(session).expanduser() / "state.json" + if not path.is_file(): + return None + parsed = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(parsed, Mapping): + raise ValueError(f"saved session state must be a mapping: {path}") + return parsed + + +def _read(state: Any, name: str, default: Any = None) -> Any: + return state.get(name, default) if isinstance(state, Mapping) else getattr(state, name, default) + + +def native_agentx_session(state: Any = None, *, env: Mapping[str, str] | None = None) -> bool: + """Resolve the native measurement contract from persisted state or source YAML. + + A supplied state is authoritative over the caller's shell. The native epoch + keeps a resumed session native even if its accepted config is unavailable; + resume validation must then report the missing pins instead of using legacy. + Diagnostic profiling templates cannot replace this session identity. + """ + source = os.environ if env is None else env + state = _session_state(state, source) + if state is not None: + mode = str(_read(state, "benchmark_mode", "") or "").strip().lower() + if mode and mode != "agentx": + return False + epoch = int(_read(state, "agentx_epoch", 0) or 0) + if mode == "agentx" and epoch: + return epoch >= 2 + backend = str(_read(state, "agentx_backend", "") or "").strip().lower() + if backend: + return backend == "native" + return any( + config_enables_native_agentx(path) + for name in ("baseline_config_path", "benchmark_source_config_path") + if (path := str(_read(state, name, "") or "").strip()) + ) + path = str(source.get("HYPERLOOM_BENCHMARK_CONFIG", "") or "").strip() + if str(source.get("HYPERLOOM_AGENTIC_BACKEND", "") or "").strip().lower() == "mlperf": + return bool(path) and config_enables_native_agentx(path) + enabled = str(source.get("HYPERLOOM_AGENTX", "") or "").strip().lower() + return enabled in {"1", "true", "yes", "on", "enable", "enabled"} or ( + bool(path) and config_enables_native_agentx(path) + ) + + +def native_agentx_optimization_session(state: Any = None, *, env: Mapping[str, str] | None = None) -> bool: + """Keep saved measurement-only sessions on their original epoch-2 contract.""" + source = os.environ if env is None else env + state = _session_state(state, source) + if not native_agentx_session(state, env=source): + return False + return state is None or int(_read(state, "agentx_epoch", 0) or 0) >= 3 + + +def managed_native_agentx_session(state: Any = None, *, env: Mapping[str, str] | None = None) -> bool: + """Use Magpie-managed serving for fresh sessions and the saved epoch-4 contract.""" + source = os.environ if env is None else env + state = _session_state(state, source) + if not native_agentx_session(state, env=source): + return False + return state is None or int(_read(state, "agentx_epoch", 0) or 0) >= 4 diff --git a/src/hyperloom/common/env_safety.py b/src/hyperloom/common/env_safety.py index f31bc62d76..fd5b9791ea 100644 --- a/src/hyperloom/common/env_safety.py +++ b/src/hyperloom/common/env_safety.py @@ -270,6 +270,18 @@ "PROFILE", "RESULT_DIR", "RESULT_FILENAME", + # Native AgentX recipe identity. Per-candidate changes would + # retarget the workload rather than tune the selected server. + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_RECIPE", + "AGENTX_CONFIG_FILE", + "AGENTX_SELECTOR", + "AGENTX_MODE", + "AGENTX_FAILED_REQUEST_THRESHOLD", + "AGENTX_DATASET", + "AGENTX_WARMUP_REQUESTS_PER_LANE", + "WEKA_LOADER_OVERRIDE", # Reroute traffic, model downloads or TLS trust. "CURL_CA_BUNDLE", "HF_ENDPOINT", @@ -285,7 +297,24 @@ ) # Env names a per-variant override may never set. -BLOCKED_VARIANT_ENV_NAMES: frozenset[str] = BLOCKED_UNTRUSTED_ENV_NAMES | BENCHMARK_SECRET_ENV_NAMES +BLOCKED_VARIANT_ENV_NAMES: frozenset[str] = ( + BLOCKED_UNTRUSTED_ENV_NAMES + | BENCHMARK_SECRET_ENV_NAMES + | frozenset( + { + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_RECIPE", + "AGENTX_CONFIG_FILE", + "AGENTX_SELECTOR", + "AGENTX_MODE", + "AGENTX_FAILED_REQUEST_THRESHOLD", + "AGENTX_DATASET", + "AGENTX_WARMUP_REQUESTS_PER_LANE", + "WEKA_LOADER_OVERRIDE", + } + ) +) # Credential-shaped name fragments, so an unlisted secret cannot be persisted into a session YAML by name alone. _SECRET_NAME_FRAGMENTS: tuple[str, ...] = ("APIKEY", "API_KEY", "TOKEN", "SECRET", "PASSWORD", "CREDENTIAL") diff --git a/src/hyperloom/common/gain_math.py b/src/hyperloom/common/gain_math.py index c462ac01d1..751a689fb9 100644 --- a/src/hyperloom/common/gain_math.py +++ b/src/hyperloom/common/gain_math.py @@ -8,7 +8,13 @@ from typing import Any from hyperloom.common.coerce import to_float -from hyperloom.common.perf_metric import GRADED_INTVTY, GRADED_TOTAL, graded_axes_of, holds_within_band +from hyperloom.common.perf_metric import ( + GRADED_INTVTY, + GRADED_TOTAL, + graded_axes_of, + passes_tput_guard, + total_tput_per_chip_of, +) def gain_pct(new: float | None, base: float) -> float | None: @@ -85,7 +91,9 @@ def _norm_conc(p: dict[str, Any]) -> int | float | str: opt_axes = graded_axes_of(o) if guard_axis else {} guard_holds: bool | None = None if guard_axis and base_axes.get(guard_axis) and opt_axes.get(guard_axis): - guard_holds = holds_within_band(opt_axes, base_axes, guard_axis, noise_pct=guard_noise_pct) + guard_holds = passes_tput_guard(opt_axes, base_axes, noise_pct=guard_noise_pct) + baseline_guard = total_tput_per_chip_of(base_axes) if guard_axis and base_axes.get(guard_axis) else None + optimized_guard = total_tput_per_chip_of(opt_axes) if guard_axis and opt_axes.get(guard_axis) else None rows.append( { "conc": c, @@ -100,8 +108,8 @@ def _norm_conc(p: dict[str, Any]) -> int | float | str: # The guard axis beside the objective, so the frontier this rung sits on is readable rather than # only the one number it was ranked by. Null off the interactivity objective, and null when a side # did not measure the axis -- which is not the same as a rung that measured it and fell outside. - "baseline_guard": base_axes.get(guard_axis) if guard_axis else None, - "optimized_guard": opt_axes.get(guard_axis) if guard_axis else None, + "baseline_guard": baseline_guard, + "optimized_guard": optimized_guard, "guard_holds": guard_holds, } ) diff --git a/src/hyperloom/common/perf_metric.py b/src/hyperloom/common/perf_metric.py index 593bae6927..ae269890e4 100644 --- a/src/hyperloom/common/perf_metric.py +++ b/src/hyperloom/common/perf_metric.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""AgentX grading: an E2E-normalised-interactivity objective guarded by per-chip throughput.""" +"""AgentX grading: median interactivity guarded by slow-tail interactivity and output throughput.""" from __future__ import annotations @@ -148,7 +148,7 @@ def _non_negative(value: Any) -> float | None: return coerced if coerced >= 0 else None -def perf_snapshot_from_mapping(source: Mapping[str, Any] | None) -> dict[str, float] | None: +def perf_snapshot_from_mapping(source: Mapping[str, Any] | None) -> dict[str, Any] | None: """The graded axes from a measurement or a ``current_best``; None unless objective and guards are all positive.""" # Requiring all of them is what stops a lane half-applying the objective. A total that is absent, null or # non-positive coalesces to input plus output, the same fallback ``agentx.mapping`` applies. @@ -165,7 +165,7 @@ def perf_snapshot_from_mapping(source: Mapping[str, Any] | None) -> dict[str, fl total = inp + out if intvty is None or intvty_p50 is None or total is None: return None - snap: dict[str, float] = { + snap: dict[str, Any] = { GRADED_INTVTY: intvty, GRADED_INTVTY_P50: intvty_p50, GRADED_TOTAL: total, @@ -180,12 +180,22 @@ def perf_snapshot_from_mapping(source: Mapping[str, Any] | None) -> dict[str, fl (GRADED_OUTPUT_PER_GPU, _positive(source.get(GRADED_OUTPUT_PER_GPU))), (GRADED_DURATION, duration), (GRADED_ERROR_RATE, error_rate), + ("agentx_gpu_count", _positive(source.get("agentx_gpu_count"))), ): if value is not None: snap[key] = value + snap.update(_native_comparison_identity(source)) return snap +def _native_comparison_identity(source: Mapping[str, Any]) -> dict[str, Any]: + return { + name: source[name] + for name in ("agentx_launch_contract", "agentx_workload_fingerprint", "agentx_candidate_fingerprint") + if name in source + } + + def output_tput_of(source: Mapping[str, Any] | None) -> float: """Output throughput from a measurement or a ``current_best``; 0.0 when absent.""" if not isinstance(source, Mapping): @@ -210,13 +220,22 @@ def total_tput_of(snapshot: Mapping[str, float] | None) -> float: return axis_of(snapshot, GRADED_TOTAL) -def graded_axes_of(source: Mapping[str, Any] | None) -> dict[str, float]: +def total_tput_per_chip_of(snapshot: Mapping[str, float] | None) -> float: + """Total token throughput per physical GPU; 0.0 when not measurable.""" + total = total_tput_of(snapshot) + if total <= 0 or not isinstance(snapshot, Mapping): + return 0.0 + gpu_count = _positive(snapshot.get("agentx_gpu_count")) + return total / gpu_count if gpu_count is not None else total + + +def graded_axes_of(source: Mapping[str, Any] | None) -> dict[str, Any]: """The graded axes *source* carries, for stamping onto a winner record.""" # A KEEP's ``current_best`` becomes the next candidate's anchor, and an anchor missing an axis degrades the whole # session to output grading. Axes are absent rather than None so a partial record is not read as a measured zero. if not isinstance(source, Mapping): return {} - axes: dict[str, float] = {} + axes: dict[str, Any] = _native_comparison_identity(source) intvty = _positive(source.get(GRADED_INTVTY)) if intvty is not None: axes[GRADED_INTVTY] = intvty @@ -241,10 +260,13 @@ def graded_axes_of(source: Mapping[str, Any] | None) -> dict[str, float]: error_rate = _non_negative(source.get(GRADED_ERROR_RATE)) if error_rate is not None: axes[GRADED_ERROR_RATE] = error_rate + gpu_count = _positive(source.get("agentx_gpu_count")) + if gpu_count is not None: + axes["agentx_gpu_count"] = gpu_count return axes -def resolve_grading_anchor_perf(state: Any) -> tuple[dict[str, float] | None, str]: +def resolve_grading_anchor_perf(state: Any) -> tuple[dict[str, Any] | None, str]: """Grading anchor: the current-best snapshot, falling back to the baseline; ``reason`` names any failure.""" # A ``current_best`` that exists but carries no axes must not fall through to ``baseline_perf`` -- that would # anchor a candidate against a recipe it was never measured on. @@ -284,12 +306,17 @@ def stamp_output_per_gpu(measurement: Any, tp: Any) -> None: measurement[GRADED_OUTPUT_PER_GPU] = out / chips -def rounds_are_comparable(candidate: Mapping[str, float], anchor: Mapping[str, float]) -> bool: +def rounds_are_comparable(candidate: Mapping[str, Any], anchor: Mapping[str, Any]) -> bool: """Whether the pair measured the same work: equal-length windows and no extra failed requests. Fails closed on an unreported input. A truncated round still publishes plausible rates, so treating "no evidence" as "comparable" is what lets one KEEP on a window it never ran. """ + identity_key = "agentx_workload_fingerprint" + if any("agentx_launch_contract" in side or identity_key in side for side in (candidate, anchor)): + identity = candidate.get(identity_key) + if not isinstance(identity, str) or len(identity) != 64 or identity != anchor.get(identity_key): + return False for side in (candidate, anchor): if not all(key in side for key in (GRADED_DURATION, GRADED_ERROR_RATE)): return False @@ -313,6 +340,25 @@ def holds_within_band( return _within_band(axis_of(candidate, key), axis_of(anchor, key), band) +def passes_tput_guard( + candidate: Mapping[str, float], + anchor: Mapping[str, float], + *, + noise_pct: float | None = None, +) -> bool: + """Whether per-chip candidate throughput holds within the band.""" + candidate_count = _positive(candidate.get("agentx_gpu_count")) + anchor_count = _positive(anchor.get("agentx_gpu_count")) + # A one-sided topology is not comparable: dividing only one side would mix + # aggregate and per-chip units. Old snapshots with neither count retain the + # historical aggregate comparison; new native results carry both counts + # from the InferenceX aggregate. + if (candidate_count is None) != (anchor_count is None): + return False + band = float(noise_pct if noise_pct is not None else parse_intvty_noise_pct()) + return _within_band(total_tput_per_chip_of(candidate), total_tput_per_chip_of(anchor), band) + + def latency_veto_reason(observed_ms: Any, budget_ms: float) -> str: """Why the latency budget refuses this candidate, or "" when it does not. @@ -397,9 +443,11 @@ def graded_on_intvty(self) -> bool: "latency_veto_reason", "output_tput_of", "parse_intvty_noise_pct", + "passes_tput_guard", "perf_snapshot_from_mapping", "resolve_grading_anchor_perf", "rounds_are_comparable", "stamp_output_per_gpu", "total_tput_of", + "total_tput_per_chip_of", ] diff --git a/src/hyperloom/common/tests/test_perf_metric.py b/src/hyperloom/common/tests/test_perf_metric.py index 589b0b0974..93fb6f1b00 100644 --- a/src/hyperloom/common/tests/test_perf_metric.py +++ b/src/hyperloom/common/tests/test_perf_metric.py @@ -18,6 +18,7 @@ intvty_serving_grading_enabled, output_tput_of, parse_intvty_noise_pct, + passes_tput_guard, perf_snapshot_from_mapping, resolve_grading_anchor_perf, total_tput_of, @@ -162,6 +163,21 @@ def test_tput_guard_rejects_regression_past_band(): assert holds_within_band(cand, anch, GRADED_TOTAL) is False +def test_tput_guard_normalizes_different_agentx_topologies_per_chip(): + anchor = perf_snapshot_from_mapping({**_BASELINE, "total_throughput": 400.0, "agentx_gpu_count": 4}) + candidate = perf_snapshot_from_mapping({**_BASELINE, "total_throughput": 600.0, "agentx_gpu_count": 8}) + assert anchor and candidate + # Aggregate throughput improved 50%, but each chip regressed 100 -> 75. + assert passes_tput_guard(candidate, anchor) is False + + +def test_tput_guard_rejects_one_sided_topology_metadata(): + anchor = perf_snapshot_from_mapping({**_BASELINE, "agentx_gpu_count": 4}) + candidate = perf_snapshot_from_mapping(_BASELINE) + assert anchor and candidate + assert passes_tput_guard(candidate, anchor) is False + + def test_vetoed_candidate_is_never_graded(): candidate = _measured(e2e_norm_intvty_p90=-20.0, input_throughput=50.0) assert _graded_gain(candidate, _BASELINE) is None diff --git a/src/hyperloom/inference_optimizer/SKILL.md b/src/hyperloom/inference_optimizer/SKILL.md index fea598822d..c8afc83de7 100644 --- a/src/hyperloom/inference_optimizer/SKILL.md +++ b/src/hyperloom/inference_optimizer/SKILL.md @@ -720,32 +720,34 @@ python3 -m hyperloom optimize \ ``` **Caller responsibility (post-classify-removal)**: the in-loop `setup` / -`classify` actions were deleted; the SKILL caller is now expected to -supply session metadata directly via CLI flags. **Any workload value the +`classify` actions were deleted; the SKILL caller must supply session metadata +through CLI flags or an explicit `--benchmark-config`. **Any workload value the operator states in the prompt (ISL, OSL, CONC, TP, EP, precision, budget, and -every `--extra-env`) MUST be forwarded as the matching CLI flag** — these flags -are the only source of truth; an omitted flag silently falls back to its default -and the operator's stated value is lost: +every `--extra-env`) MUST be forwarded as the matching CLI flag unless it is +intentionally present in that source YAML.** Do not rely on ambient workload +variables: without either source, the value falls back to its default and the +operator's stated value is lost. | Surface | CLI flag | Notes | |---|---|---| -| Model path | `--model` | required | -| Framework | `--framework` | `sglang` (default) / `vllm` / `atom` / `xdit` / `custom` — atom is single-node-only; xdit is scriptable diffusion (`img/s`, no serving server); `custom` is an operator-supplied workload and **additionally requires `--framework-path` and `--benchmark-scripts-dir`** (see below) | +| Benchmark YAML | `--benchmark-config` | Optional source Magpie config for a fresh launch. `benchmark.agentx: enable` automatically selects AgentX session mode; do not also require `HYPERLOOM_AGENTX`. A resume rejects this flag and restores the accepted materialized config, or the snapshotted source config when no baseline was accepted yet. | +| Model path | `--model` | Required unless the benchmark YAML supplies a remote `benchmark.model`. For native AgentX, use it to distinguish a local mounted checkpoint from the canonical model id in the YAML. | +| Framework | `--framework` | `sglang` (default) / `vllm` / `atom` / `xdit` / `custom`; a benchmark YAML may supply `benchmark.framework`, and a conflicting CLI value fails. Atom is single-node-only; xdit is scriptable diffusion (`img/s`, no serving server); `custom` is an operator-supplied workload and **additionally requires `--framework-path` and `--benchmark-scripts-dir`** (see below) | | Custom source tree | `--framework-path` | **Required for `--framework custom`.** The workload's own checkout; patches are authored against it. | | Custom bench scripts | `--benchmark-scripts-dir` | **Required for `--framework custom`.** Holds the entrypoint, looked up as `custom_.sh`. Every knob it reads must be forwarded as `--extra-env`; the throughput unit is whatever its report declares. | | GPU type | `--gpu-type` | rocm-smi auto-detect when unset | | Model class | `--model-class` | categorical key for the deterministic consumers (atom seed grid, framework-agent gap search token, recipe key, prompt label); when unset, Coordinator boot infers and persists it from model metadata or model-path family keywords. For richer advisory model context see Step 1.5 (`model_arch.json`) | -| Input seq length | `--isl` | Pass the prompt's ISL. Default `1024` when omitted. | -| Output seq length | `--osl` | Pass the prompt's OSL. Default `1024` when omitted. | -| Concurrency | `--conc` | Pass the prompt's CONC (max in-flight requests). Default `64`. SWEEP measures a ladder around it (under AgentX only with `--enable-conc-sweep`); `--conc-sweep-concs` overrides the workload's default ladder. | -| Tensor parallel | `--tp` | Pass the prompt's TP. Default `1`. | -| Expert parallel | `--ep` | Pass the prompt's EP for MoE. Default `1`. | -| Precision | `--precision` | Match the checkpoint (`bf16` default / `fp8` / ...). Keep consistent with `--quantize`. | -| Budget | `--max-hours` | Pass the prompt's time budget. Default `2.0`. | +| Input seq length | `--isl` | Pass the prompt's ISL. Default `1024` when omitted. Native AgentX measurement takes its distribution from the trace corpus instead. | +| Output seq length | `--osl` | Pass the prompt's OSL. Default `1024` when omitted. Native AgentX measurement takes its distribution from the trace corpus instead. | +| Concurrency | `--conc` | Pass the prompt's CONC (max in-flight requests). Default `64`. Outside native AgentX, SWEEP can measure a ladder and `--conc-sweep-concs` overrides it. Native AgentX measures this fixed recipe point, defaults sweep off, and rejects explicit `--enable-conc-sweep`. | +| Tensor parallel | `--tp` | Pass the prompt's TP. Default `1` outside native AgentX. With a native source YAML, the resolved physical `TP×PP×PCP` count is used and an explicit mismatch fails. | +| Expert parallel | `--ep` | Pass the prompt's EP for MoE. Default `1` outside native AgentX; a native source YAML resolves EP and rejects an explicit mismatch. | +| Precision | `--precision` | Match the checkpoint (`bf16` default / `fp8` / ...). A benchmark YAML may supply `benchmark.precision`. Keep consistent with `--quantize`. | +| Budget | `--max-hours` | Pass the prompt's time budget. Default `2.0`. Always set it explicitly for native AgentX: one canonical round commonly exceeds the default after model load, warmup, drain, and its 3600-second measurement. | | Latency SLA | `--max-latency-ms` | **Scriptable frameworks only** (`xdit`, `custom`); refused for serving, where AgentX already grades interactivity. Pass any stated ceiling on per-request latency ("must stay under 250 ms", "interactive workload"). A **constraint, not a target**: it composes with `--target-*` rather than competing, and refuses any KEEP whose mean end-to-end latency exceeds it — including one that reported no latency at all. Off when omitted, which does not lose a preference but does remove the SLA from the search. | | GPU power settings | `--gpu-power-cap-w` / `--gpu-perf-level` | Assertions, not requests. Only when the prompt says the cards were set to a cap or perf level; Hyperloom never changes them. The session refuses to start if a card differs. | -| Max model len | `--max-model-len` | Optional; auto-derived from ISL+OSL+headroom when omitted. | -| External reference GPU | `--compare-against-gpu` | `target_analysis` writes `target_analysis/target_baseline.json` for query/status metadata and `competitor_target.json` for both advisory and final-report comparisons. Without a target GPU it writes `reason="no_target_gpu_configured"` and clears the competitor target. AgentX reads accepted `current_best.total_throughput / state.tp` and `current_best.e2e_norm_intvty_p90` at `state.conc`; it does not reread raw results or recipes. Missing targets or axes remain unavailable. This is a cross-system advisory, not proof of identical measurement estimators or deployment, and never changes Objective or KEEP/REVERT. | +| Max model len | `--max-model-len` | Optional outside native AgentX; auto-derived from ISL+OSL+headroom when omitted. Native AgentX resolves the model context; custom workloads may cap it within that verified context. | +| External reference GPU | `--compare-against-gpu` | `target_analysis` writes `target_analysis/target_baseline.json` for query/status metadata and `competitor_target.json` for both advisory and final-report comparisons. Without a target GPU it writes `reason="no_target_gpu_configured"` and clears the competitor target. AgentX reads accepted `current_best.total_throughput`, divides by the recorded resolved recipe TP×PP×PCP GPU count (`state.tp` is only the legacy-artifact fallback), and reads `current_best.e2e_norm_intvty_p90` at `state.conc`; it does not reread raw results or recipes. Missing targets or axes remain unavailable. This is a cross-system advisory, not proof of identical measurement estimators or deployment, and never changes Objective or KEEP/REVERT. | | Target advisory | `--no-target-advisory` | Disable external-target hints in prompts without disabling final-report comparison. `primary_gap` uses the existing latency/throughput categories; the interactivity axis is displayed as interactivity. | | Quantization prelude | `--quantize` | Optional. Natural-language quantization request. Runs the quantization-agent once before the loop and rewrites `--model` to the quantized model. See Step 2b. Never runs on a resume. | | Env pins | `--extra-env NAME=VALUE` | Repeatable; forward **every** one verbatim as its own flag (do not drop any or fold into the `Environment:` block). The CLI persists them in `state.json` and serializes them into `$INFERENCE_OPTIMIZER_EXTRA_ENV`; a dropped pin is lost silently — e.g. a missing `SGLANG_USE_AITER=0` leaves the explore aiter-MoE filter blind. A `--resume-from` re-exports the persisted set, so re-pass them only to change the set. | @@ -925,8 +927,8 @@ them at runtime: - `benchmark.model` <- `--model` / `$MODEL_PATH` - `benchmark.runner_type` <- `--gpu-type` / `$GPU_TYPE` / rocm-smi auto-detect -`benchmark.benchmark_script` is deliberately NOT set in the shipped -YAMLs. At materialize time Hyperloom pins it to +Outside native AgentX, `benchmark.benchmark_script` is deliberately NOT set in +the shipped YAMLs. At materialize time Hyperloom pins it to `{framework}_{runner_type}.sh` (e.g. `sglang_mi300x.sh` / `sglang_mi355x.sh`) so Magpie's resolver hits priority 1 (explicit user override) and uses the generic script — which respects @@ -935,7 +937,8 @@ user override) and uses the generic script — which respects debug overrides; Orchestration can also route per-task via `params.benchmark_script` (sanitized). -Before a new model run, verify these fields match the environment: +Before a new non-AgentX model run, verify these generic-config fields match the +environment: - `benchmark.model`: model path. - `benchmark.envs.TP`: tensor parallel size. @@ -975,12 +978,158 @@ Operator server flags have one supported CLI entry point: `optimize --server-args ""`. The CLI exports this as `INFERENCE_OPTIMIZER_SERVER_ARGS`, and YAML materialization routes it into `EXTRA_VLLM_ARGS` / `EXTRA_SGLANG_ARGS` / `EXTRA_ATOM_ARGS` for baseline, -profile, explore, and sweep. Explicit `--max-model-len` / `$MAX_MODEL_LEN` -wins over auto `ISL+OSL+headroom`. A comma `$CONC` value such as +profile, explore, and sweep outside native AgentX. Native AgentX rejects this +flag until its pinned launchers expose an optimizer-argv hook. Outside native +AgentX, explicit `--max-model-len` / `$MAX_MODEL_LEN` wins over auto +`ISL+OSL+headroom`. A comma `$CONC` value such as `4,16,128` is accepted for compatibility; baseline uses the first value. -Use `--conc-sweep-concs` to override the ladder SWEEP measures (`256,128,64,32,16,8,4,2` synthetic, `1,4,8,10,14,20,28` under AgentX). Under AgentX the sweep is off unless `--enable-conc-sweep` is passed, since every rung is a 3600 s window. +Use `--conc-sweep-concs` to override the synthetic ladder (`256,128,64,32,16,8,4,2`) or the legacy AgentX/MLPerf ladder (`1,4,8,10,14,20,28`). AgentX defaults sweep off. Legacy AgentX and MLPerf may enable it with `--enable-conc-sweep`; native AgentX measures a fixed recipe point and rejects explicit sweep enablement. + +### Native Magpie AgentX contract + +Use Magpie-managed AgentX at commit +`d80eb4d3dad7fabe01ce81d049e2983adf2c86dd` and InferenceX commit +`408c015be4b22d14c69518643609669405507077`. Create a source Magpie YAML with +the public model identity, framework, launcher, effective image pin, and fixed +concurrency. Leave recipe internals in InferenceX: + +```yaml +benchmark: + framework: sglang + model: amd/GLM-5.2-MXFP4 + precision: fp4 + runner_type: mi355x + run_mode: local + agentx: enable + docker_image: lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260924 + gpu_selection: + auto: false + envs: + CONC: 8 + ROCR_VISIBLE_DEVICES: 0,1,2,3 +``` + +Launch with the YAML from inside the exact image it names: + +```bash +python3 -m hyperloom.inference_optimizer.cli optimize \ + --benchmark-config ./agentx-glm52.yaml \ + --max-hours 3 +``` + +Three hours is the explicit one-baseline budget for this quickstart; increase it +when more rounds are intended. Do not silently fall back to the normal two-hour +default, which is commonly shorter than one complete canonical AgentX round. + +The source `benchmark.agentx: enable` switch automatically stamps Hyperloom +session state and selects AgentX grading before preflight. Do not require or +export `HYPERLOOM_AGENTX` on this path. A fresh `HYPERLOOM_AGENTX=1` launch +also uses Magpie native AgentX, with the normal model/framework/GPU/precision/ +concurrency inputs resolved to an unambiguous upstream recipe and launcher. +New sessions use epoch 4 and preserve Hyperloom optimization. Persisted epoch-1 +sessions keep the legacy client; epoch-2 native sessions keep their original +measurement-only contract and pins; epoch-3 sessions retain their upstream +launcher contract. Never migrate a saved epoch, baseline, or +KEEP record implicitly. +`--benchmark-config` is fresh-launch only. A +resume rejects a new source-config flag. It restores the accepted materialized +config and runtime pins after baseline acceptance, or the session's snapshotted +source config and pins if baseline was not accepted yet. + +For a local checkpoint, source `benchmark.model` stays the canonical recipe id +and CLI `--model` supplies the mounted path written to `MODEL_PATH`. For a +remote Hugging Face checkpoint, omit CLI `--model`; Hyperloom uses the source +model id and removes `MODEL_PATH` from the native InferenceX subprocess. +Prefix a relative local path with `./`. +Do not copy `MODEL_PREFIX`, TP/EP, KV-offload, backend, or CPU-DRAM fields from +`amd-master.yaml`; native Magpie resolves them from the recipe. Hyperloom +pre-resolves the recipe through the same benchmark interpreter that runs it. +The fixed concurrency comes from `--conc` or `benchmark.envs.CONC` and must +occur in that recipe. `envs.TP` is not an arm selector. If one concurrency has +multiple arms, prefer the YAML-native object form: + +```yaml +agentx: + enabled: true + selector: + tp: 4 + kv_offloading: dram + kv_offload_backend: hicache +``` -Operator server flags are the workload baseline, but they are not sacred. When +Add `agentx.recipe` only when multiple recipe names match. `AGENTX_RECIPE` and +JSON `AGENTX_SELECTOR` remain legacy environment equivalents. Hyperloom removes +input `agentx.concurrency` and writes the fixed measurement value to +`benchmark.envs.CONC`. + +Native AgentX concurrency sweep defaults off. Explicit +`--enable-conc-sweep` fails preflight because the pinned launcher provides no +distinct optimized arm; `--conc-sweep-concs` does not enable it. The recipe +configures a 393-trace dataset-entry cap, and native `AGENTX_DATASET` and +`WEKA_LOADER_OVERRIDE` overrides are rejected. That value is a loader ceiling, +not a guarantee that 393 traces, sessions, or requests survive availability and +context-length filters. Canonical mode runs for 3600 seconds. +`AGENTX_MODE=fast` runs for 1200 seconds and is only for a direct +Magpie diagnostic; a full `optimize` rejects its non-publishable result. + +Omit `--tp` and `--ep` to use the recipe. Hyperloom sets outer `--tp` to the +resolved physical `TP×PP×PCP` count and `--ep` to resolved EP; explicit values +are consistency assertions and mismatches fail. Topology-changing AgentX sweeps +are rejected. The pinned launchers copy ROCR's physical values back into logical +HIP indices, so Hyperloom derives `gpu_selection.auto=false` and the zero-based +`ROCR_VISIBLE_DEVICES=0,...,N-1` mask. Explicit source values are assertions; a +nonzero or reordered mask is rejected. + +Hyperloom executes Magpie locally inside the already selected serving +container, so it does not start nested Docker. The YAML's `docker_image` +overrides and pins the effective recipe image, and Magpie includes it in the +recipe fingerprint. A pre-existing `HYPERLOOM_IMAGE` is an optional strict +consistency assertion. Hyperloom does not start or inspect the container and +therefore cannot prove the actual outer image. + +The source may omit `benchmark.inferencex_path`, as above. Preflight then reuses +a writable checkout at the tested commit or clones one into the dependency +cache. A source path only nominates a preferred checkout: a missing or +wrong-revision path falls back to the pinned clone, while an explicit correct +but non-writable checkout fails. Managed AgentX accepts the repository root +or its `inferencex-e2e/` project directory, including spaces. Saved shell-launcher +sessions retain their original path restrictions. + +The Hyperloom bridge supports local, single-node SGLang and vLLM only. Do not +use it with multi-node/disaggregated AgentX, `server_lifecycle`, or Atom. Native +AgentX bypasses Hyperloom's outer Ray actor automatically; leave +`INFERENCE_OPTIMIZER_RAY_EXEC` unset or set it to `0`, because an explicit true +value is rejected. + +Normal native fixed-concurrency measurements require `benchmark_valid=true` +and `publishable=true`. For epoch-4 profile/roofline actions, derive the +configuration from the accepted native candidate and let Magpie own the server +and torch profiler. Forward `num_steps`, `num_profiles`, `start_seconds`, +`interval_seconds`, capture/flush timeouts, and `detailed_annotations` as +structured torch profiler settings; never add a second set of framework +profiler flags. Detailed annotations require an instrumented framework; +unsupported runtimes fail explicitly instead of modifying accepted source. +Preserve the existing shape and trace-quality checks before kernel analysis. + +Use the Magpie capture manifest to select one complete time window, retaining +its rank traces and all other capture artifacts. Do not mix different +`profile_index` values. Automatic count reduction can be successful; a failed +runtime, cancellation, or incomplete capture cannot. Diagnostic results have +`benchmark_valid=false` and `publishable=false` by design and never replace +baseline or KEEP evidence. Saved epoch-3 sessions keep their old upstream +launcher and compatibility-profiler path; saved epoch-2 sessions remain +measurement-only. No saved session is automatically upgraded. + +Magpie-managed launch evidence binds candidate arguments, environment, source +files, and resolved server specification. The client revision and recipe +sources remain pinned. GEAK proxy scores cannot become KEEP results: candidates +require Critic review where applicable and canonical native remeasurement. +Hyperloom does not cryptographically prove the actual outer image. Credentials, +cache routing, output paths, and unrelated shell variables remain outside the +execution identity. + +Outside native AgentX, operator server flags are the workload baseline, but +they are not sacred. When the configuration arm has evidence or an operator hint that a pinned flag may be harmful, it may test an ablation variant with `remove_args` (or `unset_envs` for inherited @@ -993,7 +1142,8 @@ audit. ### Workload-contract reuse (baseline → explore/sweep) -`baseline` materializes its YAML once with the operator's process env +Outside native AgentX, `baseline` materializes its YAML once with the +operator's process env (`CONC` / `ISL` / `OSL` / `TP` / `MAX_MODEL_LEN` / `PRECISION` / `RUN_EVAL` / `ROCR_VISIBLE_DEVICES` + adaptive `NUM_PROMPTS` / `NUM_WARMUPS`), saves it as `baseline_config.with_envs.yaml`, and forwards the path as @@ -1166,7 +1316,8 @@ it can make `torch.cuda.is_available()` return false. Use Serving-parameter search runs through the `explore` action (the legacy `params` / `backends` actions were merged into it); candidates are written via `EXTRA_SGLANG_ARGS` / `benchmark.envs`. This is internal to -the optimizer — the launcher does not drive it. Useful InferenceX-derived +the optimizer — the launcher does not drive it. Native AgentX is excluded until +its pinned launchers expose an optimizer hook. Useful InferenceX-derived candidate families a specialist may surface: `--disable-radix-cache`, `--max-running-requests`, `--tokenizer-worker-num`, `--stream-interval`, and ROCm/TileLang envs (`SGLANG_OPT_USE_MULTI_STREAM_OVERLAP`, diff --git a/src/hyperloom/inference_optimizer/agentx/geak_proxy.py b/src/hyperloom/inference_optimizer/agentx/geak_proxy.py new file mode 100644 index 0000000000..d1ef275358 --- /dev/null +++ b/src/hyperloom/inference_optimizer/agentx/geak_proxy.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Seed GEAK's proposal workload from an observed native server configuration.""" + +from __future__ import annotations + +import copy +import shlex +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +from .identity import canonical_sha256 + +_PROXY_OWNED_OPTIONS = frozenset( + { + "--model", + "--model-path", + "--served-model-name", + "--host", + "--port", + "--tp", + "--tp-size", + "--tensor-parallel-size", + "--pp", + "--pp-size", + "--pipeline-parallel-size", + "--ep", + "--ep-size", + "--expert-parallel-size", + "--dp", + "--dp-size", + "--data-parallel-size", + "--data-parallel-rank", + "--data-parallel-start-rank", + "--data-parallel-size-local", + "--nnodes", + "--node-rank", + "--dist-init-addr", + "--enable-dp-attention", + "--enable-expert-parallel", + "--decode-context-parallel-size", + "--prefill-context-parallel-size", + } +) + + +def _tunable_flags(argv: list[str], framework: str) -> str: + if framework == "sglang" and argv[1:3] == ["-m", "sglang.launch_server"]: + args = argv[3:] + elif framework == "vllm" and argv[1:2] == ["serve"] and len(argv) >= 3: + args = argv[3:] + else: + raise ValueError("Native AgentX launch evidence has an unsupported server entry point") + retained: list[str] = [] + keep = False + for token in args: + if token.startswith("--"): + keep = token.split("=", 1)[0] not in _PROXY_OWNED_OPTIONS + if keep: + retained.append(token) + return shlex.join(retained) + + +def seed_native_geak_proxy( + handoff: dict[str, Any], *, benchmark: Mapping[str, Any], measurement: Mapping[str, Any] +) -> None: + """Pass observed tunables through GEAK's existing effective-config contract. + + GEAK owns its proxy endpoint, client, topology and profiling lifecycle. Its + measurements remain incomparable even when all server tunables match. + Only Hyperloom's subsequent canonical AgentX run can accept the product. + """ + evidence = measurement.get("agentx_server_launch") + if not isinstance(evidence, dict): + workload = benchmark.get("workload_spec") + evidence = workload.get("server_launch") if isinstance(workload, Mapping) else None + if not isinstance(evidence, dict): + raise ValueError("Native AgentX GEAK proposals require a verified server launch") + unsigned = {key: value for key, value in evidence.items() if key != "evidence_sha256"} + if evidence.get("evidence_sha256") != canonical_sha256(unsigned): + raise ValueError("Native AgentX GEAK server launch evidence changed") + argv = evidence.get("effective_argv") + if not isinstance(argv, list) or not all(isinstance(arg, str) for arg in argv): + raise ValueError("Native AgentX GEAK server launch argv is missing") + flags = _tunable_flags(argv, str(handoff.get("framework") or "")) + runtime = dict(evidence.get("runtime_environment") or {}) + if argv and Path(argv[0]).is_absolute(): + executable_dir = str(Path(argv[0]).parent) + runtime["PATH"] = ":".join(part for part in (executable_dir, runtime.get("PATH", "")) if part) + for name in ("HIP_VISIBLE_DEVICES", "ROCR_VISIBLE_DEVICES", "CUDA_VISIBLE_DEVICES"): + runtime.pop(name, None) + spec = copy.deepcopy(handoff.get("baseline_env_spec") or {}) + config = spec.setdefault("config", {}) + config.update({"server_launch_flags": flags, "extra_server_args": "", "extra_envs": runtime}) + handoff["baseline_env_spec"] = spec + handoff["accepted_flags"] = "" + handoff["accepted_env"] = shlex.join(f"{name}={value}" for name, value in runtime.items()) + handoff["native_server_launch"] = copy.deepcopy(evidence) + handoff["raw_baseline_tput"] = 0.0 + handoff["orchestrator_best_tput_same_config"] = 0.0 + handoff["same_config_reference_status"] = "unverified" + handoff["same_config_reference_verification_status"] = "unverified_workload" diff --git a/src/hyperloom/inference_optimizer/agentx/identity.py b/src/hyperloom/inference_optimizer/agentx/identity.py new file mode 100644 index 0000000000..3a7a0110cd --- /dev/null +++ b/src/hyperloom/inference_optimizer/agentx/identity.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Separate a native replay workload from the implementation being optimized.""" + +from __future__ import annotations + +import hashlib +import json +import re +import subprocess +from collections.abc import Mapping +from pathlib import Path, PurePosixPath +from typing import Any + +# Frozen protocol options for persisted InferenceX version-1 launch contracts. +# Managed sessions instead use their audited Magpie installation to validate +# argv semantics, including profiler flags and future protected options. +_PROTOCOL_ARGS = frozenset( + { + "--model", + "--model-path", + "--max-model-len", + "--context-length", + "--served-model-name", + "--host", + "--port", + "--tp", + "--tp-size", + "--tensor-parallel-size", + "--pp", + "--pp-size", + "--pipeline-parallel-size", + "--ep", + "--ep-size", + "--expert-parallel-size", + "--dp", + "--dp-size", + "--data-parallel-size", + "--data-parallel-rank", + "--data-parallel-start-rank", + "--data-parallel-size-local", + "--data-parallel-address", + "--data-parallel-rpc-port", + "--enable-dp-attention", + "--enable-expert-parallel", + "--decode-context-parallel-size", + "--prefill-context-parallel-size", + "--dcp-size", + "--pcp-size", + "--nnodes", + "--node-rank", + "--dist-init-addr", + } +) + + +def _argument_groups(argv: list[str]) -> tuple[list[str], list[list[str]]]: + prefix: list[str] = [] + groups: list[list[str]] = [] + for token in argv: + if token.startswith("--"): + if not re.fullmatch(r"--[A-Za-z0-9][A-Za-z0-9_-]*", token.partition("=")[0]): + raise ValueError("invalid long option") + groups.append([token]) + elif groups: + if "=" in groups[-1][0] or ( + token.startswith("-") and not re.fullmatch(r"-\d+(?:\.\d+)?(?:[eE][+-]?\d+)?", token) + ): + raise ValueError("ambiguous option operand") + groups[-1].append(token) + else: + prefix.append(token) + return prefix, groups + + +def _expected_launch_argv(base: list[str], overrides: Mapping[str, Any]) -> list[str]: + effective = list(base) + appended = overrides.get("append_args", []) + removed = overrides.get("remove_args", []) + replace = overrides.get("replace_args", False) + if not all( + isinstance(values, list) and all(isinstance(value, str) for value in values) for values in (appended, removed) + ): + raise ValueError("invalid argument override") + if type(replace) is not bool or len(set(removed)) != len(removed): + raise ValueError("invalid argument controls") + if appended or removed or replace: + prefix, groups = _argument_groups(base) + extra_prefix, extra = _argument_groups(appended) + if extra_prefix or any(group[0].partition("=")[0] in _PROTOCOL_ARGS for group in extra): + raise ValueError("protocol argument override") + for name in removed: + matches = [group for group in groups if group[0].partition("=")[0] == name] + if name in _PROTOCOL_ARGS or len(matches) != 1: + raise ValueError("ambiguous or protected argument removal") + groups.remove(matches[0]) + if replace: + groups = [group for group in groups if group[0].partition("=")[0] in _PROTOCOL_ARGS] + effective = prefix + [token for group in groups + extra for token in group] + if overrides.get("executable") is not None: + effective[0] = overrides["executable"] + return effective + + +def canonical_sha256(value: Any) -> str: + """Hash a JSON value using the upstream launch evidence representation.""" + encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False) + return hashlib.sha256(encoded.encode("utf-8")).hexdigest() + + +def has_launch_contract(benchmark: Mapping[str, Any]) -> bool: + """Identify the persisted optimization contract; old native sessions lack it.""" + agentx = benchmark.get("agentx") + overrides = agentx.get("launch_overrides") if isinstance(agentx, Mapping) else None + return isinstance(overrides, Mapping) and type(overrides.get("version")) is int and overrides["version"] == 1 + + +def native_workload_fingerprint(benchmark: Mapping[str, Any], static_fingerprint: str) -> str: + """Bind the replay and model while allowing explicit server implementation changes. + + Arbitrary runtime environment changes belong in launch_overrides, whose + upstream validator rejects replay/model/topology controls. The remaining + benchmark environment stays part of the fixed session workload. + """ + payload = dict(benchmark) + payload.pop("workload_spec", None) + agentx = dict(payload.get("agentx") or {}) + agentx.pop("launch_overrides", None) + payload["agentx"] = agentx + return canonical_sha256({"static_execution_fingerprint": static_fingerprint, "benchmark": payload}) + + +def _validate_managed_receipt( + benchmark: Mapping[str, Any], evidence: Mapping[str, Any], workspace: Path | None +) -> list[str]: + """Delegate managed argv and diagnostic instrumentation to audited Magpie.""" + profile = ((benchmark.get("profiler") or {}).get("torch_profiler") or {}).get("enabled") is True + if profile and workspace is None: + return ["server_launch_profile_workspace_missing"] + failure = "server_launch_profile_verification_failed" if profile else "server_launch_managed_verification_failed" + from hyperloom.common.env_safety import scrub_benchmark_process_env + from hyperloom.orchestrator.actions.executors.benchmark_backend import resolve_benchmark_interpreter + + from .native import _MAGPIE_SOURCE_IDENTITY_CODE + + code = ( + _MAGPIE_SOURCE_IDENTITY_CODE + + """ +import sys +import Magpie +from Magpie.modes.benchmark import BenchmarkConfig +from Magpie.modes.benchmark.agentx_launch import apply_launch_args, read_launch_evidence, _validate_golden_launch +package = Path(Magpie.__file__).resolve().parent +commit, _ = _resolve_magpie_source_identity(package) +_validate_magpie_execution_tree(package, commit) +payload = json.load(sys.stdin) +benchmark, evidence = payload["benchmark"], payload["evidence"] +if payload["profile"]: + config = BenchmarkConfig.from_dict(benchmark) + actual = read_launch_evidence(config, Path(payload["workspace"])) + if actual != evidence: + raise ValueError("profile receipt does not match the report") +else: + expected = apply_launch_args(evidence["base_argv"], benchmark["agentx"]["launch_overrides"], benchmark["framework"]) + if evidence["effective_argv"] != expected: + raise ValueError("managed effective argv does not match the candidate") + _validate_golden_launch(benchmark["agentx"]["resolved"]["server-launch-spec"], expected) +""" + ) + import os + + try: + checked = subprocess.run( + [resolve_benchmark_interpreter(), "-c", code], + input=json.dumps( + { + "benchmark": dict(benchmark), + "evidence": dict(evidence), + "workspace": str(workspace), + "profile": profile, + } + ), + capture_output=True, + text=True, + timeout=120, + env=scrub_benchmark_process_env(dict(os.environ)), + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return [failure] + return [] if checked.returncode == 0 else [failure] + + +def validate_server_launch(benchmark: Mapping[str, Any], evidence: Any, *, workspace: Path | None = None) -> list[str]: + """Validate the observed upstream server launch against the submitted candidate.""" + if not has_launch_contract(benchmark): + return [] + if not isinstance(evidence, dict): + return ["server_launch_missing"] + errors: list[str] = [] + if ( + type(evidence.get("version")) is not int + or evidence.get("version") != 1 + or evidence.get("framework") != benchmark.get("framework") + ): + errors.append("server_launch_identity_mismatch") + unsigned = {key: value for key, value in evidence.items() if key != "evidence_sha256"} + if evidence.get("evidence_sha256") != canonical_sha256(unsigned): + errors.append("server_launch_evidence_hash_mismatch") + overrides = benchmark["agentx"]["launch_overrides"] + resolved = benchmark["agentx"].get("resolved", {}) + spec = resolved.get("server-launch-spec") if isinstance(resolved, Mapping) else None + if evidence.get("overrides_sha256") != canonical_sha256(overrides): + errors.append("server_launch_candidate_mismatch") + for key in ("base_argv", "effective_argv"): + args = evidence.get(key) + if not isinstance(args, list) or not args or any(not isinstance(arg, str) or "\0" in arg for arg in args): + errors.append(f"server_launch_{key}_invalid") + if not isinstance(spec, Mapping) and not any(code.endswith("_argv_invalid") for code in errors): + try: + expected_argv = _expected_launch_argv(evidence["base_argv"], overrides) + except ValueError: + errors.append("server_launch_arguments_invalid") + else: + if evidence["effective_argv"] != expected_argv: + errors.append("server_launch_arguments_mismatch") + executable = overrides.get("executable") + if executable and ( + not isinstance(evidence.get("effective_argv"), list) + or not evidence["effective_argv"] + or evidence["effective_argv"][0] != executable + ): + errors.append("server_launch_executable_mismatch") + observed_env = evidence.get("effective_env") + expected_env = {**overrides.get("env", {}), **dict.fromkeys(overrides.get("unset_env", []))} + if not isinstance(observed_env, dict): + errors.append("server_launch_environment_missing") + elif observed_env != expected_env: + errors.append("server_launch_environment_mismatch") + base_env = evidence.get("base_env") + if ( + not isinstance(base_env, dict) + or base_env.keys() != expected_env.keys() + or any(value is not None and not isinstance(value, str) for value in base_env.values()) + ): + errors.append("server_launch_base_environment_invalid") + resolved_executable = evidence.get("resolved_executable") + if not isinstance(resolved_executable, str) or not PurePosixPath(resolved_executable).is_absolute(): + errors.append("server_launch_resolved_executable_invalid") + runtime_environment = evidence.get("runtime_environment") + if not isinstance(runtime_environment, dict) or any( + not isinstance(name, str) or not isinstance(value, str) for name, value in runtime_environment.items() + ): + errors.append("server_launch_runtime_environment_invalid") + if evidence.get("source_files", {}) != overrides.get("source_files", {}): + errors.append("server_launch_source_mismatch") + if evidence.get("absent_source_files", []) != overrides.get("absent_source_files", []): + errors.append("server_launch_absent_source_mismatch") + if isinstance(spec, Mapping): + if evidence.get("owner") != "magpie": + errors.append("server_launch_owner_mismatch") + if evidence.get("recipe_source_files") != spec.get("source_files"): + errors.append("server_launch_recipe_sources_mismatch") + profiler = benchmark.get("profiler") or {} + torch = profiler.get("torch_profiler") or {} + if not any(code.endswith("_argv_invalid") for code in errors): + errors.extend(_validate_managed_receipt(benchmark, evidence, workspace)) + if torch.get("enabled") is not True: + if evidence.get("server_spec_sha256") != canonical_sha256(spec): + errors.append("server_launch_spec_mismatch") + if evidence.get("base_argv") != spec.get("argv"): + errors.append("server_launch_recipe_argv_mismatch") + if evidence.get("torch_profiler") is not None: + errors.append("server_launch_unexpected_profile") + return errors + + +def verified_workload_fingerprint(benchmark: Mapping[str, Any]) -> str | None: + """Recompute the saved workload identity when interpreting benchmark artifacts.""" + workload = benchmark.get("workload_spec") + execution = workload.get("execution") if isinstance(workload, Mapping) else None + if not isinstance(execution, Mapping): + return None + static = execution.get("static_execution_fingerprint") + if not isinstance(static, str) or not re.fullmatch(r"[a-f0-9]{64}", static): + return None + fingerprint = native_workload_fingerprint(benchmark, static) + return fingerprint if fingerprint == execution.get("workload_fingerprint") else None diff --git a/src/hyperloom/inference_optimizer/agentx/managed.py b/src/hyperloom/inference_optimizer/agentx/managed.py new file mode 100644 index 0000000000..6832f82663 --- /dev/null +++ b/src/hyperloom/inference_optimizer/agentx/managed.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Bind Magpie-managed serving to its immutable InferenceX client sources.""" + +from __future__ import annotations + +import hashlib +import re +import subprocess +from collections.abc import Mapping +from copy import deepcopy +from pathlib import Path +from typing import Any + +import yaml + +from .identity import canonical_sha256, native_workload_fingerprint, verified_workload_fingerprint + + +def project_root(path: str | Path) -> Path: + """Accept a repository root or its packaged InferenceX project.""" + root = Path(path).expanduser().resolve() + roots = [ + candidate + for candidate in (root, root / "inferencex-e2e") + if (candidate / "benchmarks/benchmark_lib.sh").is_file() + ] + if len(roots) > 1: + raise ValueError(f"Ambiguous InferenceX project roots below {root}") + return roots[0] if roots else root + + +def server_spec(benchmark: Mapping[str, Any]) -> dict[str, Any] | None: + agentx = benchmark.get("agentx") + resolved = agentx.get("resolved") if isinstance(agentx, Mapping) else None + spec = resolved.get("server-launch-spec") if isinstance(resolved, Mapping) else None + return spec if isinstance(spec, dict) else None + + +def identity_benchmark(benchmark: Mapping[str, Any], expected_workload: str) -> dict[str, Any]: + """Bind diagnostic settings to an accepted workload while retaining candidate overrides.""" + result = deepcopy(dict(benchmark)) + workload = result.get("workload_spec") or {} + parent = workload.get("profile_parent") + torch = (result.get("profiler") or {}).get("torch_profiler") or {} + if not parent: + if torch.get("enabled") is True: + raise ValueError("Managed AgentX profiling requires an accepted profile_parent") + return result + if not isinstance(parent, Mapping) or torch.get("enabled") is not True: + raise ValueError("Managed AgentX profile_parent requires diagnostic torch profiling") + path = Path(str(parent.get("materialized_config") or "")) + if not path.is_absolute(): + raise ValueError("Managed AgentX profile parent must be an absolute configuration path") + content = path.read_bytes() + if hashlib.sha256(content).hexdigest() != parent.get("config_sha256"): + raise ValueError("Managed AgentX profile parent configuration changed") + loaded = yaml.safe_load(content) + accepted = loaded.get("benchmark", loaded) if isinstance(loaded, dict) else None + if not isinstance(accepted, dict) or server_spec(accepted) is None: + raise ValueError("Managed AgentX profile parent is not an accepted managed configuration") + if ((accepted.get("profiler") or {}).get("torch_profiler") or {}).get("enabled") is True: + raise ValueError("Managed AgentX profile parent must be an unprofiled benchmark") + fingerprint = verified_workload_fingerprint(accepted) + if ( + not fingerprint + or not expected_workload + or fingerprint != expected_workload + or fingerprint != parent.get("workload_fingerprint") + ): + raise ValueError("Managed AgentX profile parent does not match the session workload pin") + fixed = [] + for value in (result, accepted): + config = deepcopy({key: item for key, item in value.items() if key not in {"workload_spec", "profiler"}}) + config["agentx"].pop("launch_overrides", None) + fixed.append(config) + if fixed[0] != fixed[1]: + raise ValueError("Managed AgentX diagnostic configuration changed the accepted workload") + if "profiler" in accepted: + result["profiler"] = deepcopy(accepted["profiler"]) + else: + result.pop("profiler", None) + return result + + +def git_text(root: Path, *args: str) -> str: + result = subprocess.run(["git", "-C", str(root), *args], capture_output=True, text=True, timeout=30, check=False) + if result.returncode: + raise ValueError(f"Cannot verify InferenceX source at {root}: {result.stderr.strip()}") + return result.stdout.strip() + + +def head_blob(root: Path, relative: str) -> bytes: + prefix = git_text(root, "rev-parse", "--show-prefix") + result = subprocess.run( + ["git", "-C", str(root), "show", f"HEAD:{prefix}{relative}"], capture_output=True, timeout=30, check=False + ) + if result.returncode: + raise ValueError(f"native AgentX execution input is not tracked at HEAD: {relative}") + return result.stdout + + +def aiperf_revision(root: Path) -> str: + """Verify the harness gitlink inside either supported project layout.""" + fields = git_text(root, "ls-tree", "HEAD", "utils/aiperf").split() + if len(fields) != 4 or fields[0] != "160000" or not re.fullmatch(r"[0-9a-f]{40}", fields[2]): + raise ValueError("InferenceX utils/aiperf is not a pinned git submodule") + dependency = root / "utils/aiperf" + if not (dependency / "pyproject.toml").is_file(): + raise ValueError("InferenceX utils/aiperf submodule is not initialized") + if Path(git_text(dependency, "rev-parse", "--show-toplevel")).resolve() != dependency.resolve(): + raise ValueError("InferenceX utils/aiperf is not its own checkout") + if git_text(dependency, "rev-parse", "HEAD") != fields[2]: + raise ValueError("InferenceX utils/aiperf does not match its pinned gitlink") + if git_text(dependency, "status", "--porcelain=v1", "--untracked-files=all"): + raise ValueError("InferenceX utils/aiperf submodule is not clean") + return fields[2] + + +def _source_identity(root: Path, spec: Mapping[str, Any], config_file: str) -> dict[str, str]: + sources = spec.get("source_files") + if not isinstance(sources, Mapping) or not sources: + raise ValueError("Magpie server launch specification has no recipe source files") + relative_sources: dict[str, str] = {} + for filename, expected in sources.items(): + path = Path(filename) + try: + relative = path.relative_to(root) + except ValueError as exc: + raise ValueError(f"AgentX recipe source escapes InferenceX: {path}") from exc + if not path.is_absolute() or any((root / parent).is_symlink() for parent in (relative, *relative.parents)): + raise ValueError(f"AgentX recipe source must be an absolute path without symlinks: {path}") + actual = hashlib.sha256(path.read_bytes()).hexdigest() + if actual != expected or path.read_bytes() != head_blob(root, relative.as_posix()): + raise ValueError(f"AgentX recipe source differs from pinned HEAD: {relative}") + relative_sources[relative.as_posix()] = actual + required = {"benchmarks/srt_agentic.sh", "benchmarks/benchmark_lib.sh"} + if config_file != "magpie:custom": + required.update((config_file, "configs/runners.yaml")) + if not required <= relative_sources.keys(): + raise ValueError("Magpie server launch source identity omits required client/recipe files") + if git_text(root, "status", "--porcelain=v1", "--untracked-files=all", "--", "infx", *relative_sources): + raise ValueError("native AgentX requires clean pinned client/recipe inputs") + return relative_sources + + +def execution_identity( + *, + root: Path, + benchmark: Mapping[str, Any], + config_file: str, + expected_ref: str, + magpie: Mapping[str, Any], + expected_magpie_ref: str, + launch_environment: Mapping[str, Any], +) -> dict[str, Any]: + """Persist the managed contract without consulting obsolete shell manifests.""" + spec = server_spec(benchmark) + if spec is None or spec.get("version") != 1 or benchmark.get("benchmark_script") != "srt_agentic.sh": + raise ValueError("Managed AgentX requires server-launch-spec v1 and srt_agentic.sh") + head = git_text(root, "rev-parse", "HEAD") + if not re.fullmatch(r"[0-9a-f]{40}", head) or spec.get("client_revision") != head: + raise ValueError("Magpie server launch client revision does not match InferenceX HEAD") + if expected_ref and expected_ref != head: + raise ValueError(f"InferenceX HEAD {head} does not match the pinned ref {expected_ref}") + magpie_commit = str(magpie.get("source_commit") or "") + if not re.fullmatch(r"[0-9a-f]{40}", magpie_commit) or not re.fullmatch( + r"[0-9a-f]{64}", str(magpie.get("fingerprint") or "") + ): + raise ValueError("Magpie resolver did not provide a valid execution identity") + if expected_magpie_ref and magpie_commit != expected_magpie_ref: + raise ValueError("Magpie source commit does not match the pinned ref") + sources = _source_identity(root, spec, config_file) + aiperf = aiperf_revision(root) + if spec.get("aiperf_revision") != aiperf: + raise ValueError("Magpie AIPerf revision does not match the client gitlink") + identity = { + "execution_owner": "magpie", + "inferencex_commit": head, + "launcher": "benchmarks/srt_agentic.sh", + "config_file": config_file, + "launcher_sha256": sources["benchmarks/srt_agentic.sh"], + "aiperf_commit": aiperf, + "magpie_commit": magpie_commit, + "magpie_fingerprint": magpie["fingerprint"], + "server_spec_sha256": canonical_sha256(spec), + "transitive_inputs": sources, + } + identity["static_execution_fingerprint"] = canonical_sha256(identity) + identity["workload_fingerprint"] = native_workload_fingerprint(benchmark, identity["static_execution_fingerprint"]) + identity["launch_config_sha256"] = canonical_sha256( + {key: value for key, value in benchmark.items() if key != "workload_spec"} + ) + identity["launch_environment"] = dict(launch_environment) + identity["execution_fingerprint"] = canonical_sha256(identity) + return identity diff --git a/src/hyperloom/inference_optimizer/agentx/mapping.py b/src/hyperloom/inference_optimizer/agentx/mapping.py index 68baec1567..3d0ae33fe7 100644 --- a/src/hyperloom/inference_optimizer/agentx/mapping.py +++ b/src/hyperloom/inference_optimizer/agentx/mapping.py @@ -176,12 +176,18 @@ def _distribution(metric: Any) -> dict[str, int]: def map_corpus_shape(result: Mapping[str, Any]) -> dict[str, Any]: """Build a ``SharedState.agentx_corpus_shape`` record from a :func:`map_aiperf` result.""" + completed = result.get("completed") + if completed is None: + completed = result.get("completed_requests") + duration = result.get("duration") + if duration is None: + duration = result.get("duration_seconds") return { "corpus_loader": str(result.get("corpus_loader") or ""), "isl": dict(result.get("isl_distribution") or {}), "osl": dict(result.get("osl_distribution") or {}), - "completed_requests": int(result.get("completed") or 0), - "duration_s": float(result.get("duration") or 0.0), + "completed_requests": int(completed or 0), + "duration_s": float(duration or 0.0), "prefix_cache_hit": float(result.get("theoretical_prefix_cache_hit") or 0.0), "request_error_rate": float(result.get("request_error_rate") or 0.0), "source": "measured", diff --git a/src/hyperloom/inference_optimizer/agentx/native.py b/src/hyperloom/inference_optimizer/agentx/native.py new file mode 100644 index 0000000000..db7855961c --- /dev/null +++ b/src/hyperloom/inference_optimizer/agentx/native.py @@ -0,0 +1,1414 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Validation helpers for Magpie's native InferenceX AgentX path.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import subprocess +from pathlib import Path, PurePosixPath +from typing import Any, Mapping, MutableMapping + +from hyperloom.common.agentx_mode import native_agentx_enabled + +_AGENTIC_PREFIX = ("single_node", "agentic") +_RESOLVER_SENTINEL = "HYPERLOOM_AGENTX_RECIPE=" +_NATIVE_CHECKOUT_PATH_RE = re.compile(r"^[A-Za-z0-9_@%+=:,./-]+$") + + +# Ambient values consumed by the launchers in the exact-ref manifest below, +# plus loader/search and ROCm/framework controls inherited by their children. +# Keep this allowlist explicit: hashing every login-shell variable would make a +# resume depend on unrelated values such as PWD/SHLVL, while omitting a real +# launcher knob would let the workload change without changing its identity. +_NATIVE_AMBIENT_CONTROL_NAMES = frozenset( + { + "AGENTIC_VENV", + "BASH_ENV", + "CHUNKED_PREFILL_SIZE", + "CONC", + "CPU_BYTES_PER_RANK", + "CPU_OFFLOAD_BYTES", + "CUDAGRAPH_CAPTURE_SIZES", + "CUDA_GRAPH_CAPTURE_SIZES", + "CUDA_GRAPH_MAX_BS", + "DCP_SIZE", + "DP_ATTENTION", + "DRAFT_MODEL", + "DRAFT_MODEL_PATH", + "DURATION", + "EP_SIZE", + "ENV", + "EVAL_ONLY", + "FRAMEWORK", + "GPU_MEMORY_UTILIZATION", + "GPU_MEM_UTIL", + "HF_MODEL_ID", + "IMAGE", + "IS_AGENTIC", + "IS_MULTINODE", + "ISL", + "GCONV_PATH", + "KV_OFFLOADING", + "KV_OFFLOAD_BACKEND", + "L3_PER_RANK_GB", + "LD_LIBRARY_PATH", + "LD_AUDIT", + "LD_PRELOAD", + "LIBRARY_PATH", + "MAX_CUDAGRAPH_CAPTURE_SIZE", + "MAX_NUM_SEQS", + "MAX_RUNNING_REQUESTS", + "MAX_MODEL_LEN", + "MEM_FRACTION_STATIC", + "MODEL_DOWNLOAD_LOCK_TIMEOUT", + "MODEL", + "MODEL_NAME", + "MODEL_PATH", + "MODEL_PREFIX", + "NUM_SPEC_TOKENS", + "PATH", + "PCP_SIZE", + "PORT", + "PP_SIZE", + "PRECISION", + "PROFILE", + "PYTHONHASHSEED", + "PYTHONHOME", + "PYTHONINSPECT", + "PYTHONPATH", + "PYTHONSTARTUP", + "PYTHONUSERBASE", + "SCHEDULER_RECV_INTERVAL", + "SCENARIO_TYPE", + "SERVED_MODEL_NAME", + "SHELLOPTS", + "SHM_CAP_GB", + "SIMPLE_LAZY_OFFLOAD", + "SPEC_DECODING", + "SPEC_NUM_TOKENS", + "SYNTHETIC_ACCEPT_LEN", + "TOTAL_CPU_DRAM_GB", + "TOTAL_CPU_DRAM_PARTITION_GB", + "TP", + "RECIPE_FINGERPRINT", + "RUNNER_TYPE", + "VLLM_ENGINE_READY_TIMEOUT_S", + "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS", + "VIRTUAL_ENV", + "WITH_LM_EVAL", + "WITH_PR4521", + "WRITABLE_MODELS_DIR", + "WS", + "OSL", + } +) +_NATIVE_AMBIENT_CONTROL_PREFIXES = ( + "AITER_", + "CUDA_", + "DECODE_", + "EVAL_", + "HICACHE_", + "HIP_", + "HSA_", + "INFERENCEX_", + "KV_", + "LD_", + "LMCACHE_", + "MC_", + "MIOPEN_", + "MOONCAKE_", + "NCCL_", + "OMP_", + "PREFILL_", + "PYTORCH_", + "RCCL_", + "ROCM_", + "ROCR_", + "SGLANG_", + "SWEBENCH_", + "TILELANG_", + "TORCH_", + "TRITON_", + "UCX_", + "VLLM_", +) + +# Values that define the exact native AgentX arm selected for a session. The +# CLI snapshots every non-empty value at seed time and restores it before a +# fresh-shell resume can materialize another benchmark. Keep the resolved +# fingerprint and physical GPU count here too: the former detects recipe drift +# and the latter deliberately differs from recipe ``TP`` when PP/PCP > 1. +AGENTX_RUNTIME_PIN_NAMES = ( + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "INFERENCEX_PATH", + "HYPERLOOM_IMAGE", + "AGENTX_MODE", + "AGENTX_RECIPE", + "AGENTX_CONFIG_FILE", + "AGENTX_SELECTOR", + "AGENTX_FAILED_REQUEST_THRESHOLD", + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT", + "HYPERLOOM_AGENTX_GPU_COUNT", + "MAGPIE_REF", + "INFERENCEX_REF", +) + +# These four values existed before the native Magpie bridge and remain the +# minimum needed by the legacy/pre-baseline path. An accepted native baseline +# is held to the stronger fingerprint set in ``_restore_agentx_runtime_pins``. +AGENTX_REQUIRED_RUNTIME_PIN_NAMES = ( + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "INFERENCEX_PATH", + "HYPERLOOM_IMAGE", +) + +# Persisted epoch-2 sessions use these exact shell launcher bytes. New sessions +# resolve Magpie's server-launch-spec and never consult this historical mapping. +_NATIVE_LAUNCHER_MANIFEST: dict[tuple[str, str], tuple[str, str]] = { + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "qwen3.5-fp8-mi325x-sglang-agentic-mtp", + ): ( + "single_node/agentic/qwen3.5_fp8_mi325x_mtp.sh", + "227452668acec427535a80aea00736271742f506a9f739ac3f8621e516443e8e", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "qwen3.5-fp4-mi355x-sglang-agentic-mtp", + ): ( + "single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh", + "889f82eab6e05bc81b77e374eb71853d8c9701fd17246aa9e9be38437cb8f161", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "qwen3.5-fp8-mi300x-sglang-agentic-mtp", + ): ( + "single_node/agentic/qwen3.5_fp8_mi300x_mtp.sh", + "a56d14c8056cdebfc9d5f1f53b60c0279f12b85f062d987b6ac593bd1d3e8694", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "kimik3-fp4-mi355x-vllm-agentic-mtp", + ): ( + "single_node/agentic/kimik3_fp4_mi355x_mtp.sh", + "4de24764a0bdf0ea9a21d5f5c5c304d2754097fb8398da10d60793ff85427125", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "dsv4-fp4-mi355x-vllm-agentic-mtp", + ): ( + "single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh", + "80f26dfce576b0e1375510303f96089f92df416f52adb26a019449ebb2e85d3f", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "minimaxm3-fp8-mi300x-vllm-agentic-mtp", + ): ( + "single_node/agentic/minimaxm3_fp8_mi300x_mtp.sh", + "8fc7a35d28ea2f6ccf353c176e2d0c813d1a989f64ebf8c9cda26f83e05bb278", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "glm5.2-fp8-mi325x-sglang-agentic-mtp", + ): ( + "single_node/agentic/glm5.2_fp8_mi325x_mtp.sh", + "e48b43d03c71c439833d7369002ef92bf50fabc3b0a20c738dc751b2dbbe8ef7", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "minimaxm3-fp8-mi325x-vllm-agentic-mtp", + ): ( + "single_node/agentic/minimaxm3_fp8_mi325x_mtp.sh", + "99f47d000f94445bcf66882438388d9a14ed52b7cdaef7b8c4bdcd3d5cfe67f2", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "minimaxm3-fp4-mi355x-vllm-agentic-mtp", + ): ( + "single_node/agentic/minimaxm3_fp4_mi355x_mtp.sh", + "630cf96f51d2a288212c389a05b7df312745f4bac074e34306bf154d8cb40b47", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "glm5.2-fp4-mi355x-sglang-agentic-mtp", + ): ( + "single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh", + "5083b395d4af15cd6b668c16f744954ee0d90c2dc607e585b8610072ffc5e099", + ), + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "dsv4-fp4-mi355x-sglang-agentic-mtp", + ): ( + "single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh", + "da70afbb14b48766a0fe1e1d590ec123eda6654a4c1ca419961354545f629ddc", + ), +} + +_NATIVE_LAUNCHER_TRANSITIVE_INPUTS: dict[tuple[str, str], tuple[str, ...]] = { + ( + "3d5581562f643f9bdeb8410cd924e2c70906c966", + "single_node/agentic/kimik3_fp4_mi355x_mtp.sh", + ): ("benchmarks/single_node/agentic/apply_k3_container_patches.sh",), +} + +# Persisted epoch-3 sessions retain the launch contract of their accepted pin. +_NATIVE_LAUNCH_CONTRACT_REFS: frozenset[str] = frozenset({"421312f8984c2152f4b8eafefc93ea2fa598e80f"}) +_NATIVE_LAUNCH_CONTRACT_INPUTS = ( + "configs/agentx-launchers.json", + "utils/agentic/server_launch.py", + "utils/agentic/custom_model.py", +) + +_MAGPIE_SOURCE_IDENTITY_CODE = r""" +import hashlib +import json +import re +import subprocess +from importlib.metadata import PackageNotFoundError, distribution +from pathlib import Path +from urllib.parse import unquote, urlparse + + +_MAGPIE_EXECUTION_SUFFIXES = (".py", ".sh", ".yaml", ".yaml.example", ".json") +# The pinned source commit contains Magpie/mcp/config.json, but its wheel does +# not publish that file (pyproject.toml only publishes top-level JSON). It is +# unrelated to AgentX. Hash the complete *published* execution tree so wheels +# and clean editable/source installs have one canonical identity. +_MAGPIE_UNPUBLISHED_PATHS = frozenset({"mcp/config.json"}) +_AUDITED_MAGPIE_EXECUTION_TREES = { + # Keep the release identity for native sessions resuming their accepted pin. + "5d6aaa29f2ee78e53ec35c8bc2e082d8a10c47f0": { + "file_count": 79, + "tree_sha256": "afea9b95e992e0b715c955f5e72a7e92aca48d2c171ddff0e3d3e9082406e689", + }, + # v0.3.0 plus the generic eval source-path hotfix, independently audited from its wheel. + "a3339dc2776ee0c977fb3313fe89f56da7a91555": { + "file_count": 79, + "tree_sha256": "113f880b18ccd3ec26e6a432fcdf06a0c365520d51c3ed3c46069d33d7f07e93", + }, + "d80eb4d3dad7fabe01ce81d049e2983adf2c86dd": { + "file_count": 87, + "tree_sha256": "4c827c3d4b6bb7f756964fdecaeb43b255470ee3285f5dc89d9d320c7e9164dc", + }, + "cf805323f0a6db6c63a0aa4ffcce7acf4e830ea6": { + "file_count": 87, + "tree_sha256": "65b4acccbe70c05c033f75a5054344bd2fead13dc81878f343427c6c04142e4a", + }, + "311d56429a3aadc3cfe1e1e0e2e533482d224cf0": { + "file_count": 87, + "tree_sha256": "f8be61cf4566f4daaf3c2463a600e49b11d97bade6632a7bb5b99050b4ce5656", + }, + "938bfd7a9bc1f3970f3fbce8b65ad88a8e0958f5": { + "file_count": 87, + "tree_sha256": "af43bc3fcbe768b30b87cd5a23ff6f4d36e0e865a8db7375ca4e205970990768", + }, + "63e3b19bee66700dbb012c538f121c9de0d86a2a": { + "file_count": 87, + "tree_sha256": "624a89b8e74adda9ab95e45d692cfb58a267b7e2d4de9a96076db24626d49e20", + }, + "c5c80698fef1b89cc6882264b80d5b306d4e9328": { + "file_count": 87, + "tree_sha256": "1f475d413e7af20204da8b8abcd06a6a69916d3960f175c72d627663ce33d94f", + }, + "658562345ad1a7e5a617e3631f3acfcec0eade4a": { + "file_count": 86, + "tree_sha256": "514ca3691163bb304cef31d392d9c10bbb5a3b3570b219f95ed12c6aeae29c69", + }, + "d72965776df5416dad063c00237f6e389b841162": { + "file_count": 81, + "tree_sha256": "84dd7920ff992571c7dc6d485a4680ac3fc2afe7b6dde606dbd754d545a5729c", + }, +} + + +def _magpie_execution_tree_identity(package_root): + '''Hash every published executable/config input below ``Magpie/``.''' + package_root = Path(package_root).resolve(strict=True) + files = {} + for path in sorted(package_root.rglob("*"), key=lambda item: item.as_posix()): + relative = path.relative_to(package_root) + if "__pycache__" in relative.parts: + continue + relative_name = relative.as_posix() + if path.is_symlink(): + raise RuntimeError( + "installed Magpie execution tree must not contain symlinks: " + + relative_name + ) + if ( + not path.is_file() + or relative_name in _MAGPIE_UNPUBLISHED_PATHS + or not relative_name.endswith(_MAGPIE_EXECUTION_SUFFIXES) + ): + continue + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + files[relative_name] = digest.hexdigest() + tree_sha256 = hashlib.sha256( + json.dumps(files, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + return {"file_count": len(files), "tree_sha256": tree_sha256} + + +def _validate_magpie_execution_tree(package_root, source_commit): + expected = _AUDITED_MAGPIE_EXECUTION_TREES.get(source_commit) + if expected is None: + raise RuntimeError( + "installed Magpie commit is not audited for native AgentX: " + + (source_commit or "") + ) + actual = _magpie_execution_tree_identity(package_root) + if actual != expected: + raise RuntimeError( + "installed Magpie published execution tree differs from the audited " + f"commit: expected {expected}, got {actual}" + ) + return actual + + +def _resolve_magpie_source_identity(package_root): + '''Return the commit/url that own an imported Magpie package. + + A wheel normally lives below ``/.venv/site-packages``. Git + walks parent directories, so an unconditional ``git -C site-packages`` can + accidentally report that enclosing checkout's HEAD as Magpie's commit. + Prefer PEP 610 VCS provenance for wheels. Only consult Git for an editable + or direct source import whose package is the checkout's own top-level + ``Magpie/`` directory. + ''' + package_root = Path(package_root).resolve() + dist = None + try: + dist = distribution("magpie-eval") + except (PackageNotFoundError, OSError, TypeError, ValueError): + pass + try: + direct_url = ( + json.loads(dist.read_text("direct_url.json") or "{}") if dist else {} + ) + except (json.JSONDecodeError, OSError, TypeError, ValueError): + direct_url = {} + if not isinstance(direct_url, dict): + direct_url = {} + + source_url = str(direct_url.get("url") or "") + dir_info = direct_url.get("dir_info") + dir_info = dir_info if isinstance(dir_info, dict) else {} + editable = dir_info.get("editable") is True + vcs_info = direct_url.get("vcs_info") + vcs_info = vcs_info if isinstance(vcs_info, dict) else {} + vcs_commit = str(vcs_info.get("commit_id") or "").strip().lower() + + if dist is not None and not editable: + try: + distribution_root = Path(dist.locate_file("Magpie")).resolve(strict=True) + except (AttributeError, OSError, RuntimeError, TypeError, ValueError) as exc: + raise RuntimeError( + "cannot bind the magpie-eval distribution to imported Magpie" + ) from exc + if distribution_root != package_root: + raise RuntimeError( + "magpie-eval distribution does not own the imported Magpie package: " + f"distribution={distribution_root}, imported={package_root}" + ) + + if ( + dist is not None + and not editable + and str(vcs_info.get("vcs") or "").strip().lower() == "git" + and re.fullmatch(r"[0-9a-f]{40}", vcs_commit) + ): + return vcs_commit, source_url + + if dist is not None and not editable: + # A non-VCS wheel/local archive has no immutable source identity. Do + # not fall through to an unrelated repository above site-packages. + return "", source_url + + source_root = package_root.parent + if editable: + parsed = urlparse(source_url) + if parsed.scheme not in {"", "file"} or parsed.netloc not in {"", "localhost"}: + return "", source_url + source_root = Path(unquote(parsed.path)).resolve() + + top_proc = subprocess.run( + ["git", "-C", str(source_root), "rev-parse", "--show-toplevel"], + capture_output=True, + text=True, + check=False, + ) + if top_proc.returncode != 0: + return "", source_url + try: + git_top = Path((top_proc.stdout or "").strip()).resolve(strict=True) + except (OSError, RuntimeError): + return "", source_url + if editable and source_root != git_top: + return "", source_url + if package_root != (git_top / "Magpie").resolve(): + return "", source_url + + head_proc = subprocess.run( + ["git", "-C", str(git_top), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=False, + ) + commit = (head_proc.stdout or "").strip().lower() if head_proc.returncode == 0 else "" + if not re.fullmatch(r"[0-9a-f]{40}", commit): + return "", source_url + status_proc = subprocess.run( + [ + "git", + "-C", + str(git_top), + "status", + "--porcelain=v1", + "--untracked-files=all", + "--", + "Magpie", + ], + capture_output=True, + text=True, + check=False, + ) + if status_proc.returncode != 0: + return "", source_url + dirty = [ + line + for line in (status_proc.stdout or "").splitlines() + if line.strip() + and not (line.startswith("?? ") and "__pycache__/" in line) + ] + if dirty: + raise RuntimeError( + "native AgentX requires a clean Magpie/ source tree: " + ", ".join(dirty) + ) + return commit, source_url or git_top.as_uri() +""" + + +_RESOLVER_CODE = ( + _MAGPIE_SOURCE_IDENTITY_CODE + + r""" +import hashlib +import json +import re +from pathlib import Path +import sys + +import Magpie +from Magpie.modes.benchmark import BenchmarkConfig +from Magpie.modes.benchmark.agentx import resolve_agentx_recipe + + +def sha256_file(path): + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +package_root = Path(Magpie.__file__).resolve().parent +source_commit, source_url = _resolve_magpie_source_identity(package_root) +magpie_tree = _validate_magpie_execution_tree(package_root, source_commit) +magpie_identity = { + "source_commit": source_commit, + "source_url": source_url, + "published_execution_tree": magpie_tree, +} +magpie_execution = {"package_root": str(package_root), **magpie_identity} +magpie_execution["fingerprint"] = hashlib.sha256( + json.dumps(magpie_identity, sort_keys=True, separators=(",", ":")).encode("utf-8") +).hexdigest() + +payload = json.load(sys.stdin) +config = BenchmarkConfig.from_dict(payload["benchmark"]) +spec = resolve_agentx_recipe( + config, + payload["inferencex_path"], + runner_type=str(config.runner_type or ""), +) +result = { + "benchmark": config.to_dict(), + "recipe": spec.recipe, + "config_file": spec.config_file, + "entry": spec.entry, + "magpie_execution": magpie_execution, +} +print("HYPERLOOM_AGENTX_RECIPE=" + json.dumps(result, separators=(",", ":"))) +""" +) + + +def _run_magpie_recipe_resolver( + benchmark: MutableMapping[str, Any], + *, + inferencex_path: Path, +) -> dict[str, Any]: + """Resolve through the same interpreter that will execute Magpie.""" + from hyperloom.common.env_safety import scrub_benchmark_process_env + from hyperloom.orchestrator.actions.executors.benchmark_backend import ( + resolve_benchmark_interpreter, + ) + + inferencex_path = validate_native_checkout_path(inferencex_path) + interpreter = resolve_benchmark_interpreter() + payload = json.dumps( + { + "benchmark": {**benchmark, "inferencex_path": str(inferencex_path)}, + "inferencex_path": str(inferencex_path), + } + ) + try: + completed = subprocess.run( + [interpreter, "-c", _RESOLVER_CODE], + input=payload, + text=True, + capture_output=True, + timeout=120, + env=scrub_benchmark_process_env(dict(os.environ)), + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + raise RuntimeError(f"Unable to run Magpie AgentX recipe resolver with {interpreter!r}: {exc}") from exc + if completed.returncode != 0: + detail = (completed.stderr or completed.stdout or "").strip() + raise RuntimeError( + f"Magpie AgentX recipe resolution failed in the benchmark interpreter {interpreter!r}: {detail[-2000:]}" + ) + encoded = next( + ( + line[len(_RESOLVER_SENTINEL) :] + for line in reversed((completed.stdout or "").splitlines()) + if line.startswith(_RESOLVER_SENTINEL) + ), + "", + ) + if not encoded: + raise RuntimeError("Magpie AgentX recipe resolver returned no structured result") + try: + result = json.loads(encoded) + except json.JSONDecodeError as exc: + raise RuntimeError("Magpie AgentX recipe resolver returned invalid JSON") from exc + if not isinstance(result, dict) or not isinstance(result.get("benchmark"), dict): + raise RuntimeError("Magpie AgentX recipe resolver returned an invalid result") + from .identity import has_launch_contract + + if has_launch_contract(benchmark) and not has_launch_contract(result["benchmark"]): + raise RuntimeError("Pinned Magpie does not support the native AgentX optimization launch contract") + return result + + +def validate_native_checkout_path(inferencex_path: str | Path) -> Path: + """Normalize managed projects; keep historical shell launchers path-safe.""" + raw = str(inferencex_path or "") + if not raw: + raise ValueError("Native AgentX requires a non-empty InferenceX checkout path") + if raw != raw.strip(): + raise ValueError("Native AgentX InferenceX checkout path is shell-unsafe for the pinned Magpie launcher") + try: + resolved = Path(raw).expanduser().resolve() + except (OSError, RuntimeError, ValueError) as exc: + raise ValueError("Native AgentX InferenceX checkout path is not resolvable") from exc + from .managed import project_root + + resolved = project_root(resolved) + managed = (resolved / "benchmarks/srt_agentic.sh").is_file() + if not managed and not _NATIVE_CHECKOUT_PATH_RE.fullmatch(str(resolved)): + raise ValueError( + f"Native AgentX InferenceX checkout path is shell-unsafe for the pinned Magpie launcher: {resolved}" + ) + return resolved + + +def native_runtime_search_paths() -> dict[str, str]: + """Pin the same runtime imports during recipe selection and materialization.""" + paths = { + name: value + for name in ("PATH", "PYTHONPATH", "LD_LIBRARY_PATH", "LIBRARY_PATH") + if (value := os.environ.get(name, "").strip()) + } + if paths.get("PATH") and paths["PATH"].split(":", 1)[0] != "/opt/venv/bin": + paths["PATH"] = f"/opt/venv/bin:{paths['PATH']}" + return paths + + +def validate_native_replay_env(env: Mapping[str, Any], *, source: str) -> None: + """Reject legacy replay settings that native AgentX cannot honour.""" + overrides = sorted( + str(name) + for name, value in env.items() + if value is not None + and str(value).strip() + and ( + str(name).upper().startswith(("AIPERF_", "AGENTIC_")) + or str(name).upper() in {"AGENTX_DATASET", "AGENTX_WARMUP_REQUESTS_PER_LANE", "WEKA_LOADER_OVERRIDE"} + ) + ) + if overrides: + raise ValueError( + f"Native AgentX does not support legacy replay controls in {source}: " + + ", ".join(overrides) + + ". Remove them; use benchmark.agentx and the resolved InferenceX recipe for replay settings." + ) + + +def scrub_native_agentx_ambient_env(env: MutableMapping[str, str]) -> None: + """Remove generic replay controls and inherited Bash functions.""" + for name in tuple(env): + upper = str(name).upper() + if upper.startswith(("AIPERF_", "AGENTIC_", "BASH_FUNC_")) or upper == "WEKA_LOADER_OVERRIDE": + env.pop(name, None) + + +def native_launch_environment_identity( + resolved_benchmark: Mapping[str, Any], + *, + launch_env: Mapping[str, str] | None = None, +) -> dict[str, Any]: + """Hash the effective child environment that can alter native execution. + + This mirrors the real process boundary: Hyperloom first removes child + startup hooks and control-plane credentials, then removes generic AgentX + replay overrides, and finally Magpie overlays ``benchmark.envs``. Only + audited launcher/framework controls are retained in the identity; cache + routing and authentication values such as HF_HOME/HF_TOKEN remain usable + without being persisted or making a fresh-shell resume incomparable. + """ + from hyperloom.common.env_safety import scrub_benchmark_process_env + + source = os.environ if launch_env is None else launch_env + effective = scrub_benchmark_process_env({str(key): str(value) for key, value in source.items()}) + scrub_native_agentx_ambient_env(effective) + # Hyperloom deliberately drops its remote model identifier here: native + # launchers interpret any non-empty MODEL_PATH as a local directory. + effective.pop("MODEL_PATH", None) + raw_envs = resolved_benchmark.get("envs") + benchmark_envs = raw_envs if isinstance(raw_envs, Mapping) else {} + effective.update({str(key): str(value) for key, value in benchmark_envs.items()}) + controls = { + name: value + for name, value in effective.items() + if name.upper() in _NATIVE_AMBIENT_CONTROL_NAMES or name.upper().startswith(_NATIVE_AMBIENT_CONTROL_PREFIXES) + } + encoded = json.dumps(controls, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8") + return { + "names": sorted(controls), + "sha256": hashlib.sha256(encoded).hexdigest(), + } + + +def validate_native_launcher_name(script_name: str) -> PurePosixPath: + """Validate an explicit launcher relative to ``InferenceX/benchmarks``.""" + raw = str(script_name or "").strip() + path = PurePosixPath(raw) + if raw == "srt_agentic.sh": + return path + if ( + not raw + or path.is_absolute() + or ".." in path.parts + or path.parts[:2] != _AGENTIC_PREFIX + or path.suffix != ".sh" + or len(path.parts) != 3 + ): + raise ValueError( + "AGENTX_SERVER_SCRIPT must name one explicit InferenceX launcher " + "under single_node/agentic/ (for example " + "single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh)" + ) + return path + + +def resolve_native_launcher( + *, + inferencex_path: str | Path, + benchmark_script: str, +) -> Path: + """Resolve an explicit native launcher without modifying the checkout.""" + rel = validate_native_launcher_name(benchmark_script) + root = validate_native_checkout_path(inferencex_path) + benchmarks_root = (root / "benchmarks").resolve() + lexical_launcher = benchmarks_root / Path(*rel.parts) + if _contains_symlink(benchmarks_root, lexical_launcher): + raise ValueError(f"AgentX launcher path must not contain symlinks: {lexical_launcher}") + launcher = lexical_launcher.resolve() + agentic_root = ( + benchmarks_root + if rel.as_posix() == "srt_agentic.sh" + else (benchmarks_root / "single_node" / "agentic").resolve() + ) + try: + launcher.relative_to(agentic_root) + except ValueError as exc: + raise ValueError(f"AgentX launcher escapes {agentic_root}: {launcher}") from exc + if not launcher.is_file(): + raise FileNotFoundError( + f"Configured native AgentX launcher does not exist: {launcher}. " + "AGENTX_SERVER_SCRIPT must match the pinned InferenceX checkout." + ) + return launcher + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _canonical_benchmark_sha256(benchmark: Mapping[str, Any]) -> str: + """Hash the resolved Magpie launch config, including forwarded envs.""" + try: + payload = json.dumps( + dict(benchmark), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError) as exc: + raise ValueError("resolved native AgentX benchmark is not canonically serializable") from exc + return hashlib.sha256(payload).hexdigest() + + +def _contains_symlink(root: Path, path: Path) -> bool: + current = root + for part in path.relative_to(root).parts: + current = current / part + if current.is_symlink(): + return True + return False + + +def _head_blob(root: Path, relative_path: str) -> bytes: + completed = subprocess.run( + ["git", "-C", str(root), "show", f"HEAD:{relative_path}"], + capture_output=True, + timeout=30, + check=False, + ) + if completed.returncode != 0: + raise ValueError(f"native AgentX execution input is not tracked at HEAD: {relative_path}") + return completed.stdout + + +def _checkout_head(root: Path, *, label: str) -> str: + """Return one exact git commit, or reject an unverifiable checkout.""" + try: + completed = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except (OSError, subprocess.SubprocessError) as exc: + raise ValueError(f"cannot read {label} git identity at {root}: {exc}") from exc + head = (completed.stdout or "").strip().lower() + if completed.returncode != 0 or not re.fullmatch(r"[0-9a-f]{40}", head): + raise ValueError(f"native AgentX requires a readable {label} git checkout at {root}") + return head + + +def _launcher_manifest(root: Path, head: str) -> dict[str, tuple[str, str]]: + if head not in _NATIVE_LAUNCH_CONTRACT_REFS: + return {recipe: item for (commit, recipe), item in _NATIVE_LAUNCHER_MANIFEST.items() if commit == head} + manifest_path = root / "configs" / "agentx-launchers.json" + if _contains_symlink(root, manifest_path) or manifest_path.read_bytes() != _head_blob( + root, "configs/agentx-launchers.json" + ): + raise ValueError("Native AgentX launcher manifest changed from its pinned source") + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if not isinstance(manifest, dict) or manifest.get("version") != 1 or not isinstance(manifest.get("recipes"), dict): + raise ValueError("Native AgentX launcher manifest has an unsupported schema") + generic = manifest.get("generic", {}) + if not isinstance(generic, dict) or manifest["recipes"].keys() & generic.keys(): + raise ValueError("Native AgentX launcher manifest has invalid generic capabilities") + entries: dict[str, tuple[str, str]] = {} + for recipe, entry in {**manifest["recipes"], **generic}.items(): + if not isinstance(entry, dict) or entry.get("launch_overrides_version") != 1: + raise ValueError(f"Native AgentX recipe {recipe!r} has no supported launch contract") + script = validate_native_launcher_name(str(entry.get("benchmark_script") or "")).as_posix() + entries[recipe] = (script, hashlib.sha256(_head_blob(root, f"benchmarks/{script}")).hexdigest()) + return entries + + +def validate_native_recipe_launcher( + *, + inferencex_path: str | Path, + recipe: str, + benchmark_script: str, +) -> dict[str, str]: + """Verify the managed client or an existing session's pinned shell launcher.""" + root = validate_native_checkout_path(inferencex_path) + head = _checkout_head(root, label="InferenceX") + recipe_name = str(recipe or "").strip() + if benchmark_script == "srt_agentic.sh": + launcher = resolve_native_launcher(inferencex_path=root, benchmark_script=benchmark_script) + from .managed import head_blob + + if launcher.read_bytes() != head_blob(root, "benchmarks/srt_agentic.sh"): + raise ValueError("Managed AgentX client differs from pinned HEAD") + return { + "inferencex_commit": head, + "recipe": recipe_name, + "launcher": benchmark_script, + "launcher_sha256": _sha256_file(launcher), + } + manifest_entry = _launcher_manifest(root, head).get(recipe_name) + if manifest_entry is None: + raise ValueError( + "Native AgentX has no audited launcher mapping for " + f"InferenceX {head} recipe {recipe_name!r}. Update the exact-ref " + "compatibility manifest instead of guessing a launcher." + ) + expected_script, expected_sha256 = manifest_entry + actual_script = validate_native_launcher_name(benchmark_script).as_posix() + if actual_script != expected_script: + raise ValueError( + f"AgentX recipe {recipe_name!r} requires audited launcher " + f"{expected_script!r} at InferenceX {head}; got {actual_script!r}" + ) + launcher = resolve_native_launcher( + inferencex_path=root, + benchmark_script=actual_script, + ) + actual_sha256 = _sha256_file(launcher) + if actual_sha256 != expected_sha256: + raise ValueError( + "Native AgentX launcher bytes do not match the audited manifest: " + f"{actual_script} expected {expected_sha256}, got {actual_sha256}" + ) + return { + "inferencex_commit": head, + "recipe": recipe_name, + "launcher": actual_script, + "launcher_sha256": actual_sha256, + } + + +def native_execution_identity( + *, + inferencex_path: str | Path, + benchmark_script: str, + config_file: str, + resolved_benchmark: Mapping[str, Any], + expected_ref: str = "", + magpie_execution: Mapping[str, Any] | None = None, + expected_magpie_ref: str = "", + launch_env: Mapping[str, str] | None = None, +) -> dict[str, Any]: + """Bind a native run to clean executable inputs from one git checkout.""" + root = validate_native_checkout_path(inferencex_path) + from .managed import execution_identity, identity_benchmark, server_spec + + if server_spec(resolved_benchmark) is not None: + identity_config = identity_benchmark( + resolved_benchmark, + str( + (os.environ if launch_env is None else launch_env).get("HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT") + or "" + ), + ) + return execution_identity( + root=root, + benchmark=identity_config, + config_file=config_file, + expected_ref=expected_ref, + magpie=magpie_execution or {}, + expected_magpie_ref=expected_magpie_ref, + launch_environment=native_launch_environment_identity(resolved_benchmark, launch_env=launch_env), + ) + launcher = resolve_native_launcher( + inferencex_path=root, + benchmark_script=benchmark_script, + ) + benchmark_lib = root / "benchmarks" / "benchmark_lib.sh" + runners_config = root / "configs" / "runners.yaml" + raw_config = PurePosixPath(str(config_file or "").strip()) + if not str(raw_config) or raw_config.is_absolute() or ".." in raw_config.parts: + raise ValueError("resolved AgentX config_file must be relative to the InferenceX checkout") + lexical_config = root / Path(*raw_config.parts) + if _contains_symlink(root, lexical_config): + raise ValueError(f"AgentX config_file path must not contain symlinks: {lexical_config}") + config_path = lexical_config.resolve() + try: + config_path.relative_to(root) + except ValueError as exc: + raise ValueError(f"AgentX config_file escapes {root}: {config_path}") from exc + missing = [str(path) for path in (benchmark_lib, runners_config, config_path) if not path.is_file()] + if missing: + raise FileNotFoundError("Native AgentX execution input is missing: " + ", ".join(missing)) + for execution_path in (launcher, benchmark_lib, runners_config, config_path): + if _contains_symlink(root, execution_path): + raise ValueError(f"native AgentX execution inputs must not contain symlinks: {execution_path}") + + head = _checkout_head(root, label="InferenceX") + wanted = str(expected_ref or "").strip().lower() + if wanted and not re.fullmatch(r"[0-9a-f]{40}", wanted): + raise ValueError( + f"Native AgentX requires INFERENCEX_REF to be an immutable 40-character commit SHA; got {expected_ref!r}" + ) + if wanted and head != wanted: + raise ValueError(f"InferenceX HEAD {head} does not match the pinned ref {wanted}") + from .identity import has_launch_contract + + if has_launch_contract(resolved_benchmark) and head not in _NATIVE_LAUNCH_CONTRACT_REFS: + raise ValueError("Pinned InferenceX does not support the native AgentX optimization launch contract") + + launcher_manifest_entries = [ + (recipe, expected_sha256) + for recipe, (script, expected_sha256) in _launcher_manifest(root, head).items() + if script == benchmark_script + ] + if not launcher_manifest_entries: + raise ValueError( + f"native AgentX launcher is not audited for this exact InferenceX commit: {head} {benchmark_script!r}" + ) + launcher_sha256 = _sha256_file(launcher) + if all(launcher_sha256 != expected_sha256 for _recipe, expected_sha256 in launcher_manifest_entries): + raise ValueError(f"native AgentX launcher bytes do not match the exact-ref manifest: {benchmark_script}") + transitive_relative = sorted( + { + helper + for helper in _NATIVE_LAUNCHER_TRANSITIVE_INPUTS.get( + (head, benchmark_script), + (), + ) + } + ) + if head in _NATIVE_LAUNCH_CONTRACT_REFS: + transitive_relative = sorted( + set(transitive_relative) + | set(_NATIVE_LAUNCH_CONTRACT_INPUTS) + | { + path + for (_commit, script), paths in _NATIVE_LAUNCHER_TRANSITIVE_INPUTS.items() + if script == benchmark_script + for path in paths + } + ) + transitive_paths = [root / Path(*PurePosixPath(item).parts) for item in transitive_relative] + missing_transitive = [str(path) for path in transitive_paths if not path.is_file()] + if missing_transitive: + raise FileNotFoundError("Native AgentX transitive launcher input is missing: " + ", ".join(missing_transitive)) + + relative_inputs = ( + launcher.relative_to(root).as_posix(), + benchmark_lib.relative_to(root).as_posix(), + runners_config.relative_to(root).as_posix(), + config_path.relative_to(root).as_posix(), + "utils", + *transitive_relative, + ) + try: + status_proc = subprocess.run( + [ + "git", + "-C", + str(root), + "status", + "--porcelain=v1", + "--untracked-files=all", + "--ignore-submodules=none", + "--", + *relative_inputs, + ], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except (OSError, subprocess.SubprocessError) as exc: + raise ValueError(f"cannot verify native AgentX execution inputs at {root}: {exc}") from exc + dirty = (status_proc.stdout or "").strip() + if status_proc.returncode != 0: + detail = (status_proc.stderr or dirty).strip() + raise ValueError(f"cannot verify native AgentX execution inputs at {root}: {detail}") + if dirty: + changed = ", ".join(line[3:].strip() for line in dirty.splitlines() if line.strip()) + raise ValueError( + f"native AgentX requires clean pinned launcher/replay inputs; modified paths in {root}: {changed}" + ) + + for execution_path in ( + launcher, + benchmark_lib, + runners_config, + config_path, + *transitive_paths, + ): + if _contains_symlink(root, execution_path): + raise ValueError(f"native AgentX execution inputs must not contain symlinks: {execution_path}") + relative = execution_path.relative_to(root).as_posix() + if execution_path.read_bytes() != _head_blob(root, relative): + raise ValueError(f"native AgentX execution input differs from the pinned HEAD blob: {relative}") + + gitlink_proc = subprocess.run( + ["git", "-C", str(root), "ls-tree", "HEAD", "utils/aiperf"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + gitlink_fields = (gitlink_proc.stdout or "").strip().split() + if ( + gitlink_proc.returncode != 0 + or len(gitlink_fields) < 3 + or gitlink_fields[0] != "160000" + or not re.fullmatch(r"[0-9a-f]{40}", gitlink_fields[2]) + ): + raise ValueError("InferenceX utils/aiperf is not a pinned git submodule") + aiperf_commit = gitlink_fields[2] + aiperf_root = root / "utils" / "aiperf" + if not (aiperf_root / "pyproject.toml").is_file(): + raise ValueError( + "InferenceX utils/aiperf submodule is not initialized; run git submodule update --init -- utils/aiperf" + ) + submodule_head = subprocess.run( + ["git", "-C", str(aiperf_root), "rev-parse", "HEAD"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + actual_aiperf_commit = (submodule_head.stdout or "").strip().lower() + if submodule_head.returncode != 0 or actual_aiperf_commit != aiperf_commit: + raise ValueError(f"InferenceX utils/aiperf does not match its pinned gitlink {aiperf_commit}") + submodule_status = subprocess.run( + [ + "git", + "-C", + str(aiperf_root), + "status", + "--porcelain=v1", + "--untracked-files=all", + ], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if submodule_status.returncode != 0 or (submodule_status.stdout or "").strip(): + raise ValueError("InferenceX utils/aiperf submodule is not clean") + + magpie = dict(magpie_execution or {}) + magpie_fingerprint = str(magpie.get("fingerprint") or "").strip().lower() + magpie_commit = str(magpie.get("source_commit") or "").strip().lower() + if not re.fullmatch(r"[0-9a-f]{64}", magpie_fingerprint): + raise ValueError("Magpie resolver did not provide a valid execution fingerprint") + wanted_magpie = str(expected_magpie_ref or "").strip().lower() + if wanted_magpie and not re.fullmatch(r"[0-9a-f]{40}", wanted_magpie): + raise ValueError( + f"Native AgentX requires MAGPIE_REF to be an immutable 40-character commit SHA; got {expected_magpie_ref!r}" + ) + if wanted_magpie and magpie_commit != wanted_magpie: + raise ValueError( + f"Magpie source commit {magpie_commit or ''} does not match the pinned ref {wanted_magpie}" + ) + + identity = { + "inferencex_commit": head, + "launcher": launcher.relative_to(root).as_posix(), + "launcher_sha256": launcher_sha256, + "benchmark_lib_sha256": _sha256_file(benchmark_lib), + "runners_config_sha256": _sha256_file(runners_config), + "config_file": config_path.relative_to(root).as_posix(), + "config_sha256": _sha256_file(config_path), + "aiperf_commit": aiperf_commit, + "magpie_fingerprint": magpie_fingerprint, + "magpie_commit": magpie_commit, + "transitive_inputs": {path.relative_to(root).as_posix(): _sha256_file(path) for path in transitive_paths}, + } + # This stable layer is established by the CLI before workload + # materialization adds generated fields (for example ROCR_VISIBLE_DEVICES + # and warm-up controls). It pins the executable sources without falsely + # treating those deterministic additions as checkout drift. + identity["static_execution_fingerprint"] = hashlib.sha256( + json.dumps(identity, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + from .identity import has_launch_contract, native_workload_fingerprint + + if has_launch_contract(resolved_benchmark): + identity["workload_fingerprint"] = native_workload_fingerprint( + resolved_benchmark, identity["static_execution_fingerprint"] + ) + # The complete resolved BenchmarkConfig is accepted only after + # materialization and persisted beside the recipe. Magpie overlays its + # envs onto the InferenceX launcher environment, so this second layer also + # binds PATH/PYTHONPATH and framework-specific env changes which do not + # affect Magpie's recipe fingerprint. + identity["launch_config_sha256"] = _canonical_benchmark_sha256(resolved_benchmark) + identity["launch_environment"] = native_launch_environment_identity( + resolved_benchmark, + launch_env=launch_env, + ) + identity["execution_fingerprint"] = hashlib.sha256( + json.dumps(identity, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + return identity + + +def preview_native_recipe( + benchmark: MutableMapping[str, Any], + *, + inferencex_path: str | Path, +) -> dict[str, Any]: + """Resolve and validate the recipe identity before GPU scheduling. + + This is intentionally read-only with respect to ``benchmark``. The CLI + uses it to learn the physical GPU count (TP x PP x PCP) before it exports + workload defaults or asks the scheduler for resources. Full launch-time + validation, including the visible-device mask and outer-image attestation, + remains in :func:`resolve_native_recipe`. + """ + root = validate_native_checkout_path(inferencex_path) + if not root.is_dir(): + raise FileNotFoundError(f"Native AgentX recipe resolution requires an existing InferenceX checkout; got {root}") + resolved_payload = _run_magpie_recipe_resolver( + benchmark, + inferencex_path=root, + ) + resolved_benchmark = resolved_payload["benchmark"] + if isinstance(benchmark.get("workload_spec"), dict): + resolved_benchmark["workload_spec"] = dict(benchmark["workload_spec"]) + run_mode = str(resolved_benchmark.get("run_mode") or "local").lower() + if run_mode != "local": + raise ValueError( + "Hyperloom's native AgentX bridge requires benchmark.run_mode=local; " + "the optimizer already runs inside the recipe image" + ) + entry_raw = resolved_payload.get("entry") + if not isinstance(entry_raw, dict): + raise RuntimeError("Magpie AgentX recipe resolver omitted the resolved entry") + entry = dict(entry_raw) + validate_native_recipe_launcher( + inferencex_path=root, + recipe=str(resolved_payload.get("recipe") or ""), + benchmark_script=str(resolved_benchmark.get("benchmark_script") or ""), + ) + try: + tp = int(entry["tp"]) + pp = int(entry.get("pp", 1)) + pcp_size = int(entry.get("pcp-size", 1)) + ep = int(entry.get("ep", 1)) + resolved_envs = resolved_benchmark.get("envs") + env_conc = resolved_envs.get("CONC") if isinstance(resolved_envs, dict) else None + raw_conc = entry.get("conc", env_conc) + if raw_conc is None: + raise ValueError("Resolved AgentX recipe is missing concurrency") + conc = int(raw_conc) + except (KeyError, TypeError, ValueError) as exc: + raise ValueError("Resolved AgentX recipe has invalid topology metadata") from exc + if min(tp, pp, pcp_size, ep, conc) <= 0: + raise ValueError("Resolved AgentX recipe topology and concurrency values must be positive") + topology = { + "tp": tp, + "pp": pp, + "pcp_size": pcp_size, + "ep": ep, + "gpu_count": tp * pp * pcp_size, + "conc": conc, + "duration_seconds": int(entry.get("duration", 0) or 0), + "recipe_fingerprint": str(entry.get("recipe-fingerprint") or ""), + } + return { + **resolved_payload, + "benchmark": resolved_benchmark, + "entry": entry, + "topology": topology, + "inferencex_path": str(root), + } + + +def resolve_native_recipe( + benchmark: MutableMapping[str, Any], + *, + inferencex_path: str | Path, + expected_gpu_count: int, + outer_image: str | None = None, +) -> dict[str, Any]: + """Resolve one native AgentX point through the installed Magpie API. + + Hyperloom must know the concrete recipe topology *before* its scheduler + accounts for the serving run. Leaving resolution until Magpie starts is + unsafe: InferenceX may select TP x PP x PCP greater than Hyperloom's outer + ``--tp`` and the result would then be graded with aggregate throughput as + though it came from one GPU. + + The resolved Magpie fields are persisted back into ``benchmark``. Unknown + Hyperloom metadata (notably ``workload_spec``) is retained. + """ + resolved_payload = preview_native_recipe( + benchmark, + inferencex_path=inferencex_path, + ) + resolved_benchmark = resolved_payload["benchmark"] + entry = resolved_payload["entry"] + topology = resolved_payload["topology"] + tp = int(topology["tp"]) + pp = int(topology["pp"]) + pcp_size = int(topology["pcp_size"]) + ep = int(topology["ep"]) + gpu_count = int(topology["gpu_count"]) + recipe_fingerprint = str(topology.get("recipe_fingerprint") or "").strip() + expected_fingerprint = os.environ.get("HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", "").strip() + if expected_fingerprint and recipe_fingerprint != expected_fingerprint: + raise ValueError( + "Native AgentX recipe changed after session finalization: " + f"expected fingerprint {expected_fingerprint!r}, resolved " + f"{recipe_fingerprint!r}. Refusing to mix recipe arms in one session." + ) + if int(expected_gpu_count) != gpu_count: + raise ValueError( + "Native AgentX requires Hyperloom --tp to equal the recipe's total " + f"GPU count (TP x PP x PCP): requested {expected_gpu_count}, " + f"resolved {tp} x {pp} x {pcp_size} = {gpu_count}. This release " + "does not support topology-changing AgentX rounds." + ) + + resolved_envs = resolved_benchmark.get("envs") + if not isinstance(resolved_envs, dict): + raise RuntimeError("Magpie AgentX recipe resolver returned invalid envs") + raw_mask = str(resolved_envs.get("ROCR_VISIBLE_DEVICES") or "").strip() + mask = [part.strip() for part in raw_mask.split(",") if part.strip()] + expected_mask = [str(index) for index in range(gpu_count)] + if mask != expected_mask: + raise ValueError( + "Pinned InferenceX AgentX launchers overwrite logical " + "HIP_VISIBLE_DEVICES with ROCR_VISIBLE_DEVICES. Until that upstream " + f"bug is fixed, the mask must be exactly {','.join(expected_mask)!r}; " + f"got {raw_mask!r}." + ) + # The zero-based boundary is authoritative. Do not let Magpie's idle-GPU + # probe replace it with nonzero physical IDs that the pinned launcher would + # incorrectly reuse as logical HIP indices. + raw_selection = resolved_benchmark.get("gpu_selection") + selection = dict(raw_selection) if isinstance(raw_selection, dict) else {} + selection["auto"] = False + resolved_benchmark["gpu_selection"] = selection + + image = str(resolved_benchmark.get("docker_image") or entry.get("image") or "").strip() + declared_outer_image = str( + outer_image if outer_image is not None else os.environ.get("HYPERLOOM_IMAGE", "") + ).strip() + if not declared_outer_image: + raise ValueError( + "Native AgentX local mode requires HYPERLOOM_IMAGE to name the " + f"already-running outer image; the resolved recipe expects {image!r}." + ) + if declared_outer_image != image: + raise ValueError( + "Native AgentX outer image does not match the resolved recipe: " + f"HYPERLOOM_IMAGE={declared_outer_image!r}, recipe image={image!r}" + ) + + for key, value in resolved_benchmark.items(): + benchmark[key] = value + + workload = benchmark.get("workload_spec") + if not isinstance(workload, dict): + workload = {} + benchmark["workload_spec"] = workload + topology = { + "tp": tp, + "pp": pp, + "pcp_size": pcp_size, + "ep": ep, + "gpu_count": gpu_count, + "conc": int(topology["conc"]), + "duration_seconds": int(topology["duration_seconds"]), + "recipe_fingerprint": str(entry.get("recipe-fingerprint") or ""), + } + workload["resolved_topology"] = topology + workload["recipe"] = { + "name": str(resolved_payload.get("recipe") or ""), + "config_file": str(resolved_payload.get("config_file") or ""), + "recipe_fingerprint": str(entry.get("recipe-fingerprint") or ""), + "image": image, + "runner": str(entry.get("runner") or ""), + "model": str(entry.get("model") or ""), + "model_prefix": str(entry.get("model-prefix") or ""), + "framework": str(entry.get("framework") or ""), + "precision": str(entry.get("precision") or ""), + "concurrency": int(topology["conc"]), + "duration_seconds": int(topology["duration_seconds"]), + "launcher": str(resolved_benchmark.get("benchmark_script") or ""), + } + workload["outer_image"] = declared_outer_image + # This proves config equality only. Hyperloom cannot introspect the digest + # of the already-running outer container from inside a generic pod. + workload["outer_image_config_pinned"] = bool(declared_outer_image) + expected_execution_fingerprint = os.environ.get("HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", "").strip() + if expected_execution_fingerprint: + execution = native_execution_identity( + inferencex_path=inferencex_path, + benchmark_script=str(resolved_benchmark.get("benchmark_script") or ""), + config_file=str(resolved_payload.get("config_file") or ""), + resolved_benchmark=resolved_benchmark, + expected_ref=os.environ.get("INFERENCEX_REF", ""), + magpie_execution=resolved_payload.get("magpie_execution"), + expected_magpie_ref=os.environ.get("MAGPIE_REF", ""), + ) + actual_execution_fingerprint = execution["static_execution_fingerprint"] + if actual_execution_fingerprint != expected_execution_fingerprint: + raise ValueError( + "Native AgentX execution inputs changed after session finalization: " + f"expected {expected_execution_fingerprint!r}, resolved " + f"{actual_execution_fingerprint!r}." + ) + from .identity import has_launch_contract + + optimizing = has_launch_contract(resolved_benchmark) + pinned_field = "workload_fingerprint" if optimizing else "execution_fingerprint" + pin_name = ( + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT" + if optimizing + else "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT" + ) + materialized_fingerprint = execution[pinned_field] + accepted_materialized_fingerprint = os.environ.get(pin_name, "").strip() + if accepted_materialized_fingerprint and materialized_fingerprint != accepted_materialized_fingerprint: + raise ValueError( + "Native AgentX fixed benchmark workload changed after its first " + "materialization: expected fingerprint " + f"{accepted_materialized_fingerprint!r}, resolved " + f"{materialized_fingerprint!r}." + ) + os.environ[pin_name] = materialized_fingerprint + workload["execution"] = execution + return topology + + +__all__ = [ + "AGENTX_REQUIRED_RUNTIME_PIN_NAMES", + "AGENTX_RUNTIME_PIN_NAMES", + "native_agentx_enabled", + "native_execution_identity", + "native_launch_environment_identity", + "native_runtime_search_paths", + "preview_native_recipe", + "resolve_native_launcher", + "resolve_native_recipe", + "scrub_native_agentx_ambient_env", + "validate_native_recipe_launcher", + "validate_native_checkout_path", + "validate_native_launcher_name", +] diff --git a/src/hyperloom/inference_optimizer/agentx/runtime.py b/src/hyperloom/inference_optimizer/agentx/runtime.py index e4caaf5465..2cd8b20771 100644 --- a/src/hyperloom/inference_optimizer/agentx/runtime.py +++ b/src/hyperloom/inference_optimizer/agentx/runtime.py @@ -6,7 +6,9 @@ from __future__ import annotations from pathlib import Path -from typing import Mapping +import re +import subprocess +from typing import Mapping, MutableMapping import yaml @@ -17,35 +19,392 @@ _PREFLIGHTED_BINS: dict[str, bool] = {} -def maybe_prepare_agentx( +def _profile_compatibility_checkout( + bench: Mapping[str, object], *, + config_path: str | Path, + explicit_inferencex_path: str, env: Mapping[str, str], +) -> Path: + """Validate the one non-native leg allowed in a pinned AgentX session. + + ``ProfileExecutor`` deliberately replaces native Magpie AgentX with the + phase-gated ``aiperf_client.sh`` compatibility harness. That exception + must be narrow: it is diagnostic-only, runs from the disposable checkout + beside the materialized profile config, and stays at the session's pinned + InferenceX commit. + """ + if str(bench.get("benchmark_script") or "") != "aiperf_client.sh": + raise ValueError( + "Pinned native AgentX session requires benchmark.agentx to remain " + "enabled outside the verified profiler compatibility path" + ) + raw_workload = bench.get("workload_spec") + workload = raw_workload if isinstance(raw_workload, Mapping) else {} + if str(workload.get("harness") or "") != "hyperloom-profiler-compat": + raise ValueError("Pinned AgentX profiler compatibility config is missing its diagnostic harness identity") + raw_envs = bench.get("envs") + bench_envs = raw_envs if isinstance(raw_envs, Mapping) else {} + raw_profiler = bench.get("profiler") + profiler = raw_profiler if isinstance(raw_profiler, Mapping) else {} + raw_torch = profiler.get("torch_profiler") + torch_profiler = raw_torch if isinstance(raw_torch, Mapping) else {} + + def enabled(value: object) -> bool: + if isinstance(value, str): + return value.strip().lower() in { + "1", + "true", + "yes", + "on", + "enable", + "enabled", + } + return bool(value) + + if not (enabled(bench_envs.get("PROFILE")) or enabled(torch_profiler.get("enabled"))): + raise ValueError("Pinned AgentX profiler compatibility config has no active torch profile marker") + + required_pins = ( + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", + "HYPERLOOM_AGENTX_GPU_COUNT", + "MAGPIE_REF", + "INFERENCEX_REF", + ) + missing = [name for name in required_pins if not str(env.get(name) or "").strip()] + if missing: + raise ValueError( + "Pinned AgentX profiler compatibility path is missing session authority: " + ", ".join(missing) + ) + expected_ref = str(env.get("INFERENCEX_REF") or "").strip().lower() + if not re.fullmatch(r"[0-9a-f]{40}", expected_ref): + raise ValueError("Pinned AgentX profiler compatibility path requires a full 40-character INFERENCEX_REF") + + configured = str(bench.get("inferencex_path") or "").strip() + if not configured or not explicit_inferencex_path: + raise ValueError("Pinned AgentX profiler compatibility path requires an explicit isolated InferenceX checkout") + from .native import validate_native_checkout_path + + configured_path = Path(configured).expanduser() + explicit_path = Path(explicit_inferencex_path).expanduser() + if configured_path.is_symlink() or explicit_path.is_symlink(): + raise ValueError("Pinned AgentX profiler compatibility checkout must not be a symbolic link") + try: + resolved = validate_native_checkout_path(configured).resolve(strict=True) + explicit_resolved = validate_native_checkout_path(explicit_inferencex_path).resolve(strict=True) + config_parent = Path(config_path).expanduser().resolve(strict=True).parent + except (OSError, RuntimeError) as exc: + raise ValueError("Pinned AgentX profiler compatibility checkout is not resolvable") from exc + if ( + resolved != explicit_resolved + or resolved.name != ".agentx-profile-inferencex" + or resolved.parent != config_parent + ): + raise ValueError( + "Pinned AgentX profiler compatibility checkout is not the isolated tree beside its materialized config" + ) + try: + top_proc = subprocess.run( + ["git", "-C", str(resolved), "rev-parse", "--show-toplevel"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + head_proc = subprocess.run( + ["git", "-C", str(resolved), "rev-parse", "HEAD"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + raise ValueError("Cannot verify the pinned AgentX profiler compatibility checkout") from exc + try: + top = Path((top_proc.stdout or "").strip()).resolve(strict=True) + except (OSError, RuntimeError): + top = Path() + head = (head_proc.stdout or "").strip().lower() + if top_proc.returncode != 0 or head_proc.returncode != 0 or top != resolved or head != expected_ref: + raise ValueError( + "Pinned AgentX profiler compatibility checkout does not match the session's isolated InferenceX revision" + ) + return resolved + + +def _scrub_native_agentx_ambient_env(env: MutableMapping[str, str]) -> None: + """Remove replay controls that Magpie's local runner would inherit. + + Magpie deliberately starts its local launcher from ``os.environ.copy()``. + Native AgentX replay controls must instead come from the resolved YAML or + the pinned InferenceX launcher; otherwise an unrelated login-shell export + can change the workload without changing its recipe fingerprint. + """ + from .native import scrub_native_agentx_ambient_env + + scrub_native_agentx_ambient_env(env) + + +def maybe_prepare_agentx( + *, + env: MutableMapping[str, str], inferencex_path: str, config_path: str | Path, + allow_profile_compat: bool = False, ) -> bool: - """Deploy the AgentX client + capability-preflight aiperf for a run.""" + """Prepare native Magpie AgentX or the legacy client, including diagnostics.""" + expected_recipe = str(env.get("HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT") or "").strip() + expected_execution = str(env.get("HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT") or "").strip() + expected_materialized_execution = str( + env.get("HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT") or "" + ).strip() + expected_gpu_count = str(env.get("HYPERLOOM_AGENTX_GPU_COUNT") or "").strip() + # These pins are written only after the CLI has selected and accepted one + # native recipe. From that point on they are the authority at the process + # boundary: editing the materialized YAML must never turn a native run into + # a generic Magpie run or replace the accepted identity with self-asserted + # YAML metadata. + pinned_native_session = any((expected_recipe, expected_execution, expected_gpu_count)) try: - bench = (yaml.safe_load(Path(config_path).read_text(encoding="utf-8")) or {}).get("benchmark", {}) or {} - except Exception: # noqa: BLE001 — config unreadable: let Magpie surface it + config = yaml.safe_load(Path(config_path).read_text(encoding="utf-8")) or {} + bench = config.get("benchmark", {}) if isinstance(config, dict) else {} + bench = bench if isinstance(bench, dict) else {} + except (OSError, UnicodeDecodeError, yaml.YAMLError, TypeError, ValueError) as exc: + if pinned_native_session: + raise ValueError("Pinned native AgentX config is unreadable at the launch boundary") from exc bench = {} + + from .native import ( + native_agentx_enabled, + native_execution_identity, + resolve_native_launcher, + ) + + native = native_agentx_enabled(bench.get("agentx")) + from .identity import has_launch_contract + + optimizing = has_launch_contract(bench) + fingerprint_field = "workload_fingerprint" if optimizing else "execution_fingerprint" + materialized_pin_name = "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT" + if optimizing: + materialized_pin_name = "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT" + expected_materialized_execution = str(env.get(materialized_pin_name) or "").strip() + elif env.get("HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT") and native: + raise ValueError("Native AgentX optimization session is missing its launch contract") + explicit_path = str(inferencex_path or "").strip() + configured_path = str(bench.get("inferencex_path") or "").strip() + runtime_path = str(env.get("INFERENCEX_PATH") or "").strip() + effective_inferencex_path = ( + explicit_path or runtime_path or configured_path if native else explicit_path or configured_path or runtime_path + ) + pinned_profile_compat = False + if pinned_native_session and not native: + if not allow_profile_compat: + raise ValueError("Pinned native AgentX session requires benchmark.agentx to remain enabled") + _profile_compatibility_checkout( + bench, + config_path=config_path, + explicit_inferencex_path=explicit_path, + env=env, + ) + pinned_profile_compat = True + if native: + from .managed import server_spec + + torch_profile = ((bench.get("profiler") or {}).get("torch_profiler") or {}).get("enabled") is True + if server_spec(bench) is not None and torch_profile and not allow_profile_compat: + raise ValueError("Managed AgentX diagnostics require the explicit profile execution path") + if not effective_inferencex_path: + raise ValueError( + "Native AgentX requires an InferenceX checkout via benchmark.inferencex_path or INFERENCEX_PATH" + ) + resolve_native_launcher( + inferencex_path=effective_inferencex_path, + benchmark_script=str(bench.get("benchmark_script") or ""), + ) + # Normalize the exact environment Magpie will inherit before identity + # validation. Its local runner copies this mapping and then overlays + # benchmark.envs, so hashing the pre-scrub login shell would either + # miss the real child value or report false drift for blocked hooks. + _scrub_native_agentx_ambient_env(env) + raw_envs = bench.get("envs") + bench_envs = raw_envs if isinstance(raw_envs, dict) else {} + local_model_path = str(bench_envs.get("MODEL_PATH") or "").strip() + if local_model_path: + env["MODEL_PATH"] = local_model_path + else: + env.pop("MODEL_PATH", None) + raw_workload = bench.get("workload_spec") + workload = raw_workload if isinstance(raw_workload, dict) else {} + raw_execution = workload.get("execution") + execution = raw_execution if isinstance(raw_execution, dict) else {} + yaml_execution = str(execution.get("static_execution_fingerprint") or "").strip() + if pinned_native_session: + missing_pins = [ + name + for name, value in ( + ("HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", expected_recipe), + ("HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", expected_execution), + ( + materialized_pin_name, + expected_materialized_execution, + ), + ("HYPERLOOM_AGENTX_GPU_COUNT", expected_gpu_count), + ("MAGPIE_REF", str(env.get("MAGPIE_REF") or "").strip()), + ("INFERENCEX_REF", str(env.get("INFERENCEX_REF") or "").strip()), + ) + if not value + ] + if missing_pins: + raise ValueError("Pinned native AgentX session is missing launch authority: " + ", ".join(missing_pins)) + if not yaml_execution: + raise ValueError("Pinned native AgentX config is missing workload_spec.execution") + if yaml_execution != expected_execution: + raise ValueError( + "Native AgentX YAML execution fingerprint differs from the " + f"session pin: {yaml_execution!r} != {expected_execution!r}" + ) + yaml_materialized_execution = str(execution.get(fingerprint_field) or "").strip() + if yaml_materialized_execution != expected_materialized_execution: + raise ValueError( + "Native AgentX YAML materialized execution fingerprint " + "differs from the session pin: " + f"{yaml_materialized_execution!r} != " + f"{expected_materialized_execution!r}" + ) + + if not pinned_native_session: + # Standalone callers may use this helper as a lightweight native + # launcher/path validator before the Hyperloom CLI has finalized a + # session. There is no accepted identity to compare yet; the + # strict re-resolution below is reserved for a pinned session. + return True + + # Re-resolve at the last possible boundary. The recipe fingerprint is + # intentionally concurrency-independent, so compare the complete + # persisted topology and recipe identity as well as both fingerprints. + from .native import preview_native_recipe + + preview = preview_native_recipe( + dict(bench), + inferencex_path=effective_inferencex_path, + ) + topology = preview.get("topology") + topology = topology if isinstance(topology, dict) else {} + entry = preview.get("entry") + entry = entry if isinstance(entry, dict) else {} + resolved_benchmark = preview.get("benchmark") + resolved_benchmark = resolved_benchmark if isinstance(resolved_benchmark, dict) else {} + config_file = str(preview.get("config_file") or "") + current_recipe = { + "name": str(preview.get("recipe") or ""), + "config_file": config_file, + "recipe_fingerprint": str(topology.get("recipe_fingerprint") or ""), + "image": str(resolved_benchmark.get("docker_image") or entry.get("image") or ""), + "runner": str(entry.get("runner") or ""), + "model": str(entry.get("model") or ""), + "model_prefix": str(entry.get("model-prefix") or ""), + "framework": str(entry.get("framework") or ""), + "precision": str(entry.get("precision") or ""), + "concurrency": int(topology.get("conc") or 0), + "duration_seconds": int(topology.get("duration_seconds") or 0), + "launcher": str(resolved_benchmark.get("benchmark_script") or ""), + } + current_topology = { + key: topology.get(key) + for key in ( + "tp", + "pp", + "pcp_size", + "ep", + "gpu_count", + "conc", + "duration_seconds", + "recipe_fingerprint", + ) + } + if pinned_native_session: + if current_recipe["recipe_fingerprint"] != expected_recipe: + raise ValueError( + "Native AgentX recipe changed after materialization: " + f"expected {expected_recipe!r}, resolved " + f"{current_recipe['recipe_fingerprint']!r}" + ) + try: + pinned_gpu_count = int(expected_gpu_count) + except ValueError as exc: + raise ValueError("HYPERLOOM_AGENTX_GPU_COUNT must be a positive integer") from exc + if pinned_gpu_count <= 0 or current_topology["gpu_count"] != pinned_gpu_count: + raise ValueError( + "Native AgentX physical topology changed after materialization: " + f"expected {pinned_gpu_count} GPUs, resolved " + f"{current_topology['gpu_count']!r}" + ) + saved_topology = workload.get("resolved_topology") + saved_recipe = workload.get("recipe") + if not isinstance(saved_topology, dict) or any( + saved_topology.get(key) != value for key, value in current_topology.items() + ): + raise ValueError("Native AgentX resolved topology changed after materialization") + if not isinstance(saved_recipe, dict) or any( + saved_recipe.get(key) != value for key, value in current_recipe.items() + ): + raise ValueError("Native AgentX recipe identity changed after materialization") + + current_execution = native_execution_identity( + inferencex_path=effective_inferencex_path, + benchmark_script=str(resolved_benchmark.get("benchmark_script") or ""), + config_file=config_file, + resolved_benchmark=resolved_benchmark, + expected_ref=str(env.get("INFERENCEX_REF") or ""), + magpie_execution=preview.get("magpie_execution"), + expected_magpie_ref=str(env.get("MAGPIE_REF") or ""), + launch_env=env, + ) + if expected_execution and current_execution["static_execution_fingerprint"] != expected_execution: + raise ValueError( + "Native AgentX execution inputs changed after materialization: " + f"expected {expected_execution!r}, resolved " + f"{current_execution['static_execution_fingerprint']!r}" + ) + if expected_materialized_execution and current_execution[fingerprint_field] != expected_materialized_execution: + raise ValueError( + "Native AgentX resolved BenchmarkConfig changed after " + "materialization: expected execution fingerprint " + f"{expected_materialized_execution!r}, resolved " + f"{current_execution[fingerprint_field]!r}" + ) + if pinned_native_session and execution != current_execution: + raise ValueError("Native AgentX persisted execution identity changed after materialization") + return True + script = Path(str(bench.get("benchmark_script") or "")).name if not is_agentx_client_script(script): return False + if pinned_native_session and not pinned_profile_compat: + raise ValueError("Pinned native AgentX session cannot use an unverified generic client") + if not effective_inferencex_path: + raise ValueError("AgentX requires an InferenceX checkout via benchmark.inferencex_path or INFERENCEX_PATH") from .deploy import deploy_agentx_assets from .preflight import resolve_aiperf_bin # Deploy BEFORE preflight so the client is in place regardless of preflight memoization state. - deploy_agentx_assets(Path(inferencex_path) / "benchmarks") + + deploy_agentx_assets(Path(effective_inferencex_path) / "benchmarks") if script == MLPERF_CLIENT_SCRIPT: from .preflight import check_mlperf_harness check_mlperf_harness(env) return True aiperf_bin = resolve_aiperf_bin(env) - bench_envs = bench.get("envs") if isinstance(bench.get("envs"), dict) else {} - profiler = bench.get("profiler") if isinstance(bench.get("profiler"), dict) else {} - torch_profiler = profiler.get("torch_profiler") if isinstance(profiler.get("torch_profiler"), dict) else {} + raw_bench_envs = bench.get("envs") + bench_envs = raw_bench_envs if isinstance(raw_bench_envs, dict) else {} + raw_profiler = bench.get("profiler") + profiler = raw_profiler if isinstance(raw_profiler, dict) else {} + raw_torch_profiler = profiler.get("torch_profiler") + torch_profiler = raw_torch_profiler if isinstance(raw_torch_profiler, dict) else {} require_progress_api = str(bench_envs.get("PROFILE") or "") == "1" or bool(torch_profiler.get("enabled")) preflight_key = aiperf_bin or "" previous_check = _PREFLIGHTED_BINS.get(preflight_key) diff --git a/src/hyperloom/inference_optimizer/assets/agentx/aiperf_client.sh b/src/hyperloom/inference_optimizer/assets/agentx/aiperf_client.sh index 877db67e24..255ff05f71 100755 --- a/src/hyperloom/inference_optimizer/assets/agentx/aiperf_client.sh +++ b/src/hyperloom/inference_optimizer/assets/agentx/aiperf_client.sh @@ -279,7 +279,7 @@ _default_loader() { # apart would silently disagree the moment the derivation grows a second input. # AGENTX_CANONICAL_DATASET lets an operator declare the canonical corpus for a # model the family whitelist does not cover -- see the deviation check. -CANON_DS="${AGENTX_CANONICAL_DATASET:-$(_default_loader "$MODEL")}" +CANON_DS="${AGENTX_CANONICAL_DATASET:-$(_default_loader "${AGENTX_MODEL_ID:-$MODEL}")}" DS="${AGENTX_DATASET:-${WEKA_LOADER_OVERRIDE:-$CANON_DS}}" if [ -z "${AGENTX_DATASET:-}${WEKA_LOADER_OVERRIDE:-}" ]; then case "$DS" in diff --git a/src/hyperloom/inference_optimizer/assets/install.sh b/src/hyperloom/inference_optimizer/assets/install.sh index 2d57c01267..ad51ec7070 100755 --- a/src/hyperloom/inference_optimizer/assets/install.sh +++ b/src/hyperloom/inference_optimizer/assets/install.sh @@ -183,12 +183,11 @@ EOF } MAGPIE_REPO="${MAGPIE_REPO:-https://github.com/AMD-AGI/Magpie.git}" -# Pin Magpie to a release commit/tag instead of the default branch. Operators can -# re-pin with MAGPIE_REF=. Must stay at or above e6833b8183c6c41adf6038252337550876ca0433 -# (Magpie v0.2.0), which copies benchmark scripts via ``_copy_benchmark_script_atomic``. -# ``ensure_magpie()`` skips pip when ``import Magpie`` already succeeds, so a pre-existing -# tree on disk is NOT upgraded to this ref — only fresh installs and explicit reinstalls are. -MAGPIE_REF="${MAGPIE_REF:-e6833b8183c6c41adf6038252337550876ca0433}" +# Pin Magpie's native AgentX launch contract, eval fixes, and optional SGLang shape discovery. +# Operators can re-pin with MAGPIE_REF=. Generic benchmarks keep the +# importability contract; native AgentX additionally requires the audited +# source identity and recipe-fingerprint capabilities. +MAGPIE_REF="${MAGPIE_REF:-d80eb4d3dad7fabe01ce81d049e2983adf2c86dd}" MAGPIE_PACKAGE_SPEC="${MAGPIE_PACKAGE_SPEC:-magpie-eval @ git+${MAGPIE_REPO}@${MAGPIE_REF}}" # aiperf (SemiAnalysis AgentX benchmark client) — pinned to an immutable commit @@ -219,14 +218,10 @@ if [ -n "${MAGPIE_PATH:-}" ]; then fi MAGPIE_PATH="${MAGPIE_PATH:-${_open_source_root}/Magpie}" INFERENCEX_REPO="${INFERENCEX_REPO:-https://github.com/SemiAnalysisAI/InferenceX.git}" -# Pin InferenceX to a current default-branch HEAD *commit SHA* so the -# per-install clone is reproducible (same rationale as MAGPIE_REF). Operators -# can re-pin with INFERENCEX_REF=. -# Re-pinned to the leaderboard's current head so AgentX replays the same -# scenario, corpus generation and warmup contract the published rows were -# produced with. Keep AIPERF_REF above in lockstep (it is this commit's -# utils/aiperf submodule); re-sync when the corpus generation changes. -INFERENCEX_REF="${INFERENCEX_REF:-3d5581562f643f9bdeb8410cd924e2c70906c966}" +# Magpie owns AgentX serving; this upstream pin supplies recipes and the client. +# Keep AIPERF_REF in lockstep with inferencex-e2e/utils/aiperf. +# Existing sessions retain their accepted immutable dependency revisions. +INFERENCEX_REF="${INFERENCEX_REF:-408c015be4b22d14c69518643609669405507077}" _INFERENCEX_SHA="$(_resolve_ref_sha "$INFERENCEX_REPO" "$INFERENCEX_REF")" INFERENCEX_DEFAULT_DIR="${INFERENCEX_DEFAULT_DIR:-${_open_source_root}/InferenceX@${_INFERENCEX_SHA}}" @@ -272,7 +267,7 @@ Options: Env overrides: REPO_ROOT, MAGPIE_REPO, MAGPIE_REF (commit SHA / tag / branch the Magpie package is pinned to; - default is a commit that already copies benchmark scripts atomically), + default provides native InferenceX AgentX and atomic script copies), MAGPIE_PACKAGE_SPEC, MAGPIE_PATH, INFERENCEX_REPO, INFERENCEX_REF (commit SHA / tag / branch the InferenceX clone is pinned to; default is a current upstream HEAD SHA), @@ -1517,11 +1512,12 @@ for spec in specs: # root after import. ensure_magpie() { log "ensuring Magpie package ${MAGPIE_PACKAGE_SPEC}" + local health_probe='from hyperloom.common.agentx_mode import native_agentx_session; from hyperloom.inference_optimizer.magpie_health import magpie_health_code; exec(magpie_health_code(native_agentx=native_agentx_session()))' if [ "$CHECK_ONLY" -eq 1 ]; then - if "$PYTHON" -c "import Magpie" >/dev/null 2>&1; then - log "Magpie importable" + if "$PYTHON" -c "$health_probe" "$MAGPIE_REF" >/dev/null 2>&1; then + log "Magpie package healthy" else - warn "Magpie not importable (check-only mode, skipping pip install)" + warn "Magpie package unavailable or incompatible (check-only mode, skipping pip install)" fi return 0 fi @@ -1530,11 +1526,16 @@ ensure_magpie() { return 0 fi if [ "$DRY_RUN" -eq 0 ]; then - if "$PYTHON" -c "import Magpie" >/dev/null 2>&1; then - log "Magpie already importable; skipping pip install" + if "$PYTHON" -c "$health_probe" "$MAGPIE_REF" >/dev/null 2>&1; then + log "Magpie package healthy; skipping pip install" else "$PYTHON" -m pip install --quiet "${PIP_EXTRA[@]}" "$MAGPIE_PACKAGE_SPEC" - "$PYTHON" -c "import Magpie" >/dev/null + if ! "$PYTHON" -c "$health_probe" "$MAGPIE_REF" >/dev/null 2>&1; then + # pip keeps an installed VCS commit even when its files were patched. + # Reinstall only Magpie; the first install already resolved its deps. + "$PYTHON" -m pip install --quiet --force-reinstall --no-deps "${PIP_EXTRA[@]}" "$MAGPIE_PACKAGE_SPEC" + "$PYTHON" -c "$health_probe" "$MAGPIE_REF" >/dev/null + fi log "Magpie installed OK from ${MAGPIE_PACKAGE_SPEC}" fi local installed_root @@ -1727,86 +1728,6 @@ ensure_aiperf() { fi } -# --- 2b. Atomic-write patch for Magpie._prepare_benchmark_scripts (compat patches only) --- -# Two gaps between the pinned Magpie/InferenceX revision and what Hyperloom -# needs: SGLang custom-tokenizer trust gating for MAGPIE_TRUST_REMOTE_CODE=1 -# (Magpie's client call sites never forward the `trust` flag upstream), and -# the redundant `--concurrent-requests` flag InferenceX's `run_lm_eval` -# rejects. Magpie is invoked as a subprocess, so monkey-patching from the -# Coordinator process does not reach it; we patch the cloned source in place -# at install time. The patcher itself is idempotent + flock-serialised + -# atomic-rename (see `_magpie_patcher.py`), so re-runs are O(1) no-ops. -# -# Override the gate via PATCH_MAGPIE=0 to skip the step entirely. -ensure_magpie_compat_patches() { - if is_falsy "${PATCH_MAGPIE:-1}"; then - log "PATCH_MAGPIE is falsy — skipping Magpie compatibility patches" - return 0 - fi - if [ "$DRY_RUN" -eq 1 ]; then - log "would apply Magpie SGLang trust + eval-concurrency compatibility patches under ${MAGPIE_PATH}" - return 0 - fi - log "applying Magpie SGLang trust + eval-concurrency compatibility patches" - # Exit-code contract (read below): 0 ok · 2 remote-trust drift · 5 eval-flag survives. - # INFERENCEX_PATH is passed explicitly: the patcher also has to scrub the - # InferenceX ``benchmarks/`` copies Magpie executes and teach - # ``benchmark_lib.sh::run_lm_eval`` to tolerate the flag. This step therefore - # MUST run after ensure_inferencex has exported INFERENCEX_PATH — see the - # call ordering at the bottom of this script. - if MAGPIE_PATH="$MAGPIE_PATH" INFERENCEX_PATH="${INFERENCEX_PATH:-}" "$PYTHON" - <<'PY' -import os, sys -from hyperloom.orchestrator.actions.executors._magpie_patcher import ( - magpie_scripts_patch_status, -) -status = magpie_scripts_patch_status( - os.environ["MAGPIE_PATH"], - os.environ.get("INFERENCEX_PATH") or None, -) -print(f"_magpie_patcher: remote_trust_ok={status.remote_trust_ok} " - f"eval_flag_ok={status.eval_flag_ok}", - file=sys.stderr) -if status.ok: - sys.exit(0) -if not status.remote_trust_ok: - sys.exit(2) -# eval_flag_ok is False ONLY when a live `run_eval --concurrent-requests` -# survives in a caller script AND InferenceX's run_lm_eval would reject it -# (a defence-in-depth patch that merely could not be applied, with no live -# flag, is NOT counted as a failure -- install-time now matches the run-time -# ensure_eval_concurrency_compat judgement). This is the genuinely fatal case: -# every RUN_EVAL=true baseline aborts on 'Unknown parameter'. Distinct exit so -# install can name the failure mode. -if not status.eval_flag_ok: - sys.exit(5) -sys.exit(1) -PY - then - log "Magpie compatibility patches OK" - else - rc=$? - if [ "$rc" -eq 2 ]; then - warn "Magpie SGLang remote trust patch did not apply. If MAGPIE_TRUST_REMOTE_CODE=1 is required for custom-code models (for example Kimi/Qwen tokenizer paths), remote benchmark clients may still fail to pass trust; review _magpie_patcher.py or set PATCH_MAGPIE=0 only if this is intentional." - elif [ "$rc" -eq 5 ]; then - # Fail-loud by default: a surviving --concurrent-requests aborts EVERY - # RUN_EVAL=true baseline in InferenceX's run_lm_eval arg parser, no - # results*.json is written, and the baseline accuracy gate then stops the - # whole run with `baseline_accuracy_failed`. There is no salvage: the - # executor deliberately does NOT fall back to RUN_EVAL=false for a genuine - # baseline (a throughput-only baseline cannot satisfy the accuracy gate). - # Set MAGPIE_EVAL_FLAG_STRICT=0 only when accuracy eval is genuinely - # not required for this deployment. - if is_falsy "${MAGPIE_EVAL_FLAG_STRICT:-1}"; then - warn "Magpie redundant --concurrent-requests eval flag could not be stripped from a generic benchmark script (unrecognised run_eval line); MAGPIE_EVAL_FLAG_STRICT=${MAGPIE_EVAL_FLAG_STRICT:-} (falsy), continuing anyway — RUN_EVAL=true baselines will abort on InferenceX's 'Unknown parameter: --concurrent-requests'." - else - die "Magpie redundant --concurrent-requests eval flag could not be stripped from a generic benchmark script (unrecognised run_eval line), and InferenceX's run_lm_eval could not be taught to tolerate it. Every RUN_EVAL=true baseline will abort with 'Unknown parameter: --concurrent-requests' and the run will stop with baseline_accuracy_failed. Concurrency must flow via EVAL_CONCURRENT_REQUESTS (fallback CONC), not the flag — fix the script's run_eval line or review _magpie_patcher.py. Set MAGPIE_EVAL_FLAG_STRICT=0 to downgrade to a warning if accuracy eval is not required." - fi - else - die "Magpie compatibility patch step failed (rc=$rc): the patcher process exited before reporting remote_trust_ok/eval_flag_ok. Check MAGPIE_PATH/INFERENCEX_PATH and review _magpie_patcher.py." - fi - fi -} - # --- 3. InferenceX checkout: fresh clone from upstream --- # # Previously this function scanned a list of shared-filesystem candidates @@ -1834,14 +1755,44 @@ PY # is reproducible. We still record the resolved commit into the session # manifest (see manifest.py / _describe_dep) so runs stay traceable even # when an operator overrides INFERENCEX_REF. +normalize_inferencex_project() { + if [ ! -f "$INFERENCEX_PATH/benchmarks/benchmark_lib.sh" ] && \ + [ -f "$INFERENCEX_PATH/inferencex-e2e/benchmarks/benchmark_lib.sh" ]; then + INFERENCEX_PATH="$INFERENCEX_PATH/inferencex-e2e" + fi + export INFERENCEX_PATH +} + +ensure_inferencex_aiperf_submodule() { + normalize_inferencex_project + if [ "$CHECK_ONLY" -eq 1 ]; then + if [ ! -f "$INFERENCEX_PATH/utils/aiperf/pyproject.toml" ]; then + warn "InferenceX utils/aiperf submodule is not initialized at ${INFERENCEX_PATH}" + fi + return 0 + fi + run git -C "$INFERENCEX_PATH" submodule sync -- utils/aiperf || return 1 + if ! run git -C "$INFERENCEX_PATH" submodule update --init --depth 1 -- utils/aiperf; then + warn "shallow utils/aiperf submodule update failed; retrying full pinned gitlink" + run git -C "$INFERENCEX_PATH" submodule update --init -- utils/aiperf || return 1 + fi + if [ "$DRY_RUN" -eq 0 ] && [ ! -f "$INFERENCEX_PATH/utils/aiperf/pyproject.toml" ]; then + warn "InferenceX utils/aiperf submodule did not materialize at ${INFERENCEX_PATH}" + return 1 + fi + return 0 +} + ensure_inferencex() { if [ -n "${INFERENCEX_PATH:-}" ] && [ -d "$INFERENCEX_PATH" ]; then - log "INFERENCEX_PATH = $INFERENCEX_PATH (preserved from env; skipping fresh clone)" + normalize_inferencex_project + log "INFERENCEX_PATH = $INFERENCEX_PATH (preserved from env)" export INFERENCEX_PATH return 0 fi INFERENCEX_PATH="$INFERENCEX_DEFAULT_DIR" if [ -d "$INFERENCEX_PATH/.git" ] || [ -d "$INFERENCEX_PATH/benchmarks" ]; then + normalize_inferencex_project log "InferenceX already cloned at ${INFERENCEX_PATH}; preserving existing checkout" export INFERENCEX_PATH return 0 @@ -1863,6 +1814,10 @@ ensure_inferencex() { return 0 fi export INFERENCEX_PATH + if ! ensure_inferencex_aiperf_submodule; then + warn "InferenceX clone succeeded but its pinned utils/aiperf submodule is unavailable" + return 1 + fi log "InferenceX cloned at ${INFERENCEX_PATH} (pinned ${INFERENCEX_REF})" } @@ -2173,21 +2128,11 @@ acquire_install_lock # ALL whitespace would wrongly collapse "by pass" -> "bypass" and diverge. HYPERLOOM_BENCHMARK_BACKEND_LC="$(printf '%s' "${HYPERLOOM_BENCHMARK_BACKEND:-}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr '[:upper:]' '[:lower:]')" if [ "$HYPERLOOM_BENCHMARK_BACKEND_LC" = "bypass" ]; then - log "benchmark backend is bypass; skipping ensure_magpie + ensure_magpie_compat_patches" + log "benchmark backend is bypass; skipping ensure_magpie" else ensure_magpie fi ensure_inferencex -# Ordering matters: the Magpie script patch also scrubs the redundant -# `--concurrent-requests` eval flag from the InferenceX `benchmarks/` copies -# Magpie actually executes, and teaches `benchmark_lib.sh::run_lm_eval` to -# tolerate it. Both need $INFERENCEX_PATH, which only ensure_inferencex exports -# — running the patch before it silently skipped those targets and left -# RUN_EVAL=true baselines aborting on 'Unknown parameter'. -if [ "$HYPERLOOM_BENCHMARK_BACKEND_LC" != "bypass" ]; then - ensure_magpie_compat_patches -fi - # aiperf (AgentX client) installs whenever this build ships the AgentX assets. # # It used to be gated on INSTALL_AIPERF / HYPERLOOM_AGENTX being truthy HERE, in diff --git a/src/hyperloom/inference_optimizer/baseline_comparison/research_hints.py b/src/hyperloom/inference_optimizer/baseline_comparison/research_hints.py index 5e9d72a1de..6ae88b7b0c 100644 --- a/src/hyperloom/inference_optimizer/baseline_comparison/research_hints.py +++ b/src/hyperloom/inference_optimizer/baseline_comparison/research_hints.py @@ -357,7 +357,11 @@ def gap_for_state(target: dict[str, Any] | None, state: Any) -> dict[str, Any] | tput = best.get("total_throughput" if agentx else "tput") tput = to_float(tput) if agentx else tput tpot = None if agentx else best.get("tpot_mean_ms") - tp = to_float(getattr(state, "tp", None)) if agentx else int(getattr(state, "tp", 0) or 0) + tp = ( + to_float(best.get("agentx_gpu_count")) or to_float(getattr(state, "tp", None)) + if agentx + else int(getattr(state, "tp", 0) or 0) + ) return gap_analysis( target, our_tput_per_gpu=float(tput) / tp if isinstance(tput, (int, float)) and tput > 0 and tp and tp > 0 else None, diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/v6.py b/src/hyperloom/inference_optimizer/breakdown/collectors/v6.py index 4e09208d05..bc91608ae5 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/v6.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/v6.py @@ -12,8 +12,10 @@ from pathlib import Path from typing import Any +from hyperloom.common.agentx_mode import native_agentx_session from hyperloom.common.env_safety import redact_secret_env_values +from ...performance_display import throughput_fields from ...session.sbd_v6 import read_timeline_events from ..recorder.baseline_event import anchoring_eval_from_timeline from ..session_facts import architecture_block, grading_block, workload_signature @@ -386,7 +388,10 @@ def _graded_axes(recorded: Any) -> dict[str, Any]: from hyperloom.common.perf_metric import GRADED_AXIS_KEYS source = _mapping(recorded) - return {key: _optional_float(source.get(key)) for key in GRADED_AXIS_KEYS} + axes = {key: _optional_float(source.get(key)) for key in GRADED_AXIS_KEYS} + if "agentx_gpu_count" in source: + axes["agentx_gpu_count"] = _optional_float(source["agentx_gpu_count"]) + return axes def _baseline_from_timeline(timeline: list[dict[str, Any]], source_ids: list[str] | None = None) -> dict[str, Any]: @@ -439,6 +444,7 @@ def _baseline_from_timeline(timeline: list[dict[str, Any]], source_ids: list[str submission_valid = measurement.get("submission_valid") return { **{field: _optional_float(measurement.get(field)) for field in _BASELINE_OUTCOME_FIELDS}, + **{field: measurement[field] for field in ("throughput_tok_s", "throughput_unit") if field in measurement}, "perf": _graded_axes(measurement.get("perf")), # Published beside the axes because it qualifies them: upstream rejecting the round makes every figure in # ``perf`` a measurement of something it would not accept as a submission. Tri-state, so a session recorded @@ -670,7 +676,12 @@ def collect_v6_outcome( "baseline": baseline, "anchoring_eval": anchoring_eval_from_timeline(timeline), "final": { - "throughput_tok_s_per_gpu": _optional_float(recipe.get("throughput")), + **throughput_fields( + _optional_float(recipe.get("throughput")), + state.get("framework"), + native_agentx=native_agentx_session(state), + gpu_count=_mapping(validation.get("perf")).get("agentx_gpu_count"), + ), # The ledger's own settled figure rather than a second tally of it. # Absent a validation this is ``None``, not ``0.0``: nothing # measured is not the same as a measured zero. diff --git a/src/hyperloom/inference_optimizer/breakdown/exporter.py b/src/hyperloom/inference_optimizer/breakdown/exporter.py index 376dd0ae56..8ee4b351e6 100644 --- a/src/hyperloom/inference_optimizer/breakdown/exporter.py +++ b/src/hyperloom/inference_optimizer/breakdown/exporter.py @@ -465,18 +465,16 @@ def _fmt_attempt(d: dict[str, Any] | None, label: str) -> str: )[:600] return f"- **{label}** (`{ts}`): `{body}`" - from .. import framework_registry + from ..performance_display import format_session_metric framework = str(state.get("framework") or "") baseline_tput = state.get("baseline_tput") or 0.0 current_best = state.get("current_best") or {} cb_action = current_best.get("action") or "-" cb_tput = current_best.get("tput") - # Framework-aware primary metric: serving shows tok/s/GPU, scriptable xDiT shows per-image latency e2el_mean_ms - # (ms). - baseline_metric_s = framework_registry.format_primary_metric(framework, baseline_tput, precision=2) + baseline_metric_s = format_session_metric(state, baseline_tput, precision=2) cb_metric_s = ( - framework_registry.format_primary_metric(framework, cb_tput, precision=2) + format_session_metric(state, cb_tput, measurement=current_best, precision=2) if isinstance(cb_tput, (int, float)) else "-" ) diff --git a/src/hyperloom/inference_optimizer/breakdown/recorder/baseline_event.py b/src/hyperloom/inference_optimizer/breakdown/recorder/baseline_event.py index c4dcc8cbc0..94e1dcdfcf 100644 --- a/src/hyperloom/inference_optimizer/breakdown/recorder/baseline_event.py +++ b/src/hyperloom/inference_optimizer/breakdown/recorder/baseline_event.py @@ -10,6 +10,8 @@ from collections.abc import Mapping, Sequence from typing import Any +from ...performance_display import throughput_fields + from .event_fields import ( as_dict as _as_dict, as_list as _as_list, @@ -22,7 +24,6 @@ now_iso_seconds as _now_iso, worst_status as _worst_status, ) -from ... import framework_registry from .event_ids import event_id from .event_rows import group_rows, rows_for_event, sort_rows, wire_rows from .event_sink import RecordSink @@ -223,11 +224,12 @@ def _measurement(result: Mapping[str, Any], framework: str) -> dict[str, Any]: decides the throughput unit. """ return { - # Named as the V5 section names it, which is what a consumer selects on. - "throughput_tok_s_per_gpu": _float_or_none(result.get("output_throughput")), - # The name above is the serving case; an image framework measures - # img/s through the same key, so the unit must be stated. - "throughput_unit": framework_registry.throughput_unit(framework), + **throughput_fields( + _float_or_none(result.get("output_throughput")), + framework, + native_agentx=result.get("native_agentx_report") is True, + gpu_count=result.get("agentx_gpu_count"), + ), "ttft_mean_ms": _float_or_none(result.get("ttft_mean_ms")), "e2el_mean_ms": _float_or_none(result.get("e2el_mean_ms")), "tpot_mean_ms": _float_or_none(result.get("tpot_mean_ms")), diff --git a/src/hyperloom/inference_optimizer/breakdown/recorder/event_fields.py b/src/hyperloom/inference_optimizer/breakdown/recorder/event_fields.py index 3a5c818bfe..bc5dfbfe83 100644 --- a/src/hyperloom/inference_optimizer/breakdown/recorder/event_fields.py +++ b/src/hyperloom/inference_optimizer/breakdown/recorder/event_fields.py @@ -122,7 +122,10 @@ def graded_axes(source: Any) -> dict[str, Any]: from hyperloom.common.perf_metric import GRADED_AXIS_KEYS, graded_axes_of axes = graded_axes_of(source) - return {key: float_or_none(axes.get(key)) for key in GRADED_AXIS_KEYS} + recorded = {key: float_or_none(axes.get(key)) for key in GRADED_AXIS_KEYS} + if "agentx_gpu_count" in axes: + recorded["agentx_gpu_count"] = axes["agentx_gpu_count"] + return recorded def summarize_hot_kernels(rows: Any) -> dict[str, Any]: diff --git a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/baseline.py b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/baseline.py index 1188b6f26f..a1063f61cb 100644 --- a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/baseline.py +++ b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/baseline.py @@ -14,6 +14,8 @@ from typing import Any +from ....performance_display import format_recorded_metric, recorded_throughput + from ..base import ( Decision, RenderedSection, @@ -102,7 +104,7 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: anchor = _anchor(attempts) measurement = as_dict(anchor.get("measurement")) session = session_of(breakdown) - tput = measurement.get("throughput_tok_s_per_gpu") + tput = recorded_throughput(measurement) acc = measurement.get("accuracy") ttft = measurement.get("ttft_mean_ms") e2el = measurement.get("e2el_mean_ms") @@ -117,11 +119,9 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: warnings: list[str] = [] decisions: list[Decision] = [] - from .... import framework_registry - fw = task_config_of(breakdown).get("framework_name") if tput: - facts.append(f"Baseline: {framework_registry.format_primary_metric(fw, tput, precision=2)}.") + facts.append(f"Baseline: {format_recorded_metric(fw, measurement, precision=2)}.") decisions.append( Decision( kind="attempted", @@ -163,7 +163,8 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: md_parts.append( md_kv_list( [ - ("throughput_tok_s_per_gpu", tput), + ("throughput_tok_s_per_gpu", measurement.get("throughput_tok_s_per_gpu")), + ("throughput_tok_s", measurement.get("throughput_tok_s")), ("throughput_unit", measurement.get("throughput_unit") or None), ("accuracy", acc), ("submission_valid", measurement.get("submission_valid")), @@ -190,7 +191,7 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: a.get("end_time") or a.get("start_time"), a.get("status"), a.get("decision"), - as_dict(a.get("measurement")).get("throughput_tok_s_per_gpu"), + format_recorded_metric(fw, as_dict(a.get("measurement")), precision=2), as_dict(a.get("failure")).get("error_class"), ] for a in attempts[:10] diff --git a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py index ad4e310aba..8fef45c928 100644 --- a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py +++ b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py @@ -13,6 +13,8 @@ from typing import Any +from ....performance_display import format_recorded_metric, recorded_metric_unit, recorded_throughput + from ..base import ( Decision, RenderedSection, @@ -104,8 +106,8 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: b = as_dict(outcome.get("baseline")) validation = validation_of(breakdown) session = session_of(breakdown) - final_tput = f.get("throughput_tok_s_per_gpu") - base_tput = b.get("throughput_tok_s_per_gpu") + final_tput = recorded_throughput(f) + base_tput = recorded_throughput(b) gain_v = f.get("gain_pct") val_stack_len = validation.get("validated_at_stack_len") val_ts = validation.get("validated_ts") @@ -131,10 +133,10 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: from .... import framework_registry fw = task_config_of(breakdown).get("framework_name") - _unit = framework_registry.primary_metric_unit(fw) + _unit = recorded_metric_unit(fw, f) if final_tput: - facts.append(f"Final: {framework_registry.format_primary_metric(fw, final_tput, precision=2)}.") - if base_tput and final_tput: + facts.append(f"Final: {format_recorded_metric(fw, f, precision=2)}.") + if base_tput and final_tput and recorded_metric_unit(fw, b) == _unit: base_v = framework_registry.primary_metric_value(fw, base_tput) final_v = framework_registry.primary_metric_value(fw, final_tput) if base_v is not None and final_v is not None: @@ -223,7 +225,8 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: md_kv = md_kv_list( [ - ("final_throughput_tok_s_per_gpu", final_tput), + ("final_throughput_tok_s_per_gpu", f.get("throughput_tok_s_per_gpu")), + ("final_throughput_tok_s", f.get("throughput_tok_s")), ("throughput_unit", _unit or None), ("validated_gain_pct", gain_v), ("revalidation_pending", revalidation_pending or None), diff --git a/src/hyperloom/inference_optimizer/breakdown/reporters/cross_section.py b/src/hyperloom/inference_optimizer/breakdown/reporters/cross_section.py index 9af5133a0d..55e90b1ee1 100644 --- a/src/hyperloom/inference_optimizer/breakdown/reporters/cross_section.py +++ b/src/hyperloom/inference_optimizer/breakdown/reporters/cross_section.py @@ -9,6 +9,9 @@ from typing import Any from hyperloom.common.coerce import to_float +from hyperloom.common.perf_metric import INTVTY_OBJECTIVES + +from ...performance_display import format_recorded_metric, recorded_metric_unit, recorded_throughput from .base import RenderedSection, as_dict, outcome_of, session_of, stop_reason_of, task_config_of, validation_of @@ -220,22 +223,25 @@ def _capabilities_split( def _headline(breakdown: dict[str, Any]) -> str: """Build the one-line baseline→final throughput headline.""" - from ... import framework_registry - fw = task_config_of(breakdown).get("framework_name") outcome = outcome_of(breakdown) - b = to_float(as_dict(outcome.get("baseline")).get("throughput_tok_s_per_gpu")) - f = to_float(as_dict(outcome.get("final")).get("throughput_tok_s_per_gpu")) + baseline = as_dict(outcome.get("baseline")) + final = as_dict(outcome.get("final")) + b = to_float(recorded_throughput(baseline)) + f = to_float(recorded_throughput(final)) g = to_float(as_dict(outcome.get("final")).get("gain_pct")) if b and f and g is not None: sign = "+" if g > 0 else "" - return ( - f"baseline {framework_registry.format_primary_metric(fw, b, precision=2)} → " - f"final {framework_registry.format_primary_metric(fw, f, precision=2)} " - f"= {sign}{g:.2f}% validated gain" + comparison = ( + f"baseline {format_recorded_metric(fw, baseline, precision=2)} → " + f"final {format_recorded_metric(fw, final, precision=2)}" ) + objective = str(final.get("graded_on") or "") + if objective in INTVTY_OBJECTIVES or recorded_metric_unit(fw, baseline) != recorded_metric_unit(fw, final): + return f"{comparison}; validated gain: {sign}{g:.2f}% ({objective or 'graded metric'})" + return f"{comparison} = {sign}{g:.2f}% validated gain" if b and not f: - return f"baseline {framework_registry.format_primary_metric(fw, b, precision=2)} (no validated final)" + return f"baseline {format_recorded_metric(fw, baseline, precision=2)} (no validated final)" return "no validated throughput recorded" diff --git a/src/hyperloom/inference_optimizer/cli/__init__.py b/src/hyperloom/inference_optimizer/cli/__init__.py index 63e7e80d82..19c880ba70 100644 --- a/src/hyperloom/inference_optimizer/cli/__init__.py +++ b/src/hyperloom/inference_optimizer/cli/__init__.py @@ -7,6 +7,7 @@ import argparse import asyncio +import hashlib import json import logging import os @@ -113,6 +114,7 @@ from ..session.lock import SessionAlreadyRunning, SessionLock from ..session.paths import ( + ENV_CURRENT_SESSION_DIR, ENV_USER_DATA_PATH, asset_system_prompts_dir, make_session_dir, @@ -137,6 +139,7 @@ ) from .preflight import ( _check_gfx_arch_resolvable, + _load_dotenv_fallback, _mark_pending_install_event_failed, _persist_install_event, _preflight as _preflight, @@ -974,17 +977,16 @@ def _reset_state_file(session_dir: Path, *, framework: str) -> None: ) -def _preflight_agentx_backend(args: argparse.Namespace) -> None: - """Reject the combinations where AgentX labels work it did not do.""" - if not _agentx_enabled(): - return +def _preflight_agentx_backend(args: argparse.Namespace, state: Any = None) -> None: + """Reject benchmark combinations that cannot execute their advertised workload.""" + from hyperloom.common.agentx_mode import native_agentx_session from hyperloom.inference_optimizer import framework_registry - from hyperloom.orchestrator.actions.executors.benchmark_backend import ( - resolve_backend_name, - ) + from hyperloom.orchestrator.actions.executors.benchmark_backend import resolve_backend_name - backend = resolve_backend_name() - if backend == "bypass": + native = native_agentx_session(state) + if not native and not _agentx_enabled(): + return + if resolve_backend_name() == "bypass": print( "ERROR: HYPERLOOM_AGENTX=1 with HYPERLOOM_BENCHMARK_BACKEND=bypass. " "The bypass backend ignores benchmark_script for serving frameworks, " @@ -993,19 +995,772 @@ def _preflight_agentx_backend(args: argparse.Namespace) -> None: file=sys.stderr, ) raise SystemExit(2) - - framework = str(getattr(args, "framework", "") or "").strip().lower() - if framework and framework_registry.is_scriptable(framework): + framework = ( + str(getattr(args, "framework", "") or os.environ.get("FRAMEWORK", "")).strip().lower() + or framework_registry.DEFAULT_FRAMEWORK + ) + if framework_registry.is_scriptable(framework): print( f"ERROR: HYPERLOOM_AGENTX=1 with --framework {framework!r}, which is " - "scriptable. The AgentX switch only replaces the benchmark client of a " - "serving framework, so no trace would be replayed -- but the session " - "would still be labelled AgentX and run on AgentX budgets. Pick one.", + "scriptable. AgentX requires a serving framework to replay its trace workload.", + file=sys.stderr, + ) + raise SystemExit(2) + if native: + _preflight_native_agentx_backend(args, framework, state) + + +def _preflight_native_agentx_backend(args: argparse.Namespace, framework: str, state: Any = None) -> None: + """Check the pinned single-node launcher contract for explicit native sessions.""" + from hyperloom.inference_optimizer.agentx.native import AGENTX_REQUIRED_RUNTIME_PIN_NAMES + + if os.environ.get("HYPERLOOM_AGENTIC_BACKEND", "").strip().lower() == "mlperf": + print("ERROR: native Magpie AgentX cannot use the MLPerf client backend.", file=sys.stderr) + raise SystemExit(2) + if not os.environ.get("AGENTX_MODEL_ID", "").strip(): + model_value = str(getattr(args, "model", "") or "").strip() + if model_value.count("/") == 1 and not model_value.startswith(("/", ".", "~")): + os.environ["AGENTX_MODEL_ID"] = model_value + missing_pins = [name for name in AGENTX_REQUIRED_RUNTIME_PIN_NAMES if not os.environ.get(name, "").strip()] + if missing_pins: + print( + "ERROR: native Magpie AgentX requires these immutable runtime " + "pins before session creation: " + ", ".join(missing_pins), + file=sys.stderr, + ) + raise SystemExit(2) + if os.environ.get("INFERENCE_OPTIMIZER_RAY_EXEC", "").strip().lower() in {"1", "true", "yes", "on"}: + print( + "ERROR: native Magpie AgentX v1 cannot run inside Hyperloom's Ray " + "serving actor. Leave INFERENCE_OPTIMIZER_RAY_EXEC unset (native AgentX " + "automatically uses direct local execution) or set it to 0.", + file=sys.stderr, + ) + raise SystemExit(2) + if int(getattr(args, "nodes", 1) or 1) != 1: + print( + "ERROR: native Magpie AgentX currently supports only --nodes 1. " + "Hyperloom's native Magpie adapter requires a single-node replay.", + file=sys.stderr, + ) + raise SystemExit(2) + if framework not in {"sglang", "vllm"}: + print( + f"ERROR: native Magpie AgentX currently supports only SGLang or vLLM; got --framework {framework!r}.", + file=sys.stderr, + ) + raise SystemExit(2) + if getattr(args, "enable_conc_sweep", None) is True: + print( + "ERROR: native Magpie AgentX binds concurrency to the session workload. " + "A post-optimization concurrency sweep would change that identity. " + "Omit --enable-conc-sweep and start a separate session with --conc " + "for each desired concurrency.", file=sys.stderr, ) raise SystemExit(2) +_AGENTX_STABLE_SELECTOR_KEYS = ( + "tp", + "pp", + "pcp-size", + "ep", + "dcp-size", + "dp-attn", + "spec-decoding", + "kv-offloading", + "kv-offload-backend", +) + + +def _canonical_agentx_control_pins( + benchmark: Mapping[str, Any], + *, + resolved_recipe: str = "", + resolved_config_file: str = "", +) -> dict[str, str]: + """Canonicalize every recipe-selection/result-validity control for resume.""" + raw_agentx = benchmark.get("agentx") + agentx = dict(raw_agentx) if isinstance(raw_agentx, Mapping) else {} + raw_workload = benchmark.get("workload_spec") + workload = raw_workload if isinstance(raw_workload, Mapping) else {} + raw_workload_recipe = workload.get("recipe") + workload_recipe = raw_workload_recipe if isinstance(raw_workload_recipe, Mapping) else {} + + mode = str(agentx.get("mode") or "canonical").strip().lower() + if mode not in {"canonical", "fast"}: + raise ValueError("benchmark.agentx.mode must be 'canonical' or 'fast'") + recipe = str(resolved_recipe or agentx.get("recipe") or workload_recipe.get("name") or "").strip() + config_file = str( + resolved_config_file or agentx.get("config_file") or workload_recipe.get("config_file") or "" + ).strip() + + raw_selector = agentx.get("selector") + selector = dict(raw_selector) if isinstance(raw_selector, Mapping) else {} + if not selector: + raw_resolved = agentx.get("resolved") + resolved = raw_resolved if isinstance(raw_resolved, Mapping) else {} + selector = {key: resolved[key] for key in _AGENTX_STABLE_SELECTOR_KEYS if key in resolved} + threshold_raw = agentx.get("failed_request_threshold", 0.10) + try: + threshold = float(threshold_raw) + except (TypeError, ValueError) as exc: + raise ValueError("benchmark.agentx.failed_request_threshold must be between 0 and 1") from exc + if not 0.0 <= threshold <= 1.0: + raise ValueError("benchmark.agentx.failed_request_threshold must be between 0 and 1") + + pins = { + "AGENTX_MODE": mode, + "AGENTX_FAILED_REQUEST_THRESHOLD": repr(threshold), + } + if recipe: + pins["AGENTX_RECIPE"] = recipe + if config_file: + pins["AGENTX_CONFIG_FILE"] = config_file + if selector: + pins["AGENTX_SELECTOR"] = json.dumps( + selector, + sort_keys=True, + separators=(",", ":"), + ) + return pins + + +def _configure_benchmark_config(args: argparse.Namespace) -> bool: + """Select a source Magpie YAML and promote its AgentX switch to session mode. + + Returns True when the selected YAML enables native AgentX. This runs + before preflight and SharedState seeding so every mode-sensitive subsystem + (Ray dispatch, grading, budgets, and the baseline materializer) observes + the same decision. + """ + os.environ.pop("HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", None) + os.environ.pop("HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", None) + os.environ.pop("HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT", None) + os.environ.pop("HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT", None) + os.environ.pop("HYPERLOOM_AGENTX_GPU_COUNT", None) + os.environ.pop("HYPERLOOM_BENCHMARK_CONFIG_SHA256", None) + cli_value = str(getattr(args, "benchmark_config", "") or "").strip() + if getattr(args, "resume_from", None): + if cli_value: + raise ValueError( + "--benchmark-config cannot be used with --resume-from; the resumed " + "session uses its accepted materialized baseline config" + ) + # A resume is entirely state-owned. Do not let a stale shell from a + # different launch select a foreign YAML before state.json is loaded. + os.environ.pop("HYPERLOOM_BENCHMARK_CONFIG", None) + return False + os.environ.pop(ENV_CURRENT_SESSION_DIR, None) + inherited = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "").strip() + raw_path = cli_value or inherited + if not raw_path: + return _agentx_enabled() + + path = Path(raw_path).expanduser().resolve() + if not path.is_file(): + raise ValueError(f"benchmark config does not exist or is not a file: {path}") + + import yaml + + try: + source_bytes = path.read_bytes() + parsed = yaml.safe_load(source_bytes.decode("utf-8")) or {} + except (OSError, UnicodeDecodeError, yaml.YAMLError) as exc: + raise ValueError(f"cannot read benchmark config {path}: {exc}") from exc + benchmark = parsed.get("benchmark") if isinstance(parsed, Mapping) else None + if not isinstance(benchmark, Mapping): + raise ValueError(f"benchmark config {path} must contain a benchmark mapping") + + cli_model_supplied = getattr(args, "model", None) is not None + runtime_model_path = os.environ.get("MODEL_PATH", "").strip() + configured_framework = str(benchmark.get("framework") or "").strip().lower() + cli_framework = str(getattr(args, "framework", "") or "").strip().lower() + if configured_framework and cli_framework and configured_framework != cli_framework: + raise ValueError( + f"--framework {cli_framework!r} conflicts with benchmark.framework {configured_framework!r} in {path}" + ) + if configured_framework and not cli_framework: + args.framework = configured_framework + if getattr(args, "model", None) is None: + if runtime_model_path: + args.model = Path(runtime_model_path).expanduser() + elif benchmark.get("model"): + args.model = Path(str(benchmark["model"])) + if not getattr(args, "precision", None) and benchmark.get("precision"): + args.precision = str(benchmark["precision"]) + + configured_runner = str(benchmark.get("runner_type") or "").strip().lower() + cli_runner = str(getattr(args, "gpu_type", "") or "").strip().lower() + if configured_runner and cli_runner and configured_runner != cli_runner: + raise ValueError( + f"--gpu-type {cli_runner!r} conflicts with benchmark.runner_type {configured_runner!r} in {path}" + ) + if configured_runner and not cli_runner: + args.gpu_type = configured_runner + + os.environ["HYPERLOOM_BENCHMARK_CONFIG"] = str(path) + os.environ["HYPERLOOM_BENCHMARK_CONFIG_SHA256"] = hashlib.sha256(source_bytes).hexdigest() + from hyperloom.common.agentx_mode import native_agentx_enabled + + native = native_agentx_enabled(benchmark.get("agentx")) + if _agentx_enabled() and "agentx" in benchmark and not native: + raise ValueError("HYPERLOOM_AGENTX=1 conflicts with disabled benchmark.agentx") + native = native or _agentx_enabled() + raw_envs = benchmark.get("envs") + benchmark_envs = raw_envs if isinstance(raw_envs, Mapping) else {} + if native: + from hyperloom.inference_optimizer.agentx.native import validate_native_replay_env + + validate_native_replay_env(benchmark_envs, source=f"benchmark.envs in {path}") + if native and not cli_model_supplied and not runtime_model_path: + local_model_path = str(benchmark_envs.get("MODEL_PATH") or "").strip() + if local_model_path: + if local_model_path.count("/") == 1 and not local_model_path.startswith(("/", ".", "~")): + raise ValueError( + f"benchmark.envs.MODEL_PATH={local_model_path!r} in {path} " + "looks like a Hugging Face id, not a local checkpoint path" + ) + args.model = Path(local_model_path).expanduser() + + def _positive_yaml_int(env_name: str) -> int | None: + raw = benchmark_envs.get(env_name) + if raw in (None, ""): + return None + try: + value = int(raw) + except (TypeError, ValueError) as exc: + raise ValueError(f"benchmark.envs.{env_name} in {path} must be a positive integer") from exc + if value <= 0: + raise ValueError(f"benchmark.envs.{env_name} in {path} must be a positive integer") + return value + + # A selected YAML is a real CLI input, not merely a template. Project its + # workload values before _run_optimize exports defaults. Native AgentX TP + # is handled after recipe resolution because source envs.TP is tensor + # parallelism, while Hyperloom's --tp accounts for every physical rank + # (TP x PP x PCP). + projected = ( + ("conc", "CONC"), + ("ep", "EP"), + ("isl", "ISL"), + ("osl", "OSL"), + ("max_model_len", "MAX_MODEL_LEN"), + ) + for attr, env_name in projected: + if getattr(args, attr, None) is None: + value = _positive_yaml_int(env_name) + if value is not None: + setattr(args, attr, value) + raw_agentx = benchmark.get("agentx") + if getattr(args, "conc", None) is None and isinstance(raw_agentx, Mapping): + configured_conc = raw_agentx.get("concurrency") + if configured_conc is not None: + args.conc = int(configured_conc) + if args.conc <= 0: + raise ValueError("benchmark.agentx.concurrency must be a positive integer") + if not native and getattr(args, "tp", None) is None: + yaml_tp = _positive_yaml_int("TP") + if yaml_tp is not None: + args.tp = yaml_tp + + if native: + raw_mode = os.environ.get("HYPERLOOM_AGENTX", "").strip().lower() + if raw_mode and raw_mode not in {"1", "true", "yes", "on", "enable", "enabled"}: + raise ValueError( + f"benchmark.agentx is enabled in {path}, but " + f"HYPERLOOM_AGENTX={raw_mode!r} explicitly disables the session" + ) + os.environ["HYPERLOOM_AGENTX"] = "1" + top_model_id = str(benchmark.get("model") or "").strip() + env_model_id = str(benchmark_envs.get("AGENTX_MODEL_ID") or "").strip() + if top_model_id and env_model_id and top_model_id != env_model_id: + raise ValueError( + f"benchmark.model={top_model_id!r} conflicts with " + f"benchmark.envs.AGENTX_MODEL_ID={env_model_id!r} in {path}" + ) + top_script = str(benchmark.get("benchmark_script") or "").strip() + env_script = str(benchmark_envs.get("AGENTX_SERVER_SCRIPT") or "").strip() + if top_script and env_script and top_script != env_script: + raise ValueError( + f"benchmark.benchmark_script={top_script!r} conflicts with " + "benchmark.envs.AGENTX_SERVER_SCRIPT=" + f"{env_script!r} in {path}" + ) + canonical_model_id = env_model_id or top_model_id + selected_model = str(getattr(args, "model", "") or "").strip() + selected_is_remote = selected_model.count("/") == 1 and not selected_model.startswith(("/", ".", "~")) + if ( + (cli_model_supplied or runtime_model_path) + and selected_is_remote + and canonical_model_id + and selected_model != canonical_model_id + ): + source = "--model" if cli_model_supplied else "MODEL_PATH" + raise ValueError( + f"{source}={selected_model!r} conflicts with canonical AgentX " + f"model {canonical_model_id!r} in {path}; only a local path may " + "override the recipe's model id" + ) + source_pins = { + "AGENTX_MODEL_ID": str(canonical_model_id).strip(), + "AGENTX_SERVER_SCRIPT": str(env_script or top_script).strip(), + "INFERENCEX_PATH": str(benchmark.get("inferencex_path") or "").strip(), + } + for name, source_value in source_pins.items(): + if not source_value: + continue + if name == "INFERENCEX_PATH": + source_path = Path(source_value).expanduser() + if not source_path.is_absolute(): + source_path = path.parent / source_path + source_value = str(source_path.resolve()) + current = os.environ.get(name, "").strip() + comparable_current = current + if current and name == "INFERENCEX_PATH": + comparable_current = str(Path(current).expanduser().resolve()) + if comparable_current and comparable_current != source_value: + raise ValueError(f"{name}={current!r} conflicts with {source_value!r} in {path}") + os.environ[name] = source_value + + return native + + +def _finalize_benchmark_config(args: argparse.Namespace) -> bool: + """Resolve native recipe topology after dependency preflight. + + ``_configure_benchmark_config`` runs before preflight so mode and identity + are known early, but a clean install may not have a Magpie interpreter or + a pinned InferenceX checkout yet. This second step runs immediately after + preflight has installed/validated both and before session state or GPU + resource accounting is created. + """ + if getattr(args, "resume_from", None): + return False + raw_path = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "").strip() + if not raw_path: + return False + path = Path(raw_path).expanduser().resolve() + + import copy + import yaml + + try: + source_bytes = path.read_bytes() + expected_source_hash = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG_SHA256", "").strip() + actual_source_hash = hashlib.sha256(source_bytes).hexdigest() + if not expected_source_hash or actual_source_hash != expected_source_hash: + raise ValueError(f"benchmark config changed after initial validation; refusing the mutable source {path}") + parsed = yaml.safe_load(source_bytes.decode("utf-8")) or {} + except (OSError, UnicodeDecodeError, yaml.YAMLError) as exc: + raise ValueError(f"cannot read benchmark config {path}: {exc}") from exc + benchmark = parsed.get("benchmark") if isinstance(parsed, Mapping) else None + if not isinstance(benchmark, Mapping): + raise ValueError(f"benchmark config {path} must contain a benchmark mapping") + + from hyperloom.inference_optimizer.agentx.native import ( + native_agentx_enabled, + native_execution_identity, + native_runtime_search_paths, + preview_native_recipe, + ) + + if not native_agentx_enabled(benchmark.get("agentx")): + return False + inferencex_path = os.environ.get("INFERENCEX_PATH", "").strip() + if not inferencex_path: + raise ValueError("native AgentX preflight did not produce an INFERENCEX_PATH") + + preview_benchmark = copy.deepcopy(dict(benchmark)) + preview_benchmark["inferencex_path"] = inferencex_path + if getattr(args, "precision", None): + # Materialization projects the process PRECISION over the source YAML. + # Resolve that same arm before scheduling GPUs; otherwise a CLI + # precision override could preview one recipe and execute another. + preview_benchmark["precision"] = str(args.precision).strip() + preview_envs_raw = preview_benchmark.get("envs") + preview_envs = preview_envs_raw if isinstance(preview_envs_raw, dict) else {} + preview_benchmark["envs"] = preview_envs + preview_envs.update(native_runtime_search_paths()) + for name in ( + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_MODE", + "AGENTX_RECIPE", + "AGENTX_CONFIG_FILE", + "AGENTX_SELECTOR", + "AGENTX_FAILED_REQUEST_THRESHOLD", + ): + value = os.environ.get(name, "").strip() + if value: + preview_envs[name] = value + canonical_model = os.environ.get("AGENTX_MODEL_ID", "").strip() + canonical_script = os.environ.get("AGENTX_SERVER_SCRIPT", "").strip() + if canonical_model: + preview_benchmark["model"] = canonical_model + if canonical_script: + preview_benchmark["benchmark_script"] = canonical_script + + requested_conc = getattr(args, "conc", None) + raw_agentx = preview_benchmark.get("agentx") + if requested_conc is None and isinstance(raw_agentx, Mapping): + configured_conc = raw_agentx.get("concurrency") + if configured_conc not in (None, ""): + try: + requested_conc = int(configured_conc) + except (TypeError, ValueError) as exc: + raise ValueError(f"benchmark.agentx.concurrency in {path} must be a positive integer") from exc + if requested_conc <= 0: + raise ValueError(f"benchmark.agentx.concurrency in {path} must be a positive integer") + if requested_conc is not None: + preview_envs["CONC"] = int(requested_conc) + from hyperloom.orchestrator.actions.executors._workload_envs import ( + _native_agentx_config, + ) + + preview_benchmark["agentx"] = _native_agentx_config( + preview_envs, + raw_agentx, + ) + + try: + preview = preview_native_recipe( + preview_benchmark, + inferencex_path=inferencex_path, + ) + except (OSError, RuntimeError, ValueError) as exc: + raise ValueError(f"cannot resolve native AgentX recipe from {path}: {exc}") from exc + topology = preview["topology"] + selected_recipe = str(preview.get("recipe") or "").strip() + selected_config_file = str(preview.get("config_file") or "").strip() + if not selected_recipe or not selected_config_file: + raise ValueError("resolved AgentX recipe did not provide recipe/config-file identity") + recipe_fingerprint = str(topology.get("recipe_fingerprint") or "").strip() + if not recipe_fingerprint: + raise ValueError("resolved AgentX recipe did not provide a fingerprint") + resolved_benchmark = preview.get("benchmark") + control_source = resolved_benchmark if isinstance(resolved_benchmark, Mapping) else preview_benchmark + for name, field in (("AGENTX_MODEL_ID", "model"), ("AGENTX_SERVER_SCRIPT", "benchmark_script")): + value = str(control_source.get(field) or "").strip() + if value: + os.environ[name] = value + execution = native_execution_identity( + inferencex_path=inferencex_path, + benchmark_script=str(control_source.get("benchmark_script") or ""), + config_file=selected_config_file, + resolved_benchmark=control_source, + expected_ref=os.environ.get("INFERENCEX_REF", ""), + magpie_execution=preview.get("magpie_execution"), + expected_magpie_ref=os.environ.get("MAGPIE_REF", ""), + ) + os.environ["HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT"] = recipe_fingerprint + os.environ["HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT"] = execution["static_execution_fingerprint"] + gpu_count = int(topology["gpu_count"]) + os.environ["HYPERLOOM_AGENTX_GPU_COUNT"] = str(gpu_count) + for name, value in _canonical_agentx_control_pins( + control_source, + resolved_recipe=selected_recipe, + resolved_config_file=selected_config_file, + ).items(): + os.environ[name] = value + explicit_tp = getattr(args, "tp", None) + if explicit_tp is not None and int(explicit_tp) != gpu_count: + raise ValueError( + f"--tp {explicit_tp} conflicts with the resolved AgentX physical " + f"GPU count {gpu_count} (TP x PP x PCP) in {path}" + ) + args.tp = gpu_count + args.conc = int(topology["conc"]) + explicit_ep = getattr(args, "ep", None) + resolved_ep = int(topology["ep"]) + if explicit_ep is not None and int(explicit_ep) != resolved_ep: + raise ValueError(f"--ep {explicit_ep} conflicts with resolved AgentX EP {resolved_ep} in {path}") + args.ep = resolved_ep + declared_image = os.environ.get("HYPERLOOM_IMAGE", "").strip() + recipe_image = str(preview["entry"].get("image") or "").strip() + if declared_image and declared_image != recipe_image: + raise ValueError( + f"HYPERLOOM_IMAGE does not match the resolved AgentX recipe: {declared_image!r} != {recipe_image!r}" + ) + if not recipe_image: + raise ValueError("resolved AgentX recipe did not declare a runtime image") + # The YAML's docker_image (or the recipe default when omitted) is the + # session image pin. Preserve a shell value only when it agrees exactly; + # no duplicate export is required for the normal --benchmark-config path. + os.environ["HYPERLOOM_IMAGE"] = recipe_image + return True + + +def _restore_agentx_env_from_state(state: Any) -> bool: + """Re-export and normalize a persisted AgentX session mode.""" + if str(getattr(state, "benchmark_mode", "") or "").strip().lower() != "agentx": + return False + raw = os.environ.get("HYPERLOOM_AGENTX", "").strip() + if raw and raw.lower() not in { + "1", + "true", + "yes", + "on", + "enable", + "enabled", + }: + raise ValueError(f"HYPERLOOM_AGENTX={raw!r} conflicts with the saved AgentX session mode") + backend = str(getattr(state, "agentx_backend", "") or "").strip().lower() + if int(getattr(state, "agentx_epoch", 0) or 0) == 1: + from hyperloom.common.agentx_workload import agentic_backend + + backend = "aiperf" if backend in {"", "legacy"} else backend + explicit = os.environ.get("HYPERLOOM_AGENTIC_BACKEND", "").strip() + if explicit and agentic_backend() != backend: + raise ValueError("HYPERLOOM_AGENTIC_BACKEND conflicts with the saved AgentX client") + os.environ["HYPERLOOM_AGENTIC_BACKEND"] = backend + if raw == "1": + return False + os.environ["HYPERLOOM_AGENTX"] = "1" + return True + + +def _restore_agentx_runtime_pins_from_state( + state: Any, + *, + operator_pins: Mapping[str, str] | None = None, +) -> dict[str, str]: + """Restore a native AgentX session's identity from its accepted baseline. + + Some resume actions, notably ``profile``, start from a shipped template + rather than the saved baseline YAML. Project the immutable native runtime + pins back into the process before those actions build a workload so a new + shell cannot silently choose another model, launcher, checkout, or image. + """ + from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session + + if not native_agentx_session(state): + for name in ( + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT", + "HYPERLOOM_AGENTX_GPU_COUNT", + ): + os.environ.pop(name, None) + return {} + + config_path = str(getattr(state, "baseline_config_path", "") or "").strip() + raw_seed_pins = getattr(state, "agentx_runtime_pins", {}) + seed_pins = ( + {str(name): str(value).strip() for name, value in raw_seed_pins.items()} + if isinstance(raw_seed_pins, Mapping) + else {} + ) + if seed_pins.get("INFERENCEX_PATH"): + seed_pins["INFERENCEX_PATH"] = str(Path(seed_pins["INFERENCEX_PATH"]).expanduser().resolve()) + + if config_path: + import yaml + + try: + parsed = yaml.safe_load(Path(config_path).expanduser().read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError) as exc: + raise ValueError(f"cannot read the saved AgentX baseline config {config_path!r}: {exc}") from exc + + benchmark = parsed.get("benchmark") if isinstance(parsed, Mapping) else None + if not isinstance(benchmark, Mapping): + raise ValueError(f"saved AgentX baseline config {config_path!r} has no benchmark mapping") + pin_source = config_path + else: + seed_agentx: dict[str, Any] = {"enabled": True} + if native_agentx_optimization_session(state): + seed_agentx["launch_overrides"] = {"version": 1} + if seed_pins.get("AGENTX_MODE"): + seed_agentx["mode"] = seed_pins["AGENTX_MODE"] + if seed_pins.get("AGENTX_RECIPE"): + seed_agentx["recipe"] = seed_pins["AGENTX_RECIPE"] + if seed_pins.get("AGENTX_CONFIG_FILE"): + seed_agentx["config_file"] = seed_pins["AGENTX_CONFIG_FILE"] + if seed_pins.get("AGENTX_SELECTOR"): + try: + seed_selector = json.loads(seed_pins["AGENTX_SELECTOR"]) + except json.JSONDecodeError as exc: + raise ValueError("seed-time AGENTX_SELECTOR is not valid JSON") from exc + if not isinstance(seed_selector, dict): + raise ValueError("seed-time AGENTX_SELECTOR must be a JSON object") + seed_agentx["selector"] = seed_selector + if seed_pins.get("AGENTX_FAILED_REQUEST_THRESHOLD"): + seed_agentx["failed_request_threshold"] = seed_pins["AGENTX_FAILED_REQUEST_THRESHOLD"] + benchmark = { + "agentx": seed_agentx, + "model": seed_pins.get("AGENTX_MODEL_ID", ""), + "benchmark_script": seed_pins.get("AGENTX_SERVER_SCRIPT", ""), + "inferencex_path": seed_pins.get("INFERENCEX_PATH", ""), + "envs": dict(seed_pins), + "workload_spec": {"outer_image": seed_pins.get("HYPERLOOM_IMAGE", "")}, + } + pin_source = "seed-time AgentX session state" + + from hyperloom.inference_optimizer.agentx.native import ( + AGENTX_REQUIRED_RUNTIME_PIN_NAMES, + AGENTX_RUNTIME_PIN_NAMES, + native_agentx_enabled, + validate_native_launcher_name, + ) + + if not native_agentx_enabled(benchmark.get("agentx")): + raise ValueError(f"saved AgentX pin source {pin_source!r} does not enable native AgentX") + from hyperloom.inference_optimizer.agentx.identity import has_launch_contract + + if native_agentx_optimization_session(state) != has_launch_contract(benchmark): + raise ValueError(f"saved AgentX launch contract in {pin_source!r} conflicts with the session epoch") + + raw_envs = benchmark.get("envs") + envs = raw_envs if isinstance(raw_envs, Mapping) else {} + raw_workload = benchmark.get("workload_spec") + workload = raw_workload if isinstance(raw_workload, Mapping) else {} + raw_recipe = workload.get("recipe") + recipe = raw_recipe if isinstance(raw_recipe, Mapping) else {} + raw_execution = workload.get("execution") + execution = raw_execution if isinstance(raw_execution, Mapping) else {} + + model_id = str(envs.get("AGENTX_MODEL_ID") or benchmark.get("model") or "").strip() + launcher = str(envs.get("AGENTX_SERVER_SCRIPT") or benchmark.get("benchmark_script") or "").strip() + inferencex_path = str(benchmark.get("inferencex_path") or "").strip() + outer_image = str(workload.get("outer_image") or recipe.get("image") or benchmark.get("docker_image") or "").strip() + recipe_fingerprint = str(recipe.get("recipe_fingerprint") or "").strip() + raw_topology = workload.get("resolved_topology") + topology = raw_topology if isinstance(raw_topology, Mapping) else {} + gpu_count = str(topology.get("gpu_count") or "").strip() + + if launcher: + validate_native_launcher_name(launcher) + if inferencex_path: + inferencex_path = str(Path(inferencex_path).expanduser().resolve()) + + baseline_pins = { + "AGENTX_MODEL_ID": model_id, + "AGENTX_SERVER_SCRIPT": launcher, + "INFERENCEX_PATH": inferencex_path, + "HYPERLOOM_IMAGE": outer_image, + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": recipe_fingerprint, + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": str( + execution.get("static_execution_fingerprint") or "" + ).strip(), + "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT": str( + execution.get("execution_fingerprint") or "" + ).strip(), + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT": str(execution.get("workload_fingerprint") or "").strip(), + "HYPERLOOM_AGENTX_GPU_COUNT": gpu_count, + "MAGPIE_REF": str(execution.get("magpie_commit") or seed_pins.get("MAGPIE_REF") or "").strip(), + "INFERENCEX_REF": str(execution.get("inferencex_commit") or seed_pins.get("INFERENCEX_REF") or "").strip(), + } + baseline_pins.update(_canonical_agentx_control_pins(benchmark)) + if native_agentx_optimization_session(state): + baseline_pins.pop("HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT", None) + seed_pins.pop("HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT", None) + for name, baseline_value in baseline_pins.items(): + if seed_pins.get(name) and baseline_value and seed_pins[name] != baseline_value: + raise ValueError( + f"{name} in {pin_source!r} conflicts with the seed-time AgentX session pin {seed_pins[name]!r}" + ) + + # Baseline-derived identity/fingerprint values outrank the seed fallback. + # Selector controls do not necessarily survive as env keys in Magpie's + # materialized YAML, so retain their exact seed-time spelling as well. + saved_pins = { + name: baseline_pins.get(name, "") or seed_pins.get(name, "") + for name in AGENTX_RUNTIME_PIN_NAMES + if baseline_pins.get(name, "") or seed_pins.get(name, "") + } + required_pins = list(AGENTX_REQUIRED_RUNTIME_PIN_NAMES) + if config_path: + required_pins.extend( + ( + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", + ( + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT" + if native_agentx_optimization_session(state) + else "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT" + ), + "HYPERLOOM_AGENTX_GPU_COUNT", + "MAGPIE_REF", + "INFERENCEX_REF", + ) + ) + missing = [name for name in required_pins if not saved_pins.get(name)] + if missing: + raise ValueError( + f"saved native AgentX pin source {pin_source!r} is missing required session pin(s): {', '.join(missing)}" + ) + + current_pins: dict[str, str] = {} + for name in AGENTX_RUNTIME_PIN_NAMES: + saved = saved_pins.get(name, "") + # On resume, dependency preflight runs before state is loaded and may + # auto-detect/clone then overwrite INFERENCEX_PATH. Only the value that + # existed in the operator's shell before preflight is a conflicting + # choice; a preflight-generated path must not outrank session state. + current = ( + str(operator_pins.get(name) or "").strip() + if operator_pins is not None + else os.environ.get(name, "").strip() + ) + comparable_current = current + if current and name == "INFERENCEX_PATH": + comparable_current = str(Path(current).expanduser().resolve()) + if comparable_current and (not saved or comparable_current != saved): + expected = repr(saved) if saved else "unset" + raise ValueError( + f"{name}={current!r} conflicts with the saved AgentX session pin {expected} from {pin_source!r}" + ) + current_pins[name] = current + + restored: dict[str, str] = {} + for name, saved in saved_pins.items(): + live = os.environ.get(name, "").strip() + if not current_pins[name] or live != saved: + os.environ[name] = saved + restored[name] = saved + for name in AGENTX_RUNTIME_PIN_NAMES: + if name not in saved_pins: + os.environ.pop(name, None) + if config_path: + # Env-only dispatch (including Ray) must inherit the accepted native + # identity after resume cleared the caller's source YAML. + os.environ["HYPERLOOM_BENCHMARK_CONFIG"] = str(Path(config_path).expanduser().resolve()) + return restored + + +def _enforce_agentx_resume_workload( + args: argparse.Namespace, + state: Any, +) -> None: + """Keep a resumed native AgentX session on its original workload point.""" + from hyperloom.common.agentx_mode import native_agentx_session + + if not native_agentx_session(state): + return + for name in ("tp", "ep", "conc"): + saved = int(getattr(state, name, 0) or 0) + if saved <= 0: + raise ValueError(f"saved AgentX session is missing a positive {name} pin") + requested = getattr(args, name, None) + if requested is not None and int(requested) != saved: + raise ValueError( + f"--{name.replace('_', '-')} {requested} conflicts with the saved " + f"AgentX workload value {saved}; resume cannot select another recipe point" + ) + setattr(args, name, saved) + saved_precision = str(getattr(state, "precision", "") or "").strip() + if not saved_precision: + raise ValueError("saved AgentX session is missing its precision pin") + requested_precision = str(getattr(args, "precision", "") or "").strip() + if requested_precision and requested_precision.lower() != saved_precision.lower(): + raise ValueError( + f"--precision {requested_precision!r} conflicts with the saved AgentX precision {saved_precision!r}" + ) + args.precision = saved_precision + + def _apply_agentx_budget_profile(args: argparse.Namespace) -> None: """Warn when AgentX is enabled without an explicit session budget.""" if not _agentx_enabled(): @@ -1074,9 +1829,11 @@ def _resolve_run_max_model_len_inner(args: argparse.Namespace) -> tuple[int, str # Model not on disk yet (uncached HF id). print( "WARNING: HYPERLOOM_AGENTX is on but the model's native context could " - "not be read (weights not on disk yet), so MAX_MODEL_LEN falls back " - "to the synthetic ISL+OSL derivation and the server will be booted at " - "that width. Pre-fetch the weights, or pass --max-model-len.", + "not be read (weights not on disk yet), so Hyperloom's compatibility " + "profile server falls back to the synthetic ISL+OSL width. The native " + "InferenceX AgentX launcher ignores this fallback and uses the model's " + "own context. Pre-fetch the weights, or pass --max-model-len, before " + "requesting a profiler-compatibility leg.", file=sys.stderr, ) return ( @@ -1561,6 +2318,109 @@ def _persist_preflight_failure_artifacts( return session_dir +def _resolve_fresh_gpu(args: argparse.Namespace) -> str: + """Resolve the actual target GPU before native recipe selection.""" + # Resolve real target GPU: probe > --gpu-type hint; probe wins to catch wrong-host typos that corrupt KB. + user_specified = _resolve_amd_gpu_type(args.gpu_type) or "" + if _should_remote_probe_gpu(args): + from ..multi_node._internal.gpu_probe import remote_autodetect_gpu_type + + probed = remote_autodetect_gpu_type() or "" + if probed: + print(f"GPU probe : {probed} (remote {(args.mn_backend or 'rayjob').lower()})") + else: + probed = _autodetect_gpu_type() or "" + gpu_type, gpu_warnings = _resolve_gpu_type( + user_specified=user_specified, + probed=probed, + ) + for line in gpu_warnings: + print(line, file=sys.stderr) + if probed and not user_specified: + print(f"GPU type : {gpu_type} (auto-detected)") + runner_gpu_type = _gpu_runner_type(gpu_type) + if gpu_type and runner_gpu_type != gpu_type: + print( + f"WARN: {gpu_type} uses {runner_gpu_type} as Magpie " + f"runner_type (same gfx942/CDNA3 arch; Magpie has no " + f"sglang_{gpu_type}.sh / vllm_{gpu_type}.sh yet)", + file=sys.stderr, + ) + args.gpu_type = gpu_type or None + if runner_gpu_type: + os.environ["TARGET_GPU_TYPE"] = gpu_type + os.environ["GPU_TYPE"] = runner_gpu_type + print(f"GPU type : {gpu_type}") + print(f"Magpie runner : {runner_gpu_type} (will inject runner_type into Magpie YAML)") + else: + os.environ.pop("TARGET_GPU_TYPE", None) + os.environ.pop("GPU_TYPE", None) + args.gpu_type = None + print("GPU type : (Magpie will auto-detect)") + return gpu_type + + +async def _prepare_fresh_workload(args: argparse.Namespace) -> tuple[str, str]: + """Resolve model, framework and target hardware before selecting a recipe.""" + # Resolve model path: --model > $MODEL_PATH; fail fast rather than silently use the YAML hardcoded model. + if not args.model: + args.model = os.environ.get("MODEL_PATH") or "" + if not args.model: + print( + "ERROR: model is required. Pass --model or set " + "MODEL_PATH env (or use --resume-from to " + "continue an existing session).", + file=sys.stderr, + ) + sys.exit(2) + # Re-export so subprocess executors inject the resolved model into the Magpie YAML, not its hardcoded model. + from hyperloom.common.model_paths import resolve_serving_model_path + + os.environ["MODEL_PATH"] = resolve_serving_model_path(str(args.model)) or str(args.model) + + # Quantization prelude (one-shot, before any session/baseline work): if --quantize was passed, quantize the + # source model now and rewrite args.model to the exported quantized model. + await _run_quantization_prelude(args) + + # Resolve framework: --framework > $FRAMEWORK > "sglang" (session-wide; no framework mixing). + framework = _registered_framework_or_exit( + args.framework or os.environ.get("FRAMEWORK", ""), + hint="set $FRAMEWORK accordingly or pass --framework", + ) + _enforce_expected_framework(framework) + os.environ["FRAMEWORK"] = framework + print(f"Framework : {framework}") + _apply_operator_supplied_paths(args, framework) + + # B3: --framework atom auto-tightens incompatible phases (see _apply_atom_auto_tighten). + if framework == "atom": + _apply_atom_auto_tighten(args) + + gpu_type = _resolve_fresh_gpu(args) + _require_custom_entrypoint(framework, gpu_type=_gpu_runner_type(gpu_type) or gpu_type) + _check_gfx_arch_resolvable(args.gpu_type) + resolved = argparse.Namespace(**vars(args)) + _resolve_workload_knobs(resolved) + args.precision = resolved.precision + args.conc = resolved.conc + return framework, gpu_type + + +def _preflight_for_session(args: argparse.Namespace) -> dict[str, str]: + """Read persisted mode during resume preflight without committing the session pointer.""" + previous = os.environ.get(ENV_CURRENT_SESSION_DIR) + if args.resume_from: + os.environ[ENV_CURRENT_SESSION_DIR] = str(Path(args.resume_from).expanduser().resolve()) + try: + return _preflight(args) + finally: + if args.resume_from: + if previous is None: + os.environ.pop(ENV_CURRENT_SESSION_DIR, None) + else: + os.environ[ENV_CURRENT_SESSION_DIR] = previous + + def _start_experience_kb() -> None: """Bring the workspace's Experience KB service to serving, then check its Experiences can be collected.""" from hyperloom_kb import ConfigurationError, LocalServiceError, RemoteClientError @@ -1582,6 +2442,29 @@ def _start_experience_kb() -> None: async def _run_optimize(args: argparse.Namespace) -> int: """Run the ``optimize`` subcommand end to end.""" + try: + # ``HYPERLOOM_BENCHMARK_CONFIG`` is an allowed trusted .env setting. + # Load it before topology/default export so the YAML is a true launch + # input rather than a late materialization detail. _preflight repeats + # the idempotent load so its install trace still records the source. + _load_dotenv_fallback() + _configure_benchmark_config(args) + except ValueError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + raise SystemExit(2) from exc + + resume_operator_agentx_pins: dict[str, str] | None = None + if args.resume_from: + from hyperloom.inference_optimizer.agentx.native import ( + AGENTX_RUNTIME_PIN_NAMES, + ) + + resume_operator_agentx_pins = { + name: os.environ.get(name, "").strip() + for name in AGENTX_RUNTIME_PIN_NAMES + if not name.startswith("HYPERLOOM_AGENTX_") + } + # Surface --nodes (CLI flag wins) before _preflight runs. nodes_resolved = max(1, int(args.nodes)) tp_resolved = max(1, int(getattr(args, "tp", 1) or 1)) @@ -1694,7 +2577,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: # ANTHROPIC_BASE_URL). codex_follows_claude = _codex_model_should_follow_claude() try: - resolved_urls = _preflight(args) + resolved_urls = _preflight_for_session(args) except Exception as exc: try: _persist_preflight_failure_artifacts(args, exc) @@ -1702,6 +2585,34 @@ async def _run_optimize(args: argparse.Namespace) -> int: log.warning("failed to preserve SBD V6 preflight failure", exc_info=True) raise + if args.resume_from: + # A resume is state-owned. The second idempotent dotenv load inside + # preflight may have reintroduced a foreign source YAML after + # _configure_benchmark_config deliberately cleared it. + os.environ.pop("HYPERLOOM_BENCHMARK_CONFIG", None) + else: + framework, gpu_type = await _prepare_fresh_workload(args) + try: + from .agentx_source import prepare_native_agentx_source + + prepare_native_agentx_source(args) + _finalize_benchmark_config(args) + except ValueError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + raise SystemExit(2) from exc + # Native AgentX topology is only knowable after preflight has installed + # Magpie and validated the pinned InferenceX checkout. Replace the + # provisional defaults exported above before any session state or GPU + # resource request is created. + tp_resolved = max(1, int(getattr(args, "tp", 1) or 1)) + ep_resolved = max(1, int(getattr(args, "ep", 1) or 1)) + _export_workload_envs_for_optimize( + args, + nodes_resolved=nodes_resolved, + tp_resolved=tp_resolved, + ep_resolved=ep_resolved, + ) + _resolve_models_for_run( args, resolved_urls, @@ -1711,7 +2622,6 @@ async def _run_optimize(args: argparse.Namespace) -> int: _start_experience_kb() # Before either session branch: these are read by the fresh-launch seeding AND by the resume path, so this is the # one place that covers both. - _preflight_agentx_backend(args) if not args.resume_from: _scope_error = latency_budget_scope_error( getattr(args, "framework", None) or os.environ.get("FRAMEWORK", ""), getattr(args, "max_latency_ms", None) @@ -1719,10 +2629,15 @@ async def _run_optimize(args: argparse.Namespace) -> int: if _scope_error: print(f"ERROR: {_scope_error}.", file=sys.stderr) raise SystemExit(2) - _apply_agentx_budget_profile(args) from hyperloom.orchestrator.actions.executors._subprocess_kill import resolve_benchmark_timeouts - resolve_benchmark_timeouts() + # A resume may be launched from a fresh shell, where AgentX mode is only in + # state.json. Defer mode-sensitive checks until that state has been loaded + # and re-exported; fresh launches can settle them immediately. + if not args.resume_from: + _preflight_agentx_backend(args) + _apply_agentx_budget_profile(args) + resolve_benchmark_timeouts() # A fresh launch has nothing to restore, so settle the budget before the # manifest and the workload env read it. A resume keeps ``None`` until # ``_restore_budget_and_objective`` has had its chance at the session's own. @@ -1797,6 +2712,40 @@ async def _run_optimize(args: argparse.Namespace) -> int: file=sys.stderr, ) sys.exit(2) + if _restore_agentx_env_from_state(state): + print(" re-exported HYPERLOOM_AGENTX: 1 (persisted session mode)") + if ( + str(getattr(state, "benchmark_mode", "") or "").strip().lower() == "agentx" + and not str(getattr(state, "baseline_config_path", "") or "").strip() + ): + source_config = str(getattr(state, "benchmark_source_config_path", "") or "").strip() + if source_config: + source_path = Path(source_config).expanduser().resolve() + if not source_path.is_file(): + print( + "ERROR: cannot resume this AgentX session -- saved " + f"source benchmark config is missing: {source_path}", + file=sys.stderr, + ) + sys.exit(2) + os.environ["HYPERLOOM_BENCHMARK_CONFIG"] = str(source_path) + print(f" restored AgentX source config: {source_path} (baseline not yet accepted)") + try: + restored_agentx_pins = _restore_agentx_runtime_pins_from_state( + state, + operator_pins=resume_operator_agentx_pins, + ) + except ValueError as exc: + print(f"ERROR: cannot resume this AgentX session -- {exc}", file=sys.stderr) + sys.exit(2) + if restored_agentx_pins: + print(" restored native AgentX session pins: " + ", ".join(sorted(restored_agentx_pins))) + resume_preflight_args = argparse.Namespace(**vars(args)) + resume_preflight_args.framework = state.framework or getattr(args, "framework", "") + _preflight_agentx_backend(resume_preflight_args, state) + _apply_agentx_budget_profile(args) + resolve_benchmark_timeouts() + _latency_conflict = latency_budget_scope_error( state.framework, getattr(args, "max_latency_ms", None) ) or latency_budget_resume_conflict(state, getattr(args, "max_latency_ms", None)) @@ -1861,6 +2810,11 @@ async def _run_optimize(args: argparse.Namespace) -> int: # Resolve workload knobs with the resumed state as the fallback source (explicit --isl/--conc/... on this # resume still win), then project the resolved values into env so resume sees the same workload contract (not # YAML defaults). + try: + _enforce_agentx_resume_workload(args, state) + except ValueError as exc: + print(f"ERROR: cannot resume this AgentX session -- {exc}", file=sys.stderr) + sys.exit(2) _resolve_workload_knobs(args, state) _resume_max_model_len = getattr(args, "max_model_len", None) or getattr(state, "max_model_len", 0) or 0 for env_name, val in ( @@ -2054,85 +3008,6 @@ async def _run_optimize(args: argparse.Namespace) -> int: print(line) # No resume backfill needed for roofline (roofline_snapshots restored by SharedState.from_dict). else: - # Resolve model path: --model > $MODEL_PATH; fail fast rather than silently use the YAML hardcoded model. - if not args.model: - args.model = os.environ.get("MODEL_PATH") or "" - if not args.model: - print( - "ERROR: model is required. Pass --model or set " - "MODEL_PATH env (or use --resume-from to " - "continue an existing session).", - file=sys.stderr, - ) - sys.exit(2) - # Re-export so subprocess executors inject the resolved model into the Magpie YAML, not its hardcoded model. - from hyperloom.common.model_paths import resolve_serving_model_path - - os.environ["MODEL_PATH"] = resolve_serving_model_path(str(args.model)) or str(args.model) - - # Quantization prelude (one-shot, before any session/baseline work): if --quantize was passed, quantize the - # source model now and rewrite args.model to the exported quantized model. - await _run_quantization_prelude(args) - - # Resolve framework: --framework > $FRAMEWORK > "sglang" (session-wide; no framework mixing). - framework = _registered_framework_or_exit( - args.framework or os.environ.get("FRAMEWORK", ""), - hint="set $FRAMEWORK accordingly or pass --framework", - ) - _enforce_expected_framework(framework) - os.environ["FRAMEWORK"] = framework - print(f"Framework : {framework}") - _apply_operator_supplied_paths(args, framework) - - # B3: --framework atom auto-tightens incompatible phases (see _apply_atom_auto_tighten). - if framework == "atom": - _apply_atom_auto_tighten(args) - - # Resolve real target GPU: probe > --gpu-type hint; probe wins to catch wrong-host typos that corrupt KB. - user_specified = _resolve_amd_gpu_type(args.gpu_type) or "" - if _should_remote_probe_gpu(args): - from ..multi_node._internal.gpu_probe import remote_autodetect_gpu_type - - probed = remote_autodetect_gpu_type() or "" - if probed: - print(f"GPU probe : {probed} (remote {(args.mn_backend or 'rayjob').lower()})") - else: - probed = _autodetect_gpu_type() or "" - gpu_type, gpu_warnings = _resolve_gpu_type( - user_specified=user_specified, - probed=probed, - ) - for line in gpu_warnings: - print(line, file=sys.stderr) - if probed and not user_specified: - print(f"GPU type : {gpu_type} (auto-detected)") - runner_gpu_type = _gpu_runner_type(gpu_type) - if gpu_type and runner_gpu_type != gpu_type: - print( - f"WARN: {gpu_type} uses {runner_gpu_type} as Magpie " - f"runner_type (same gfx942/CDNA3 arch; Magpie has no " - f"sglang_{gpu_type}.sh / vllm_{gpu_type}.sh yet)", - file=sys.stderr, - ) - args.gpu_type = gpu_type or None - if runner_gpu_type: - os.environ["TARGET_GPU_TYPE"] = gpu_type - os.environ["GPU_TYPE"] = runner_gpu_type - print(f"GPU type : {gpu_type}") - print(f"Magpie runner : {runner_gpu_type} (will inject runner_type into Magpie YAML)") - else: - os.environ.pop("TARGET_GPU_TYPE", None) - os.environ.pop("GPU_TYPE", None) - args.gpu_type = None - print("GPU type : (Magpie will auto-detect)") - _require_custom_entrypoint(framework, gpu_type=runner_gpu_type or gpu_type) - - # Runs here, not in _preflight, because the question it asks -- will provenance be able to name the ISA? -- is - # unanswerable until args.gpu_type is final. - _check_gfx_arch_resolvable(args.gpu_type) - - # Resolve workload knobs (flag > default; no resume state on a fresh launch) so ISL/OSL/CONC/TP/EP are - # authoritative reals before MAX_MODEL_LEN auto-derivation and env projection (issue #903). _resolve_workload_knobs(args) # MAX_MODEL_LEN is operator-overridable. max_model_len, max_model_len_source = _resolve_run_max_model_len(args) diff --git a/src/hyperloom/inference_optimizer/cli/agentx_source.py b/src/hyperloom/inference_optimizer/cli/agentx_source.py new file mode 100644 index 0000000000..966eb28b57 --- /dev/null +++ b/src/hyperloom/inference_optimizer/cli/agentx_source.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Build a fresh native AgentX source from resolved operator workload inputs.""" + +from __future__ import annotations + +import argparse +import hashlib +import os +import re +import tempfile +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import yaml + +from hyperloom.common.agentx_mode import native_agentx_enabled, native_agentx_session + + +def _source_benchmark() -> dict[str, Any]: + source = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "").strip() + if not source: + return {} + content = Path(source).read_bytes() + expected = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG_SHA256", "").strip() + if not expected or hashlib.sha256(content).hexdigest() != expected: + raise ValueError(f"benchmark config changed after initial validation: {source}") + parsed = yaml.safe_load(content) + benchmark = parsed.get("benchmark") if isinstance(parsed, Mapping) else None + if not isinstance(benchmark, dict): + raise ValueError(f"benchmark config must contain a benchmark mapping: {source}") + if "agentx" in benchmark and not native_agentx_enabled(benchmark["agentx"]): + raise ValueError("HYPERLOOM_AGENTX=1 conflicts with disabled benchmark.agentx") + if "agentx" not in benchmark: + benchmark.pop("benchmark_script", None) + return benchmark + + +def _canonical_model(args: argparse.Namespace, benchmark: Mapping[str, Any]) -> str: + model = str(os.environ.get("AGENTX_MODEL_ID") or benchmark.get("model") or getattr(args, "model", "") or "").strip() + if not re.fullmatch(r"[\w.-]+/[\w.-]+", model) or Path(model).expanduser().exists(): + raise ValueError( + "native AgentX requires a canonical model id for recipe selection; " + "pass --model or set AGENTX_MODEL_ID when serving a local checkpoint" + ) + return model + + +def _workload_envs(args: argparse.Namespace, benchmark: dict[str, Any], model: str) -> dict[str, Any]: + """Preserve source inputs and project only explicit topology/context overrides.""" + envs = dict(benchmark.get("envs") or {}) + envs.update(CONC=int(args.conc), AGENTX_MODEL_ID=model) + for attr, key in (("tp", "TP"), ("ep", "EP_SIZE"), ("max_model_len", "MAX_MODEL_LEN")): + value = getattr(args, attr, None) + if value is not None: + envs[key] = int(value) + model_path = str(os.environ.get("MODEL_PATH") or args.model or "") + if Path(model_path).expanduser().exists(): + envs["MODEL_PATH"] = str(Path(model_path).expanduser().resolve()) + image = os.environ.get("HYPERLOOM_IMAGE", "").strip() + if image: + source_image = str(benchmark.get("docker_image") or "").strip() + if source_image and source_image != image: + raise ValueError("HYPERLOOM_IMAGE conflicts with benchmark.docker_image") + benchmark["docker_image"] = image + return envs + + +def prepare_native_agentx_source(args: argparse.Namespace) -> bool: + """Snapshot fresh managed AgentX inputs without modifying an operator's YAML.""" + if getattr(args, "resume_from", None) or not native_agentx_session(): + return False + benchmark = _source_benchmark() + model = _canonical_model(args, benchmark) + runner = str(getattr(args, "gpu_type", "") or "").strip().lower() + if not runner: + raise ValueError("native AgentX requires a detected GPU or explicit --gpu-type to resolve its recipe") + configured_runner = str(benchmark.get("runner_type") or "").strip().lower() + if configured_runner and configured_runner != runner: + raise ValueError(f"benchmark.runner_type {configured_runner!r} conflicts with detected GPU {runner!r}") + raw_agentx = benchmark.get("agentx") + agentx = dict(raw_agentx) if isinstance(raw_agentx, Mapping) else {} + agentx["enabled"] = True + agentx.setdefault("launch_overrides", {"version": 1}) + benchmark.update( + agentx=agentx, + model=model, + framework=str(args.framework or os.environ.get("FRAMEWORK") or ""), + precision=str(args.precision), + runner_type=runner, + ) + benchmark["envs"] = _workload_envs(args, benchmark, model) + with tempfile.NamedTemporaryFile(mode="w", prefix="hyperloom-agentx-", suffix=".yaml", delete=False) as stream: + yaml.safe_dump({"benchmark": benchmark}, stream, sort_keys=False) + source = Path(stream.name) + args._generated_agentx_source = str(source) + os.environ["HYPERLOOM_BENCHMARK_CONFIG"] = str(source) + os.environ["HYPERLOOM_BENCHMARK_CONFIG_SHA256"] = hashlib.sha256(source.read_bytes()).hexdigest() + os.environ["AGENTX_MODEL_ID"] = model + return True diff --git a/src/hyperloom/inference_optimizer/cli/bootstrap.py b/src/hyperloom/inference_optimizer/cli/bootstrap.py index 7c61186ee5..fd5f557aab 100644 --- a/src/hyperloom/inference_optimizer/cli/bootstrap.py +++ b/src/hyperloom/inference_optimizer/cli/bootstrap.py @@ -6,6 +6,7 @@ from __future__ import annotations import argparse +import hashlib import json import logging import os @@ -15,6 +16,7 @@ from typing import Any from hyperloom.common.coerce import to_unix +from hyperloom.common.agentx_workload import agentic_backend from hyperloom.common.env import forge_explicitly_enabled from hyperloom.common.gpu_partition import published_shape from hyperloom.common.perf_metric import is_agentx_mode @@ -58,9 +60,8 @@ def resolve_model_display_name(args: argparse.Namespace) -> str: # Bump when a change makes previously recorded AgentX measurements incomparable. -# The MLPerf client is a different workload, but it is opt-in: the epoch stays -# so aiperf sessions remain resumable. The backend name is what resume compares. -AGENTX_MEASUREMENT_EPOCH = 1 +AGENTX_MEASUREMENT_EPOCH = 4 +LEGACY_AGENTX_MEASUREMENT_EPOCH = 1 def resolve_framework_version(args: argparse.Namespace) -> str: @@ -112,8 +113,15 @@ def seed_grading(framework: str, benchmark_mode: str) -> dict[str, Any]: def agentx_state_is_stale(state: Any) -> str: """Return why a resumed session's AgentX state is unusable, or ``\"\"``.""" - want_mode = "agentx" if _agentx_enabled() else "synthetic" had_mode = str(getattr(state, "benchmark_mode", "") or "") + raw_mode = os.environ.get("HYPERLOOM_AGENTX") + # A bare --resume-from in a fresh shell carries no mode declaration. In + # that case the persisted session is authoritative; only an explicitly + # supplied HYPERLOOM_AGENTX value is a request to switch modes. + if raw_mode is None or not raw_mode.strip(): + want_mode = had_mode or "synthetic" + else: + want_mode = "agentx" if _agentx_enabled() else "synthetic" if had_mode and had_mode != want_mode: return ( f"session was measured in benchmark_mode={had_mode!r} but this run is " @@ -121,18 +129,32 @@ def agentx_state_is_stale(state: Any) -> str: "sets of measurements would overwrite each other" ) if want_mode == "agentx": + from hyperloom.common.agentx_mode import native_agentx_session + had_epoch = int(getattr(state, "agentx_epoch", 0) or 0) - if had_epoch != AGENTX_MEASUREMENT_EPOCH: + backend = str(getattr(state, "agentx_backend", "") or "").strip().lower() + expected_backends = {"native"} if native_agentx_session(state) else {"legacy", "aiperf", "mlperf"} + if backend and backend not in expected_backends: + return f"session AgentX backend {backend!r} conflicts with its measurement epoch {had_epoch}" + if had_epoch == LEGACY_AGENTX_MEASUREMENT_EPOCH: + from hyperloom.common.agentx_mode import config_enables_native_agentx + + accepted = str(getattr(state, "baseline_config_path", "") or "").strip() + if accepted and config_enables_native_agentx(accepted): + return "session epoch 1 conflicts with its accepted native AgentX baseline" + if had_epoch not in {LEGACY_AGENTX_MEASUREMENT_EPOCH, 2, 3, AGENTX_MEASUREMENT_EPOCH}: return ( - f"session carries AgentX epoch {had_epoch}, this build measures " - f"epoch {AGENTX_MEASUREMENT_EPOCH}; the recorded results describe " + f"session carries unsupported AgentX epoch {had_epoch}; the recorded results describe " "a different workload and cannot anchor or be compared against" ) from hyperloom.common.agentx_workload import agentic_backend - # Sessions recorded before the backend was persisted are aiperf. - had_backend = str(getattr(state, "agentx_backend", "") or "") or "aiperf" - want_backend = agentic_backend() + # A bare resume keeps the recorded client, including the public MLPerf backend. + had_backend = ("native" if had_epoch >= 2 else "aiperf") if backend in {"", "legacy"} else backend + explicit_backend = str(os.environ.get("HYPERLOOM_AGENTIC_BACKEND", "") or "").strip() + want_backend = agentic_backend() if explicit_backend else had_backend + if had_backend == "native" and want_backend == "aiperf": + want_backend = "native" if had_backend != want_backend: return ( f"session was measured with agentic backend {had_backend!r} but this " @@ -333,13 +355,41 @@ def _int_arg(arg_name: str, default: int) -> int: # Canonical model identity (prefers the quantize prelude's pinned source name). _model_identity = resolve_model_display_name(args) - benchmark_mode = "agentx" if _agentx_enabled() else "synthetic" - if _agentx_enabled(): - from hyperloom.common.agentx_workload import agentic_backend + from hyperloom.common.agentx_mode import native_agentx_session + + native_agentx = native_agentx_session() + benchmark_mode = "agentx" if native_agentx or _agentx_enabled() else "synthetic" + _agentx_runtime_pins: dict[str, str] = {} + _benchmark_source_config_path = "" + if native_agentx: + from hyperloom.inference_optimizer.agentx.native import ( + AGENTX_RUNTIME_PIN_NAMES, + ) - agentx_backend = agentic_backend() - else: - agentx_backend = "" + for _pin_name in AGENTX_RUNTIME_PIN_NAMES: + _pin_value = os.environ.get(_pin_name, "").strip() + if _pin_value: + if _pin_name == "INFERENCEX_PATH": + _pin_value = str(Path(_pin_value).expanduser().resolve()) + _agentx_runtime_pins[_pin_name] = _pin_value + _source_config = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "").strip() + if _source_config: + _source_path = Path(_source_config).expanduser().resolve() + if not _source_path.is_file(): + raise FileNotFoundError( + f"AgentX source benchmark config disappeared before session seed: {_source_path}" + ) + _source_bytes = _source_path.read_bytes() + _expected_source_hash = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG_SHA256", "").strip() + _actual_source_hash = hashlib.sha256(_source_bytes).hexdigest() + if not _expected_source_hash or _actual_source_hash != _expected_source_hash: + raise ValueError(f"AgentX source benchmark config changed after finalization: {_source_path}") + _snapshot_path = session_dir / "benchmark.source.yaml" + _snapshot_path.write_bytes(_source_bytes) + _benchmark_source_config_path = str(_snapshot_path) + os.environ["HYPERLOOM_BENCHMARK_CONFIG"] = str(_snapshot_path) + if str(getattr(args, "_generated_agentx_source", "")) == str(_source_path): + _source_path.unlink() state = SharedState( session_id=session_id, claw_session_id=(os.environ.get("CLAW_SESSION_ID") or "").strip(), @@ -419,20 +469,28 @@ def _int_arg(arg_name: str, default: int) -> int: static_recon_enabled=bool(getattr(args, "static_recon", True)), target_advisory_enabled=bool(getattr(args, "target_advisory", True)), recipe_sediment_enabled=bool(getattr(args, "recipe_sediment", True)), - # SWEEP-phase concurrency sweep: defaults OFF under AgentX because each - # rung is a 3600s window and the session grades at a fixed CONC. - # Pass --enable-conc-sweep explicitly to override. conc_sweep_enabled=( - not is_agentx_mode(benchmark_mode) if args.enable_conc_sweep is None else args.enable_conc_sweep + bool(getattr(args, "enable_conc_sweep", None)) + if getattr(args, "enable_conc_sweep", None) is not None + else not is_agentx_mode(benchmark_mode) ), benchmark_mode=benchmark_mode, - agentx_epoch=AGENTX_MEASUREMENT_EPOCH if _agentx_enabled() else 0, - agentx_backend=agentx_backend, + agentx_backend=("native" if native_agentx else agentic_backend() if benchmark_mode == "agentx" else ""), + agentx_epoch=( + AGENTX_MEASUREMENT_EPOCH + if native_agentx + else LEGACY_AGENTX_MEASUREMENT_EPOCH + if benchmark_mode == "agentx" + else 0 + ), + agentx_runtime_pins=_agentx_runtime_pins, + benchmark_source_config_path=_benchmark_source_config_path, grading=seed_grading(os.environ.get("FRAMEWORK", "sglang"), benchmark_mode), conc_sweep_concs=_parse_conc_sweep_concs(args, benchmark_mode), conc_sweep_total_budget_sec=int( getattr(args, "conc_sweep_total_budget_sec", 9000) or 0, ), + active_inferencex_path=_agentx_runtime_pins.get("INFERENCEX_PATH", ""), ) state.save(session_dir) return state @@ -480,12 +538,10 @@ def _print_final_summary( print(f" stop_reason : {stop_reason}") print(f" session_id : {state.session_id}") print(f" model : {state.model_name}") - from .. import framework_registry + from ..performance_display import format_session_metric - print( - f" baseline : {framework_registry.format_primary_metric(getattr(state, 'framework', ''), state.baseline_tput)}" - ) - if stop_reason == "baseline_failed": + print(f" baseline : {format_session_metric(state, state.baseline_tput)}") + if session_dir is not None and stop_reason == "baseline_failed": failure_summary = _read_failure_summary(session_dir) if failure_summary and failure_summary.get("root_cause"): print( diff --git a/src/hyperloom/inference_optimizer/cli/parser.py b/src/hyperloom/inference_optimizer/cli/parser.py index e96982fffe..76777d3b40 100644 --- a/src/hyperloom/inference_optimizer/cli/parser.py +++ b/src/hyperloom/inference_optimizer/cli/parser.py @@ -194,9 +194,11 @@ def _build_parser() -> argparse.ArgumentParser: "-m", type=Path, default=None, - help="Model path (required for new runs; ignored when " - "--resume-from is set — model is read from manifest.json/" - "state.json)", + help="Local model path or Hugging Face id. Required for a fresh run " + "unless --benchmark-config supplies benchmark.model. For native " + "AgentX, benchmark.model remains the canonical recipe id while a " + "local --model overrides only MODEL_PATH. Ignored with --resume-from, " + "which restores the saved model.", ) opt.add_argument( "--quantize", @@ -277,8 +279,10 @@ def _build_parser() -> argparse.ArgumentParser: "--framework", choices=list(framework_registry.names()), default=None, - help="Inference framework to benchmark / optimize. Resolution order: " - "--framework > sglang (default). Selection is " + help="Inference framework to benchmark / optimize. With " + "--benchmark-config, benchmark.framework supplies the value when this " + "flag is omitted and a conflicting flag is rejected; otherwise the " + "default is sglang. Selection is " "session-wide; mixing frameworks in a single session is not " "supported: --resume-from uses the framework the session was " "created with, and refuses a --framework that differs. NOTE: --framework atom is single-node-only " @@ -312,6 +316,22 @@ def _build_parser() -> argparse.ArgumentParser: "that gate is the only correctness signal, and a missing one scores " "zero, so every candidate is rejected.", ) + opt.add_argument( + "--benchmark-config", + default=None, + metavar="YAML", + help=( + "Magpie benchmark YAML for the initial baseline. The resolved path " + "is reused by downstream rounds. A source config containing " + "`benchmark.agentx: enable` selects native AgentX measurement; " + "HYPERLOOM_AGENTX does not need to be exported separately. For an " + "ambiguous recipe point, use benchmark.agentx.selector; " + "benchmark.envs.TP is not a selector. Fresh " + "launches only. A resume restores the accepted materialized config " + "and runtime pins, or the snapshotted source config when no baseline " + "was accepted yet." + ), + ) opt.add_argument( "--nodes", type=int, @@ -349,26 +369,31 @@ def _build_parser() -> argparse.ArgumentParser: "--tp", type=int, default=None, - help="Tensor parallel size. Pass `--tp N` directly from the prompt's " - f"Environment block. Default: {DEFAULT_TP}.", + help="Tensor-parallel size outside native AgentX (default: " + f"{DEFAULT_TP}). Native AgentX instead resolves the physical GPU count " + "as recipe TP x PP x PCP; omit this flag to use that count, or pass it " + "as an exact consistency assertion. Custom native workloads require " + "an explicit single-node TP.", ) opt.add_argument( "--conc", type=_positive_int_arg, default=None, - help="Magpie client concurrency cap (max in-flight requests). " - "Pass `--conc N` directly from the prompt. Use " - f"--conc-sweep-concs for a concurrency ladder. Default: {DEFAULT_CONC}.", + help="Magpie client concurrency cap (max in-flight requests). It may " + "also come from benchmark.envs.CONC. Native AgentX requires the fixed " + "value to exist in the selected recipe or explicit custom workload. " + "Native optimization sessions can sweep supported concurrency points. The default is " + f"{DEFAULT_CONC} and --conc-sweep-concs configures a ladder.", ) opt.add_argument( "--max-model-len", dest="max_model_len", type=_positive_int_arg, default=None, - help="Explicit server-facing MAX_MODEL_LEN. Resolution: " - "--max-model-len > auto(ISL+OSL+headroom, " - "clamped to native context). Explicit values are preserved and " - "exported into the materialized Magpie YAML.", + help="Explicit server-facing MAX_MODEL_LEN. Synthetic resolution is --max-model-len > " + "auto(ISL+OSL+headroom, clamped to native context). Native InferenceX " + "AgentX registered recipes retain their context policy; a custom " + "workload validates this limit against the model's native context.", ) opt.add_argument( "--server-args", @@ -378,6 +403,8 @@ def _build_parser() -> argparse.ArgumentParser: help="Framework server args to apply in every phase. Routed through " "the framework-specific EXTRA_*_ARGS env in Magpie YAMLs " "(EXTRA_VLLM_ARGS / EXTRA_SGLANG_ARGS / EXTRA_ATOM_ARGS). " + "New native AgentX sessions use the formal launch-override contract; " + "persisted epoch-2 sessions retain their frozen launcher. " 'Example: --server-args "--kv-cache-dtype fp8_e4m3 ' '--gpu-memory-utilization 0.85".', ) @@ -385,12 +412,14 @@ def _build_parser() -> argparse.ArgumentParser: "--ep", type=int, default=None, - help="Expert-parallel size for MoE inference. 1 (default) keeps " + help="Expert-parallel size for MoE inference outside native AgentX. " + "1 (default) keeps " "experts sharded by TP (legacy behaviour). >=2 enables true " "expert parallelism: sglang adds `--expert-parallel-size N`, " "vllm adds `--enable-expert-parallel`. Typical: EP=TP for " - "DSr1/DSv3 on multi-node. Default: 1. " - "EP > TP is rejected at server-restart time.", + "DSr1/DSv3 on multi-node. EP > TP is rejected at server-restart time. " + "Native AgentX resolves EP from the recipe when omitted and treats an " + "explicit value as an exact consistency assertion.", ) opt.add_argument( "--pd-mode", @@ -487,7 +516,9 @@ def _build_parser() -> argparse.ArgumentParser: # for this many hours" from "the operator said nothing", and a default # would make an explicit value indistinguishable from absence. default=None, - help=f"Wall-clock budget in hours (default {DEFAULT_MAX_HOURS})", + help=f"Wall-clock budget in hours (default {DEFAULT_MAX_HOURS}). " + "Set it explicitly for native AgentX: model load, warmup, drain, and " + "the canonical 3600-second measurement commonly exceed the default.", ) opt.add_argument( "--extend-hours", @@ -510,8 +541,20 @@ def _build_parser() -> argparse.ArgumentParser: "min(120, max_hours * 60 * 0.02). Pass 0 to disable closing phase." ), ) - opt.add_argument("--isl", type=int, default=None, help=f"Input sequence length (default {DEFAULT_ISL})") - opt.add_argument("--osl", type=int, default=None, help=f"Output sequence length (default {DEFAULT_OSL})") + opt.add_argument( + "--isl", + type=int, + default=None, + help=f"Input sequence length (default {DEFAULT_ISL}). Native AgentX " + "measurement uses the trace corpus distribution instead.", + ) + opt.add_argument( + "--osl", + type=int, + default=None, + help=f"Output sequence length (default {DEFAULT_OSL}). Native AgentX " + "measurement uses the trace corpus distribution instead.", + ) opt.add_argument( "--profile-osl", dest="profile_osl", @@ -542,7 +585,13 @@ def _build_parser() -> argparse.ArgumentParser: "back. Omit the flag to leave the baseline unchanged." ), ) - opt.add_argument("--precision", type=str, default=None, help=f"Model precision (default {DEFAULT_PRECISION})") + opt.add_argument( + "--precision", + type=str, + default=None, + help=f"Model precision (default {DEFAULT_PRECISION}); " + "benchmark.precision supplies it when --benchmark-config is used.", + ) opt.add_argument( "--framework-version", dest="framework_version", @@ -979,7 +1028,10 @@ def _build_parser() -> argparse.ArgumentParser: "to use plain ``profile`` instead (lighter — captures the " "trace only, skips trace_analyze). Behaviour is otherwise " "identical (same idempotency keys, same pending-task " - "dispatch gate, same watermark anchor update).", + "dispatch gate, same watermark anchor update). Native AgentX " + "measurement itself produces no PyTorch trace; this analysis uses the " + "generic-server compatibility profile and is diagnostic, not " + "recipe-identical.", ) opt.add_argument( "--research-scout", @@ -1049,20 +1101,23 @@ def _build_parser() -> argparse.ArgumentParser: help="Run a post-optimization concurrency sweep (baseline vs " "current_best across CONC) and write " "reports/conc_sweep_summary.json + conc_sweep_raw.csv. " - "On by default, off under AgentX (each rung is a 3600s window); " - "force either way with --enable-conc-sweep / --no-enable-conc-sweep.", + "On by default for synthetic workloads, off under AgentX. " + "Native AgentX keeps concurrency fixed and rejects an explicit enable; " + "legacy AgentX may enable it explicitly.", ) opt.add_argument( "--conc-sweep-concs", dest="conc_sweep_concs", type=str, default=None, - help="Comma-separated CONC ladder for --enable-conc-sweep. Ordered " + help="Comma-separated CONC ladder for --enable-conc-sweep outside " + "native AgentX. Ordered " "high-to-low internally for single-server arm reuse, so the order given " - "does not matter. Defaults to the ladder for the workload: " - "256,128,64,32,16,8,4,2 synthetic, 1,4,8,10,14,20,28 under " - "HYPERLOOM_AGENTX (an agentic request carries orders of magnitude more " - "prompt, so the same card saturates far lower).", + "does not matter. The synthetic default is " + "256,128,64,32,16,8,4,2; legacy AgentX uses 1,4,8,10,14,20,28. " + "Native AgentX rejects an explicitly enabled " + "sweep until the pinned launcher exposes an optimizer-argv hook; this " + "option alone never enables a sweep.", ) opt.add_argument( "--conc-sweep-total-budget-sec", diff --git a/src/hyperloom/inference_optimizer/cli/preflight.py b/src/hyperloom/inference_optimizer/cli/preflight.py index 64ecdb420c..e4ab7c1689 100644 --- a/src/hyperloom/inference_optimizer/cli/preflight.py +++ b/src/hyperloom/inference_optimizer/cli/preflight.py @@ -51,6 +51,7 @@ detect_gfx_arch, ) from hyperloom.common.timeutil import now_iso +from hyperloom.inference_optimizer.magpie_health import magpie_health_code from .credentials import ( _is_stale_proxy_url, @@ -1810,7 +1811,33 @@ def _count(p: Path) -> int: _INFERENCEX_REPO_DEFAULT = "https://github.com/SemiAnalysisAI/InferenceX.git" # MUST stay in lockstep with INFERENCEX_REF in assets/install.sh. -_INFERENCEX_REF_DEFAULT = "3d5581562f643f9bdeb8410cd924e2c70906c966" +_INFERENCEX_REF_DEFAULT = "408c015be4b22d14c69518643609669405507077" +_MAGPIE_REF_DEFAULT = "d80eb4d3dad7fabe01ce81d049e2983adf2c86dd" + + +def _native_agentx_preflight_requested( + args: argparse.Namespace | None, +) -> bool: + """Detect native AgentX before preflight is allowed to mutate checkouts.""" + from hyperloom.common.agentx_mode import native_agentx_session + + resume_from = str(getattr(args, "resume_from", "") or "").strip() + if not resume_from: + return native_agentx_session() + state_path = Path(resume_from).expanduser() / "state.json" + try: + state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, TypeError, ValueError): + return False + if not native_agentx_session(state): + return False + pins = state.get("agentx_runtime_pins") + if isinstance(pins, dict): + for name in ("MAGPIE_REF", "INFERENCEX_REF"): + saved = str(pins.get(name) or "").strip() + if saved and not os.environ.get(name, "").strip(): + os.environ[name] = saved + return True def _inferencex_head_sha(path: Path | str) -> str: @@ -1830,19 +1857,59 @@ def _inferencex_head_sha(path: Path | str) -> str: def _inferencex_ref_matches(path: Path | str, ref: str) -> bool: """Whether the checkout at ``path`` is at ``ref``.""" if not re.fullmatch(r"[0-9a-fA-F]{7,40}", ref or ""): + # Legacy/profile-compatible checkouts may intentionally follow a branch + # or tag, which cannot be compared to HEAD without a network lookup. return True head = _inferencex_head_sha(path) if not head: + # Preserve support for a tarball/source drop without git metadata. The + # native AgentX path applies a stricter check below. return True return head.startswith(ref.lower()) or ref.lower().startswith(head) -def _inferencex_checkout_ok(path: Path | str, *, ref: str | None = None) -> bool: +def _inferencex_checkout_ok( + path: Path | str, + *, + ref: str | None = None, + require_agentx_submodule: bool = False, +) -> bool: """True when ``path`` is a usable InferenceX checkout at the expected ref.""" + from ..agentx.managed import project_root + + try: + path = project_root(path) + except ValueError: + return False if not (Path(path) / "benchmarks" / "benchmark_lib.sh").is_file(): return False if ref is None: ref = os.environ.get("INFERENCEX_REF") or _INFERENCEX_REF_DEFAULT + if require_agentx_submodule: + # Native AgentX executes upstream recipe code verbatim, so unlike the + # legacy compatibility path it must fail closed on an unverifiable pin. + if not re.fullmatch(r"[0-9a-fA-F]{40}", ref or ""): + return False + head = _inferencex_head_sha(path) + if not head or head.lower() != ref.lower(): + return False + root = Path(path) + aiperf = root / "utils" / "aiperf" + if not (aiperf / "pyproject.toml").is_file(): + return False + gitlink = subprocess.run( + ["git", "-C", str(root), "ls-tree", "HEAD", "utils/aiperf"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + fields = (gitlink.stdout or "").strip().split() + if gitlink.returncode != 0 or len(fields) < 3 or fields[0] != "160000": + return False + submodule_head = _inferencex_head_sha(aiperf) + if not submodule_head or submodule_head.lower() != fields[2].lower(): + return False if not ref: return True return _inferencex_ref_matches(path, ref) @@ -1854,6 +1921,11 @@ def _inferencex_dest_name(ref: str) -> str: return f"InferenceX@{slug}" +def _native_inferencex_dest_name(ref: str) -> str: + """Keep native AgentX away from generic compatibility-patched checkouts.""" + return f"{_inferencex_dest_name(ref)}-agentx" + + def _ensure_eval_concurrency_compat(magpie_path: str, inferencex_path: str) -> bool: """Scrub the fatal ``--concurrent-requests`` eval flag from the resolved trees.""" try: @@ -1930,7 +2002,11 @@ def _ensure_client_trust_compat(magpie_path: str) -> bool: return ok -def _clone_inferencex(dest: Path) -> str | None: +def _clone_inferencex( + dest: Path, + *, + initialize_agentx_submodule: bool = False, +) -> str | None: """Clone InferenceX into ``dest`` (writable), pinned to INFERENCEX_REF.""" repo = os.environ.get("INFERENCEX_REPO") or _INFERENCEX_REPO_DEFAULT ref = os.environ.get("INFERENCEX_REF") or _INFERENCEX_REF_DEFAULT @@ -1955,12 +2031,58 @@ def _clone_inferencex(dest: Path) -> str | None: check=True, timeout=600, ) + from ..agentx.managed import project_root + + project = project_root(dest) + submodule = str((project / "utils/aiperf").relative_to(dest)) + if initialize_agentx_submodule: + subprocess.run( + ["git", "-C", dest_str, "submodule", "sync", "--", submodule], + check=True, + timeout=60, + ) + try: + subprocess.run( + [ + "git", + "-C", + dest_str, + "submodule", + "update", + "--init", + "--depth", + "1", + "--", + submodule, + ], + check=True, + timeout=600, + ) + except subprocess.CalledProcessError: + subprocess.run( + [ + "git", + "-C", + dest_str, + "submodule", + "update", + "--init", + "--", + submodule, + ], + check=True, + timeout=600, + ) # ref="" : the tree was just checked out at `ref` by construction, so re-deriving the pin here would only # re-read what we wrote. - if not _inferencex_checkout_ok(dest, ref=""): + if not _inferencex_checkout_ok( + dest, + ref=ref, + require_agentx_submodule=initialize_agentx_submodule, + ): raise OSError(f"clone reported success but {dest_str} is missing benchmarks/benchmark_lib.sh") log.info("InferenceX cloned into %s at %s", dest_str, _inferencex_head_sha(dest) or ref) - return dest_str + return str(project) except (subprocess.CalledProcessError, subprocess.TimeoutExpired, OSError) as exc: log.warning("InferenceX clone into %s failed: %s", dest_str, exc) shutil.rmtree(dest, ignore_errors=True) @@ -2193,6 +2315,16 @@ def _preflight( # After the legacy normalization, which can create the credentials a header references. _expand_custom_header_env_refs() + _native_agentx_preflight_active = _native_agentx_preflight_requested(args) + if _native_agentx_preflight_active: + from ..agentx.native import validate_native_replay_env + + try: + validate_native_replay_env(os.environ, source="environment") + except ValueError as exc: + print(f"Preflight: ERROR — {exc}", file=sys.stderr) + raise SystemExit(2) from exc + # Fail fast on missing credentials after the fallback loaders. _run_install_step( install_event, @@ -2390,24 +2522,69 @@ def _prepare_kb_install_step() -> dict[str, Any]: magpie_python = benchmark_python magpie_installed = False magpie_spec: str | None = None + magpie_ref = os.environ.get("MAGPIE_REF") or _MAGPIE_REF_DEFAULT + os.environ["MAGPIE_REF"] = magpie_ref + if _native_agentx_preflight_active and not re.fullmatch(r"[0-9a-fA-F]{40}", magpie_ref): + print( + "Preflight: ERROR — native AgentX requires MAGPIE_REF to be one " + f"immutable 40-character commit SHA; got {magpie_ref!r}.", + file=sys.stderr, + ) + raise SystemExit(2) + health_code = magpie_health_code(native_agentx=_native_agentx_preflight_active) try: if not _magpie_backend_active: print(f"Preflight: benchmark backend is {benchmark_backend!r}; skipping Magpie install/import") check = None else: - check = subprocess.run([magpie_python, "-c", "import Magpie"], capture_output=True) + check = subprocess.run( + [ + magpie_python, + "-c", + health_code, + magpie_ref, + ], + capture_output=True, + ) if _magpie_backend_active and check is not None and check.returncode != 0: magpie_repo = os.environ.get("MAGPIE_REPO", "https://github.com/AMD-AGI/Magpie.git") - magpie_ref = os.environ.get("MAGPIE_REF", "e6833b8183c6c41adf6038252337550876ca0433") magpie_spec = os.environ.get( "MAGPIE_PACKAGE_SPEC", f"magpie-eval @ git+{magpie_repo}@{magpie_ref}", ) - print(f"Preflight: Magpie not importable; installing {magpie_spec} ...") + unavailable = ( + "native AgentX capability unavailable" if _native_agentx_preflight_active else "not importable" + ) + print(f"Preflight: Magpie {unavailable}; installing {magpie_spec} ...") subprocess.run( [magpie_python, "-m", "pip", "install", "--quiet", *pip_extra, magpie_spec], check=True, ) + health = subprocess.run( + [magpie_python, "-c", health_code, magpie_ref], + capture_output=True, + ) + if health.returncode != 0: + # A VCS package at the right commit can still contain local + # compatibility patches; pip otherwise keeps that installation. + subprocess.run( + [ + magpie_python, + "-m", + "pip", + "install", + "--quiet", + "--force-reinstall", + "--no-deps", + *pip_extra, + magpie_spec, + ], + check=True, + ) + subprocess.run( + [magpie_python, "-c", health_code, magpie_ref], + check=True, + ) magpie_installed = True print("Preflight: Magpie installed OK") if _magpie_backend_active and not os.environ.get("MAGPIE_PATH", "").strip(): @@ -2450,6 +2627,17 @@ def _prepare_kb_install_step() -> dict[str, Any]: ) # 3. InferenceX — required for GSM8K accuracy eval; lm-eval deps auto-install at runtime via benchmark_lib.sh. + _want_ref = os.environ.get("INFERENCEX_REF") or _INFERENCEX_REF_DEFAULT + # Make the implicit default explicit for the native execution-identity + # check and for child processes recorded in session provenance. + os.environ["INFERENCEX_REF"] = _want_ref + if _native_agentx_preflight_active and not re.fullmatch(r"[0-9a-fA-F]{40}", _want_ref): + print( + "Preflight: ERROR — native AgentX requires INFERENCEX_REF to be " + f"one immutable 40-character commit SHA; got {_want_ref!r}.", + file=sys.stderr, + ) + raise SystemExit(2) inferencex_path = os.environ.get("INFERENCEX_PATH", "").strip() inferencex_cloned = False if not inferencex_path: @@ -2463,13 +2651,22 @@ def _prepare_kb_install_step() -> dict[str, Any]: # InferenceX detection order: Magpie submodule (canonical post-install.sh) → installer's per-revision cache # checkout (InferenceX@, resolved via resolve_dep_dir so a process that did not inherit INFERENCEX_PATH # still finds it; falls back to the bare dir). - _want_ref = os.environ.get("INFERENCEX_REF") or _INFERENCEX_REF_DEFAULT - for candidate in ( - magpie_root / "InferenceX", - _resolve_dep_dir(_inferencex_dest_name(_want_ref)), - _resolve_dep_dir("InferenceX"), - ): - if _inferencex_checkout_ok(candidate): + candidates = ( + (_resolve_dep_dir(_native_inferencex_dest_name(_want_ref)),) + if _native_agentx_preflight_active + else ( + magpie_root / "InferenceX", + _resolve_dep_dir(_inferencex_dest_name(_want_ref)), + _resolve_dep_dir("InferenceX"), + ) + ) + for candidate in candidates: + candidate_ok = ( + _inferencex_checkout_ok(candidate, require_agentx_submodule=True) + if _native_agentx_preflight_active + else _inferencex_checkout_ok(candidate) + ) + if candidate_ok: if os.access(candidate, os.W_OK): inferencex_path = str(candidate) break @@ -2487,16 +2684,32 @@ def _prepare_kb_install_step() -> dict[str, Any]: ) # When no writable checkout at the pin was found, clone one ourselves. baseline cannot run without InferenceX, so # a clone failure is a hard error. - if not (inferencex_path and _inferencex_checkout_ok(inferencex_path)): + inferencex_ok = bool(inferencex_path) and ( + _inferencex_checkout_ok(inferencex_path, require_agentx_submodule=True) + if _native_agentx_preflight_active + else _inferencex_checkout_ok(inferencex_path) + ) + if not inferencex_ok: from ..session.paths import deps_cache_root as _open_source_default - _ref = os.environ.get("INFERENCEX_REF") or _INFERENCEX_REF_DEFAULT + _ref = _want_ref # Per-revision dir, matching install.sh: a shared name is what allowed a pre-bump clone to be reused forever. - dest = _open_source_default() / _inferencex_dest_name(_ref) + dest_name = ( + _native_inferencex_dest_name(_ref) if _native_agentx_preflight_active else _inferencex_dest_name(_ref) + ) + dest = _open_source_default() / dest_name print(f"Preflight: no InferenceX checkout at {_ref[:12]}; cloning into {dest} ...") - inferencex_path = _clone_inferencex(dest) + inferencex_path = _clone_inferencex( + dest, + initialize_agentx_submodule=_native_agentx_preflight_active, + ) inferencex_cloned = bool(inferencex_path) - if not (inferencex_path and _inferencex_checkout_ok(inferencex_path)): + inferencex_ok = bool(inferencex_path) and ( + _inferencex_checkout_ok(inferencex_path, require_agentx_submodule=True) + if _native_agentx_preflight_active + else _inferencex_checkout_ok(inferencex_path) + ) + if not inferencex_ok: print( "Preflight: ERROR — InferenceX checkout missing and clone " "failed. baseline cannot run without it. Set INFERENCEX_PATH " @@ -2531,7 +2744,10 @@ def _prepare_kb_install_step() -> dict[str, Any]: exc=exc, ) raise exc - # Always overwrite (not setdefault): a stale/broken INFERENCEX_PATH must not survive into the child env. + from ..agentx.managed import project_root + + inferencex_path = str(project_root(inferencex_path)) + # Always overwrite: child processes use the normalized project directory. os.environ["INFERENCEX_PATH"] = inferencex_path # A round cd's into this checkout and bash reads the benchmark script off it for the whole run, so a revocable # mount that flaps fails the round on an exit code the measurement had nothing to do with. Recording it here is @@ -2567,7 +2783,7 @@ def _prepare_kb_install_step() -> dict[str, Any]: # InferenceX itself (above), entirely outside install.sh, so the Magpie script patches install.sh applies are # applied here as well. try: - if _magpie_backend_active: + if _magpie_backend_active and not _native_agentx_preflight_active: # Trust patch first, mirroring install.sh: the eval-concurrency strip removes the very `run_eval ... # --concurrent-requests` line the legacy MI300X trust patcher matches on, so the reverse order would leave # a tree permanently unpatchable by that path. @@ -2579,7 +2795,7 @@ def _prepare_kb_install_step() -> dict[str, Any]: else: trust_ok = True concurrency_ok = True - anchors_ok = _report_inferencex_patch_anchors(inferencex_path) + anchors_ok = True if _native_agentx_preflight_active else _report_inferencex_patch_anchors(inferencex_path) except BaseException as exc: _fail_install_step( install_event, @@ -2593,8 +2809,20 @@ def _prepare_kb_install_step() -> dict[str, Any]: install_event, step_id="patch_magpie_eval_concurrency", category="patch", - status=("skipped" if not _magpie_backend_active else "applied" if patch_ok else "warned"), - skip_reason=None if _magpie_backend_active else "benchmark_backend_not_magpie", + status=( + "skipped" + if not _magpie_backend_active or _native_agentx_preflight_active + else "applied" + if patch_ok + else "warned" + ), + skip_reason=( + "benchmark_backend_not_magpie" + if not _magpie_backend_active + else "native_agentx_uses_pinned_upstream_launcher" + if _native_agentx_preflight_active + else None + ), detail={ "client_trust_compatible": trust_ok, "eval_concurrency_compatible": concurrency_ok, diff --git a/src/hyperloom/inference_optimizer/magpie_health.py b/src/hyperloom/inference_optimizer/magpie_health.py new file mode 100644 index 0000000000..020caff3ed --- /dev/null +++ b/src/hyperloom/inference_optimizer/magpie_health.py @@ -0,0 +1,36 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Magpie package health checks shared by the installer and CLI preflight.""" + +from __future__ import annotations + + +def magpie_health_code(*, native_agentx: bool) -> str: + """Keep generic imports compatible; audit native AgentX's execution contract.""" + if not native_agentx: + return "import Magpie\n" + from .agentx.native import _MAGPIE_SOURCE_IDENTITY_CODE + + return ( + _MAGPIE_SOURCE_IDENTITY_CODE + + """ +import inspect +import re +import sys +from pathlib import Path + +import Magpie +from Magpie.modes.benchmark import AgentXConfig +from Magpie.modes.benchmark.agentx import _expand_single_node_agentx_entries + +assert AgentXConfig.from_value("enable") +assert "run-eval" in inspect.getsource(_expand_single_node_agentx_entries) +expected = sys.argv[1].strip().lower() +package_root = Path(Magpie.__file__).resolve().parent +commit, _source_url = _resolve_magpie_source_identity(package_root) +_validate_magpie_execution_tree(package_root, commit) +if re.fullmatch(r"[0-9a-f]{7,40}", expected): + assert commit and (commit.startswith(expected) or expected.startswith(commit)) +""" + ) diff --git a/src/hyperloom/inference_optimizer/performance_display.py b/src/hyperloom/inference_optimizer/performance_display.py new file mode 100644 index 0000000000..438cf7ff9e --- /dev/null +++ b/src/hyperloom/inference_optimizer/performance_display.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Display throughput without changing the measurements used for grading.""" + +from __future__ import annotations + +import math +from collections.abc import Mapping +from typing import Any + +from hyperloom.common.agentx_mode import native_agentx_session + +from . import framework_registry + + +def throughput_fields( + value: float | None, + framework: str | None, + *, + native_agentx: bool = False, + gpu_count: Any = None, +) -> dict[str, Any]: + """Project a measurement into display fields, using only its recorded topology. + + Native AgentX reports aggregate output throughput. TP and EP can overlap, + so neither the process environment nor their product is a GPU count. An + absent count leaves the per-GPU field null and explicitly labels the total. + Legacy serving and scriptable measurements retain their existing units. + """ + fields = { + "throughput_tok_s_per_gpu": value, + "throughput_unit": framework_registry.throughput_unit(framework), + } + if not native_agentx: + return fields + known_count = ( + isinstance(gpu_count, (int, float)) + and not isinstance(gpu_count, bool) + and math.isfinite(gpu_count) + and gpu_count > 0 + and float(gpu_count).is_integer() + ) + fields.update( + throughput_tok_s=value, + throughput_tok_s_per_gpu=(value / gpu_count if value is not None else None) if known_count else None, + throughput_unit="tok/s/GPU" if known_count else "tok/s (aggregate)", + ) + return fields + + +def recorded_throughput(measurement: Mapping[str, Any]) -> float | None: + """Read a display measurement, preferring the measured per-GPU figure.""" + value = measurement.get("throughput_tok_s_per_gpu") + return measurement.get("throughput_tok_s") if value is None else value + + +def recorded_metric_unit(framework: str | None, measurement: Mapping[str, Any]) -> str: + """The primary display unit, including native totals without known topology.""" + if measurement.get("throughput_unit") == "tok/s (aggregate)": + return "tok/s (aggregate)" + return framework_registry.primary_metric_unit(framework) + + +def format_recorded_metric(framework: str | None, measurement: Mapping[str, Any], *, precision: int = 1) -> str: + """Format display fields without dividing a per-GPU value a second time.""" + value = recorded_throughput(measurement) + if measurement.get("throughput_unit") == "tok/s (aggregate)": + number = "n/a" if value is None else f"{value:.{precision}f}" + return f"{number} tok/s (aggregate)" + return framework_registry.format_primary_metric(framework, value, precision=precision) + + +def format_measurement_metric(framework: str | None, measurement: Mapping[str, Any], *, precision: int = 1) -> str: + """Format an executor result; its native protocol marker identifies totals.""" + fields = throughput_fields( + measurement.get("output_throughput"), + framework, + native_agentx=measurement.get("native_agentx_report") is True, + gpu_count=measurement.get("agentx_gpu_count"), + ) + return format_recorded_metric(framework, fields, precision=precision) + + +def format_session_metric( + state: Any, + value: float | None, + *, + measurement: Mapping[str, Any] | None = None, + precision: int = 1, +) -> str: + """Format a state or saved report's raw baseline/current-best throughput.""" + read = state.get if isinstance(state, Mapping) else lambda name, default=None: getattr(state, name, default) + framework = read("framework", "") + perf = measurement if measurement is not None else read("baseline_perf", None) or {} + fields = throughput_fields( + value, + framework, + native_agentx=read("native_agentx", False) is True or native_agentx_session(state), + gpu_count=perf.get("agentx_gpu_count"), + ) + return format_recorded_metric(framework, fields, precision=precision) diff --git a/src/hyperloom/inference_optimizer/tests/fixtures/inferencex_anchor_contract.json b/src/hyperloom/inference_optimizer/tests/fixtures/inferencex_anchor_contract.json index 8b286081c6..078420c716 100644 --- a/src/hyperloom/inference_optimizer/tests/fixtures/inferencex_anchor_contract.json +++ b/src/hyperloom/inference_optimizer/tests/fixtures/inferencex_anchor_contract.json @@ -1,29 +1,29 @@ { - "anchors_fingerprint": "4917512af88fd31cf58f6f29c878266b4ec702ba3d9815dedb41b00177d0d7de", + "anchors_fingerprint": "1b17f5de54f5da34cfb1bd085fb7ea9e64a2dcb15c81b4d53e4145d670e05449", "files": { - "benchmarks/benchmark_lib.sh": { + "inferencex-e2e/benchmarks/benchmark_lib.sh": { "anchors": { "eval_dest": 1, "eval_start": 1, "num_prompts": 1 }, - "sha256": "23dc5b2fcc2b66fb6f6afc509f7400da062f43f758d22f61881f73a3c441ce18" + "sha256": "f3b8903721800b51da091b78dee1f54213dd35ea9811e57b4abeecfe7a0f0c43" }, - "utils/bench_serving/benchmark_serving.py": { + "inferencex-e2e/infx/bench_serving/benchmark_serving.py": { "anchors": { "profile_extra_body": 1 }, - "sha256": "9bb3e5096ddc74398f8a33ec2dc7c787afa6ae7bf8f5dd2c8b2e387a280a7785" + "sha256": "f98529033750e38763603cc9c83bcebfd364cf58f9fd5e867eec9dcdde0e7a44" } }, "magpie_patch": { "applies": true, - "path": "benchmarks/benchmark_lib.sh" + "path": "inferencex-e2e/benchmarks/benchmark_lib.sh" }, "probe_target": { - "path": "utils/evals/patches/lm_eval_sitecustomize.py", - "sha256": "b46fbc383c896d16b2cf3e7f862ada35b97612d6179e71a5a7145e4dd15b2ea2" + "path": "inferencex-e2e/infx/evals/patches/lm_eval_sitecustomize.py", + "sha256": "7ddc246bedcdca084b213973bb39ed34272a47fff08d2d4cbbdcf0a5909a8d3f" }, - "ref": "3d5581562f643f9bdeb8410cd924e2c70906c966", + "ref": "408c015be4b22d14c69518643609669405507077", "refresh_with": "python scripts/refresh_inferencex_anchor_contract.py" } diff --git a/src/hyperloom/inference_optimizer/tests/test_accuracy_gate_units.py b/src/hyperloom/inference_optimizer/tests/test_accuracy_gate_units.py index aefb127221..6a683529e1 100644 --- a/src/hyperloom/inference_optimizer/tests/test_accuracy_gate_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_accuracy_gate_units.py @@ -417,6 +417,152 @@ def test_reports_none_when_the_field_is_absent_or_not_a_number(self, tmp_path): (tmp_path / "inferencex_result.json").write_text(json.dumps({"request_error_rate": "n/a"}), encoding="utf-8") assert ag.parse_agentx_error_rate(tmp_path) is None + def test_reads_native_magpie_ratio_from_benchmark_report_as_percent(self, tmp_path): + (tmp_path / "benchmark_report.json").write_text( + json.dumps({"agentx_metrics": {"requests": {"error_rate": 1 / 11, "threshold": 0.1}}}), + encoding="utf-8", + ) + assert ag.parse_agentx_error_rate(tmp_path) == pytest.approx(100 / 11) + + def test_reconstructs_native_raw_accounting_as_percent(self, tmp_path): + (tmp_path / "inferencex_result.json").write_text( + json.dumps( + { + "num_requests_successful": 8, + "request_accounting": {"records_error_dropped": 2}, + } + ), + encoding="utf-8", + ) + assert ag.parse_agentx_error_rate(tmp_path) == pytest.approx(20.0) + + def test_native_report_error_rate_reaches_quality_gate(self, tmp_path): + (tmp_path / "benchmark_report.json").write_text( + json.dumps({"agentx_metrics": {"requests": {"error_rate": 0.2, "threshold": 0.1}}}), + encoding="utf-8", + ) + result = ag.parse_eval_results(tmp_path, framework="sglang", benchmark_mode="agentx") + assert result["accuracy"] == 0.0 + assert result["error_rate"] == pytest.approx(20.0) + assert result["error_rate_threshold"] == pytest.approx(10.0) + + @pytest.mark.parametrize( + ("successful", "dropped", "warmup", "errors", "measured_errors"), + [(1279, 87, 87, 1, 0), (100, 13, 10, 5, 3), (100, 12, 10, 2, 2), (8, 2, 0, 2, 2)], + ) + def test_raw_accounting_excludes_only_warmup_errors( + self, tmp_path, successful, dropped, warmup, errors, measured_errors + ): + (tmp_path / "inferencex_result.json").write_text( + json.dumps( + { + "num_requests_successful": successful, + "num_requests_total": successful + dropped, + "request_accounting": { + "records_dropped_total": dropped, + "records_warmup_dropped": warmup, + "records_error_dropped": errors, + }, + } + ), + encoding="utf-8", + ) + + result = ag.parse_eval_results(tmp_path, framework="sglang", benchmark_mode="agentx") + + assert result["error_rate"] == pytest.approx(100 * measured_errors / (successful + measured_errors)) + assert result["accuracy"] == (1.0 if result["error_rate"] <= 10.0 else 0.0) + + def test_legacy_raw_accounting_does_not_guess_warmup_overlap(self, tmp_path): + (tmp_path / "inferencex_result.json").write_text( + json.dumps( + { + "num_requests_successful": 8, + "request_accounting": {"records_error_dropped": 2, "records_warmup_dropped": 10}, + } + ), + encoding="utf-8", + ) + assert ag.parse_agentx_error_rate(tmp_path) == pytest.approx(20.0) + + @pytest.mark.parametrize( + ("field", "value"), + [ + ("records_dropped_total", 9), + ("records_dropped_total", 16), + ("records_dropped_total", -1), + ("records_dropped_total", None), + ("records_dropped_total", True), + ("records_dropped_total", 13.0), + ("records_warmup_dropped", None), + ("records_warmup_dropped", -1), + ("records_error_dropped", 14), + ("records_error_dropped", -1), + ("records_error_dropped", True), + ("num_requests_successful", None), + ("num_requests_total", 114), + ("num_requests_total", None), + ], + ) + def test_inconsistent_raw_accounting_fails_closed_without_using_an_older_result(self, tmp_path, field, value): + import os + + older = tmp_path / "benchmark_report.json" + older.write_text(json.dumps({"request_error_rate": 0.0}), encoding="utf-8") + os.utime(older, (1, 1)) + raw = { + "num_requests_successful": 100, + "num_requests_total": 113, + "request_accounting": { + "records_dropped_total": 13, + "records_warmup_dropped": 10, + "records_error_dropped": 5, + }, + } + if field.startswith("records_"): + raw["request_accounting"][field] = value + else: + raw[field] = value + (tmp_path / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + + result = ag.parse_eval_results(tmp_path, framework="sglang", benchmark_mode="agentx") + + assert result["error_rate"] is None + assert result["accuracy"] == 0.0 + + def test_native_report_uses_its_bound_request_threshold(self, tmp_path): + (tmp_path / "benchmark_report.json").write_text( + json.dumps({"agentx_metrics": {"requests": {"error_rate": 0.15, "threshold": 0.2}}}), + encoding="utf-8", + ) + result = ag.parse_eval_results(tmp_path, framework="sglang", benchmark_mode="agentx") + assert result["accuracy"] == 1.0 + assert result["error_rate"] == pytest.approx(15.0) + assert result["error_rate_threshold"] == pytest.approx(20.0) + + @pytest.mark.parametrize("threshold", [None, -0.01, 1.01, float("nan"), float("inf"), True, "0.2"]) + def test_native_report_invalid_request_threshold_fails_closed(self, tmp_path, threshold): + requests = {"error_rate": 0.01} + if threshold is not None: + requests["threshold"] = threshold + (tmp_path / "benchmark_report.json").write_text( + json.dumps({"agentx_metrics": {"requests": requests}}), + encoding="utf-8", + ) + result = ag.parse_eval_results(tmp_path, framework="sglang", benchmark_mode="agentx") + assert result["accuracy"] == 0.0 + assert result["error_rate"] == pytest.approx(1.0) + assert result["error_rate_threshold"] is None + + def test_legacy_result_keeps_the_default_request_threshold(self, tmp_path): + (tmp_path / "inferencex_result.json").write_text( + json.dumps({"request_error_rate": 9.0}), + encoding="utf-8", + ) + result = ag.parse_eval_results(tmp_path, framework="vllm", benchmark_mode="agentx") + assert result["accuracy"] == 1.0 + assert result["error_rate_threshold"] == pytest.approx(10.0) + def test_a_mapped_result_missing_the_metric_reports_none(self, tmp_path): """aiperf omits the metric when nothing completed. Coalescing that to 0.0 would report a perfect error rate for a run that measured nothing, diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_baseline_timeout.py b/src/hyperloom/inference_optimizer/tests/test_agentx_baseline_timeout.py index a9ca690b45..a1d9b831d0 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_baseline_timeout.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_baseline_timeout.py @@ -25,6 +25,7 @@ def _clear(monkeypatch): _timeouts._AGENTX_SAID.clear() for key in ( "HYPERLOOM_AGENTX", + "AGENTX_MODEL_ID", "AGENTX_WARMUP_GRACE_PERIOD", "AGENTX_WARMUP_GRACE_CONC", "CONC", @@ -55,10 +56,17 @@ def test_profile_retains_nonbenchmark_budget(monkeypatch, tmp_path): def test_agentx_switch_keeps_profile_yaml_cap(monkeypatch): _clear(monkeypatch) + monkeypatch.setenv("AGENTX_MODEL_ID", "amd/test-model") monkeypatch.setenv("AGENTX_DURATION", "50000") - bench = {"framework": "sglang", "timeout_seconds": 14400} - apply_agentx_switch(bench, active=True) + bench = { + "framework": "sglang", + "timeout_seconds": 14400, + "envs": {"PROFILE": "1"}, + "profiler": {"torch_profiler": {"enabled": True}}, + } + apply_agentx_switch(bench, active=True, allow_profile_compat=True) assert bench["timeout_seconds"] == 14400 + assert bench["benchmark_script"] == "aiperf_client.sh" @pytest.mark.parametrize("conc", ["1", "4", "8"]) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_budget_and_guards.py b/src/hyperloom/inference_optimizer/tests/test_agentx_budget_and_guards.py index dd0509aec7..7e03dd743d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_budget_and_guards.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_budget_and_guards.py @@ -6,12 +6,20 @@ from __future__ import annotations import argparse +import hashlib +import os +from pathlib import Path import pytest +import yaml from hyperloom.inference_optimizer.cli import ( _apply_agentx_budget_profile, + _configure_benchmark_config, + _finalize_benchmark_config, _preflight_agentx_backend, + _restore_agentx_env_from_state, + _restore_agentx_runtime_pins_from_state, ) from hyperloom.inference_optimizer.cli.bootstrap import ( AGENTX_MEASUREMENT_EPOCH, @@ -20,6 +28,25 @@ from hyperloom.inference_optimizer.cli.parser import DEFAULT_MAX_HOURS +_MAGPIE_REF = "c" * 40 +_INFERENCEX_REF = "d" * 40 +_RECIPE_FINGERPRINT = "a" * 64 +_EXECUTION_FINGERPRINT = "b" * 64 + + +@pytest.fixture(autouse=True) +def isolate_source_config(monkeypatch): + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", "") + + +@pytest.fixture +def native_config(monkeypatch, tmp_path): + source = tmp_path / "native-source.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + return source + + def _budget_args(**over) -> argparse.Namespace: base = dict( # What the parser produces when ``--max-hours`` is absent: the flag @@ -35,10 +62,60 @@ def _off(monkeypatch): monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) +def _blank_agentx_runtime_pins(monkeypatch) -> None: + from hyperloom.inference_optimizer.agentx.native import ( + AGENTX_RUNTIME_PIN_NAMES, + ) + + for name in AGENTX_RUNTIME_PIN_NAMES: + monkeypatch.setenv(name, "") + + def _on(monkeypatch): + _blank_agentx_runtime_pins(monkeypatch) monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.delenv("HYPERLOOM_AGENTIC_BACKEND", raising=False) + monkeypatch.setenv("AGENTX_MODEL_ID", "acme/Test-Model") + monkeypatch.setenv( + "AGENTX_SERVER_SCRIPT", + "single_node/agentic/test_fp4_mi300x_vllm_mtp.sh", + ) + monkeypatch.setenv("INFERENCEX_PATH", "/tmp/inferencex") + monkeypatch.setenv("HYPERLOOM_IMAGE", "example/agentx:test") + monkeypatch.setenv( + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + _RECIPE_FINGERPRINT, + ) + monkeypatch.setenv( + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT", + _EXECUTION_FINGERPRINT, + ) + monkeypatch.setenv("HYPERLOOM_AGENTX_GPU_COUNT", "1") + monkeypatch.setenv("MAGPIE_REF", _MAGPIE_REF) + monkeypatch.setenv("INFERENCEX_REF", _INFERENCEX_REF) + + +def _pin_source_hash(monkeypatch, config: Path) -> None: + monkeypatch.setenv( + "HYPERLOOM_BENCHMARK_CONFIG_SHA256", + hashlib.sha256(config.read_bytes()).hexdigest(), + ) + + +def _stub_native_execution_identity(monkeypatch, native_agentx) -> None: + monkeypatch.setattr( + native_agentx, + "native_execution_identity", + lambda **_kwargs: { + "static_execution_fingerprint": _EXECUTION_FINGERPRINT, + "execution_fingerprint": _EXECUTION_FINGERPRINT, + "inferencex_commit": _INFERENCEX_REF, + "magpie_commit": _MAGPIE_REF, + }, + ) + # --- budget profile ----------------------------------------------------------- @@ -96,6 +173,7 @@ def test_budget_profile_preserves_operator_values(monkeypatch): def test_bypass_guard_allows_magpie(monkeypatch): _on(monkeypatch) monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + monkeypatch.delenv("INFERENCE_OPTIMIZER_RAY_EXEC", raising=False) _preflight_agentx_backend(argparse.Namespace()) # must not raise @@ -114,6 +192,16 @@ def test_bypass_guard_rejects_the_silent_combination(monkeypatch): assert ei.value.code == 2 +@pytest.mark.usefixtures("native_config") +def test_guard_rejects_explicit_ray_execution(monkeypatch): + _on(monkeypatch) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", "1") + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend(argparse.Namespace(framework="sglang", nodes=1)) + assert exc.value.code == 2 + + def test_guard_rejects_agentx_with_a_scriptable_framework(monkeypatch): """The other way for the switch to no-op while every gate still fires.""" _on(monkeypatch) @@ -130,20 +218,520 @@ def test_guard_allows_agentx_with_a_serving_framework(monkeypatch): _preflight_agentx_backend(argparse.Namespace(framework=fw)) # must not raise +@pytest.mark.usefixtures("native_config") +def test_guard_rejects_native_agentx_multi_node(monkeypatch): + _on(monkeypatch) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend(argparse.Namespace(framework="sglang", nodes=2)) + assert exc.value.code == 2 + + +@pytest.mark.usefixtures("native_config") +def test_guard_rejects_framework_without_native_argv_bridge(monkeypatch): + _on(monkeypatch) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend(argparse.Namespace(framework="atom", nodes=1)) + assert exc.value.code == 2 + + +@pytest.mark.parametrize("framework", ["atom", "xdit"]) +@pytest.mark.usefixtures("native_config") +def test_guard_rejects_unsupported_framework_from_environment(monkeypatch, framework): + _on(monkeypatch) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + monkeypatch.setenv("FRAMEWORK", framework) + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend(argparse.Namespace(nodes=1)) + assert exc.value.code == 2 + + def test_scriptable_guard_is_inert_without_agentx(monkeypatch): """A scriptable run on its own is perfectly normal.""" _off(monkeypatch) _preflight_agentx_backend(argparse.Namespace(framework="xdit")) # must not raise +def test_benchmark_yaml_agentx_switch_sets_session_mode_and_pins(monkeypatch, tmp_path): + for name in ( + "HYPERLOOM_AGENTX", + "HYPERLOOM_BENCHMARK_CONFIG", + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "INFERENCEX_PATH", + ): + # _configure_benchmark_config writes directly to os.environ. Seed a + # tracked value so monkeypatch restores/removes every write at teardown. + monkeypatch.setenv(name, "") + _off(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + config = tmp_path / "agentx.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "framework": "sglang", + "model": "amd/GLM-5.2-MXFP4", + "precision": "fp4", + "agentx": "enable", + "benchmark_script": "single_node/agentic/glm.sh", + "inferencex_path": str(inferencex), + } + } + ), + encoding="utf-8", + ) + args = argparse.Namespace( + benchmark_config=str(config), + resume_from=None, + framework=None, + model=None, + precision=None, + ) + + assert _configure_benchmark_config(args) is True + assert os.environ["HYPERLOOM_AGENTX"] == "1" + assert os.environ["HYPERLOOM_BENCHMARK_CONFIG"] == str(config.resolve()) + assert os.environ["AGENTX_MODEL_ID"] == "amd/GLM-5.2-MXFP4" + assert os.environ["AGENTX_SERVER_SCRIPT"] == "single_node/agentic/glm.sh" + assert os.environ["INFERENCEX_PATH"] == str(inferencex.resolve()) + assert args.framework == "sglang" + assert args.model == Path("amd/GLM-5.2-MXFP4") + assert args.precision == "fp4" + + +def test_benchmark_yaml_projects_workload_before_preflight_without_resolving( + monkeypatch, + tmp_path, +): + """Early configuration must not need an installed Magpie interpreter.""" + for name in ( + "HYPERLOOM_AGENTX", + "HYPERLOOM_BENCHMARK_CONFIG", + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "INFERENCEX_PATH", + ): + monkeypatch.setenv(name, "") + _off(monkeypatch) + from hyperloom.inference_optimizer.agentx import native as native_agentx + + def _fail_resolver(*_args, **_kwargs): + pytest.fail("recipe resolution must run after dependency preflight") + + monkeypatch.setattr( + native_agentx, + "preview_native_recipe", + _fail_resolver, + ) + monkeypatch.setattr(native_agentx, "_run_magpie_recipe_resolver", _fail_resolver) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + local_model = tmp_path / "models" / "glm" + config = tmp_path / "agentx.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": "enable", + "framework": "sglang", + "runner_type": "mi355x", + "model": "amd/GLM-5.2-MXFP4", + "benchmark_script": "single_node/agentic/glm.sh", + "inferencex_path": str(inferencex), + "envs": { + "MODEL_PATH": str(local_model), + "CONC": 64, + "ISL": 1024, + "OSL": 1024, + }, + } + } + ), + encoding="utf-8", + ) + args = argparse.Namespace( + benchmark_config=str(config), + resume_from=None, + framework=None, + gpu_type=None, + model=None, + precision=None, + tp=None, + ep=None, + conc=None, + isl=None, + osl=None, + max_model_len=None, + ) + + assert _configure_benchmark_config(args) is True + assert args.model == local_model + assert args.gpu_type == "mi355x" + assert args.conc == 64 + assert args.isl == 1024 + assert args.osl == 1024 + assert args.tp is None + assert os.environ["AGENTX_MODEL_ID"] == "amd/GLM-5.2-MXFP4" + + +def test_non_agentx_benchmark_yaml_projects_physical_tp_and_concurrency( + monkeypatch, + tmp_path, +): + monkeypatch.setenv("HYPERLOOM_AGENTX", "") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", "") + _off(monkeypatch) + config = tmp_path / "synthetic.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": "disable", + "framework": "sglang", + "model": "/models/qwen", + "envs": {"TP": 8, "EP": 4, "CONC": 32}, + } + } + ), + encoding="utf-8", + ) + args = argparse.Namespace( + benchmark_config=str(config), + resume_from=None, + framework=None, + gpu_type=None, + model=None, + precision=None, + tp=None, + ep=None, + conc=None, + isl=None, + osl=None, + max_model_len=None, + ) + + assert _configure_benchmark_config(args) is False + assert args.tp == 8 + assert args.ep == 4 + assert args.conc == 32 + + +def test_finalize_benchmark_yaml_preserves_read_and_source_change_errors(monkeypatch, tmp_path): + config = tmp_path / "agentx.yaml" + config.write_bytes(b"benchmark: \xff") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(config)) + _pin_source_hash(monkeypatch, config) + args = argparse.Namespace(resume_from=None) + + with pytest.raises(ValueError, match="cannot read benchmark config") as invalid_utf8: + _finalize_benchmark_config(args) + assert str(config) in str(invalid_utf8.value) + assert isinstance(invalid_utf8.value.__cause__, UnicodeDecodeError) + + config.write_bytes(b"benchmark: \xfe") + with pytest.raises(ValueError, match="benchmark config changed after initial validation") as changed_source: + _finalize_benchmark_config(args) + assert str(config) in str(changed_source.value) + assert changed_source.value.__cause__ is None + + +def test_finalize_benchmark_yaml_projects_resolved_agentx_topology( + monkeypatch, + tmp_path, +): + _blank_agentx_runtime_pins(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + config = tmp_path / "agentx.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": { + "enabled": True, + "concurrency": 8, + "resolved": {"stale": True}, + }, + "precision": "fp4", + "envs": {"CONC": 8}, + } + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(config)) + _pin_source_hash(monkeypatch, config) + monkeypatch.setenv("INFERENCEX_PATH", str(inferencex)) + monkeypatch.setenv("AGENTX_MODEL_ID", "amd/GLM-5.2-MXFP4") + monkeypatch.setenv( + "AGENTX_SERVER_SCRIPT", + "single_node/agentic/glm.sh", + ) + monkeypatch.setenv("HYPERLOOM_IMAGE", "rocm/agentx:accepted") + captured = {} + + def _preview(benchmark, *, inferencex_path): + captured["benchmark"] = benchmark + captured["inferencex_path"] = inferencex_path + return { + "recipe": "glm5-agentic", + "config_file": "configs/amd-master.yaml", + "topology": { + "tp": 2, + "pp": 2, + "pcp_size": 2, + "ep": 4, + "gpu_count": 8, + "conc": 32, + "duration_seconds": 3600, + "recipe_fingerprint": _RECIPE_FINGERPRINT, + }, + "entry": {"image": "rocm/agentx:accepted"}, + "magpie_execution": { + "fingerprint": "e" * 64, + "source_commit": _MAGPIE_REF, + }, + } + + from hyperloom.inference_optimizer.agentx import native as native_agentx + + monkeypatch.setattr(native_agentx, "preview_native_recipe", _preview) + _stub_native_execution_identity(monkeypatch, native_agentx) + args = argparse.Namespace( + resume_from=None, + tp=None, + ep=None, + conc=32, + precision="bf16", + ) + + assert _finalize_benchmark_config(args) is True + assert args.tp == 8 + assert args.ep == 4 + assert args.conc == 32 + assert captured["inferencex_path"] == str(inferencex) + assert captured["benchmark"]["envs"]["CONC"] == 32 + assert captured["benchmark"]["envs"]["AGENTX_MODEL_ID"] == ("amd/GLM-5.2-MXFP4") + assert captured["benchmark"]["precision"] == "bf16" + assert "concurrency" not in captured["benchmark"]["agentx"] + assert "resolved" not in captured["benchmark"]["agentx"] + assert os.environ["HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT"] == _RECIPE_FINGERPRINT + assert os.environ["HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT"] == _EXECUTION_FINGERPRINT + assert os.environ["HYPERLOOM_AGENTX_GPU_COUNT"] == "8" + assert os.environ["AGENTX_MODE"] == "canonical" + assert os.environ["AGENTX_RECIPE"] == "glm5-agentic" + assert os.environ["AGENTX_CONFIG_FILE"] == "configs/amd-master.yaml" + assert os.environ["AGENTX_FAILED_REQUEST_THRESHOLD"] == "0.1" + + +@pytest.mark.parametrize( + ("requested_tp", "requested_ep", "message"), + [ + (4, None, "physical GPU count 8"), + (None, 2, "resolved AgentX EP 4"), + ], +) +def test_finalize_benchmark_yaml_rejects_resolved_topology_mismatch( + monkeypatch, + tmp_path, + requested_tp, + requested_ep, + message, +): + _blank_agentx_runtime_pins(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + config = tmp_path / "agentx.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": "enable", + "envs": {"CONC": 64}, + } + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(config)) + _pin_source_hash(monkeypatch, config) + monkeypatch.setenv("INFERENCEX_PATH", str(inferencex)) + monkeypatch.setenv("HYPERLOOM_IMAGE", "rocm/agentx:accepted") + from hyperloom.inference_optimizer.agentx import native as native_agentx + + monkeypatch.setattr( + native_agentx, + "preview_native_recipe", + lambda *_args, **_kwargs: { + "recipe": "glm5-agentic", + "config_file": "configs/amd-master.yaml", + "topology": { + "tp": 2, + "pp": 2, + "pcp_size": 2, + "ep": 4, + "gpu_count": 8, + "conc": 64, + "duration_seconds": 3600, + "recipe_fingerprint": _RECIPE_FINGERPRINT, + }, + "entry": {"image": "rocm/agentx:accepted"}, + "magpie_execution": { + "fingerprint": "e" * 64, + "source_commit": _MAGPIE_REF, + }, + }, + ) + _stub_native_execution_identity(monkeypatch, native_agentx) + args = argparse.Namespace( + resume_from=None, + tp=requested_tp, + ep=requested_ep, + conc=None, + precision=None, + ) + + with pytest.raises(ValueError, match=message): + _finalize_benchmark_config(args) + + +def test_resume_ignores_inherited_benchmark_yaml(monkeypatch, tmp_path): + foreign = tmp_path / "foreign.yaml" + foreign.write_text( + yaml.safe_dump({"benchmark": {"agentx": "enable", "framework": "sglang"}}), + encoding="utf-8", + ) + _off(monkeypatch) + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(foreign)) + args = argparse.Namespace( + benchmark_config=None, + resume_from=str(tmp_path / "session"), + ) + + assert _configure_benchmark_config(args) is False + assert "HYPERLOOM_BENCHMARK_CONFIG" not in os.environ + assert "HYPERLOOM_AGENTX" not in os.environ + + +@pytest.mark.usefixtures("native_config") +@pytest.mark.parametrize("epoch", [None, 2, 3, 4]) +def test_native_preflight_rejects_explicit_concurrency_sweep(monkeypatch, epoch): + _on(monkeypatch) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + monkeypatch.delenv("INFERENCE_OPTIMIZER_RAY_EXEC", raising=False) + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend( + argparse.Namespace( + framework="sglang", + nodes=1, + enable_conc_sweep=True, + ), + _St("agentx", epoch) if epoch else None, + ) + assert exc.value.code == 2 + + # --- resume staleness --------------------------------------------------------- class _St: - def __init__(self, mode="", epoch=0, backend=""): + def __init__( + self, + mode="", + epoch=0, + backend="", + baseline_config_path="", + agentx_runtime_pins=None, + active_inferencex_path="", + ): self.benchmark_mode = mode self.agentx_epoch = epoch self.agentx_backend = backend + self.baseline_config_path = baseline_config_path + self.agentx_runtime_pins = dict(agentx_runtime_pins or {}) + self.active_inferencex_path = active_inferencex_path + + +def _saved_native_baseline(tmp_path): + inferencex_path = tmp_path / "InferenceX" + inferencex_path.mkdir() + baseline_path = tmp_path / "accepted-agentx.yaml" + pins = { + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm_fp4_mi355x_sglang.sh", + "INFERENCEX_PATH": str(inferencex_path), + "HYPERLOOM_IMAGE": "rocm/agentx:accepted", + "AGENTX_MODE": "canonical", + "AGENTX_RECIPE": "glm5-agentic", + "AGENTX_CONFIG_FILE": "configs/amd-master.yaml", + "AGENTX_SELECTOR": '{"ep":4,"tp":4}', + "AGENTX_FAILED_REQUEST_THRESHOLD": "0.1", + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": _RECIPE_FINGERPRINT, + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": _EXECUTION_FINGERPRINT, + "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT": _EXECUTION_FINGERPRINT, + "HYPERLOOM_AGENTX_GPU_COUNT": "4", + "MAGPIE_REF": _MAGPIE_REF, + "INFERENCEX_REF": _INFERENCEX_REF, + } + baseline_path.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": { + "enabled": True, + "mode": "canonical", + "recipe": pins["AGENTX_RECIPE"], + "config_file": pins["AGENTX_CONFIG_FILE"], + "selector": {"tp": 4, "ep": 4}, + "failed_request_threshold": 0.10, + }, + "model": pins["AGENTX_MODEL_ID"], + "benchmark_script": pins["AGENTX_SERVER_SCRIPT"], + "inferencex_path": pins["INFERENCEX_PATH"], + "docker_image": pins["HYPERLOOM_IMAGE"], + "envs": { + "AGENTX_MODEL_ID": pins["AGENTX_MODEL_ID"], + "AGENTX_SERVER_SCRIPT": pins["AGENTX_SERVER_SCRIPT"], + }, + "workload_spec": { + "outer_image": pins["HYPERLOOM_IMAGE"], + "recipe": { + "name": pins["AGENTX_RECIPE"], + "config_file": pins["AGENTX_CONFIG_FILE"], + "image": pins["HYPERLOOM_IMAGE"], + "recipe_fingerprint": _RECIPE_FINGERPRINT, + }, + "resolved_topology": { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "ep": 4, + "gpu_count": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": _RECIPE_FINGERPRINT, + }, + "execution": { + "static_execution_fingerprint": _EXECUTION_FINGERPRINT, + "execution_fingerprint": _EXECUTION_FINGERPRINT, + "magpie_commit": _MAGPIE_REF, + "inferencex_commit": _INFERENCEX_REF, + }, + }, + } + } + ), + encoding="utf-8", + ) + state = _St( + "agentx", + 2, + baseline_config_path=str(baseline_path), + ) + return state, pins def test_resume_accepts_matching_agentx_state(monkeypatch): @@ -151,6 +739,29 @@ def test_resume_accepts_matching_agentx_state(monkeypatch): assert agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH)) == "" +def test_epoch_three_resume_pins_workload_without_freezing_candidate_execution(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + state.agentx_epoch = 3 + config = Path(state.baseline_config_path) + parsed = yaml.safe_load(config.read_text()) + parsed["benchmark"]["agentx"]["launch_overrides"] = {"version": 1} + parsed["benchmark"]["workload_spec"]["execution"]["workload_fingerprint"] = "f" * 64 + config.write_text(yaml.safe_dump(parsed)) + _blank_agentx_runtime_pins(monkeypatch) + restored = _restore_agentx_runtime_pins_from_state(state) + assert restored["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] == "f" * 64 + assert "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT" not in restored + assert "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT" not in os.environ + + +def test_resume_rejects_implicit_launch_contract_upgrade(monkeypatch, tmp_path): + state, _ = _saved_native_baseline(tmp_path) + state.agentx_epoch = 3 + _blank_agentx_runtime_pins(monkeypatch) + with pytest.raises(ValueError, match="launch contract.*conflicts with the session epoch"): + _restore_agentx_runtime_pins_from_state(state) + + def test_resume_accepts_matching_synthetic_state(monkeypatch): _off(monkeypatch) assert agentx_state_is_stale(_St("synthetic", 0)) == "" @@ -160,29 +771,233 @@ def test_resume_rejects_mode_switch(monkeypatch): """The KEEP ledger keys on server args alone, so the rows would collide.""" _on(monkeypatch) assert "benchmark_mode" in agentx_state_is_stale(_St("synthetic", 0)) - _off(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "0") assert "benchmark_mode" in agentx_state_is_stale(_St("agentx", 1)) def test_resume_rejects_a_backend_switch(monkeypatch): """aiperf and mlperf sessions must not anchor each other, epoch unchanged.""" _on(monkeypatch) - assert agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH, "aiperf")) == "" + assert agentx_state_is_stale(_St("agentx", 1, "aiperf")) == "" monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf") - reason = agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH, "aiperf")) + reason = agentx_state_is_stale(_St("agentx", 1, "aiperf")) assert "backend" in reason - assert agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH, "mlperf")) == "" + assert agentx_state_is_stale(_St("agentx", 1, "mlperf")) == "" def test_resume_treats_a_missing_backend_as_aiperf(monkeypatch): _on(monkeypatch) - assert agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH, "")) == "" + assert agentx_state_is_stale(_St("agentx", 1, "")) == "" + + +def test_fresh_shell_resume_inherits_persisted_agentx_mode(monkeypatch): + """An omitted mode on resume is not an implicit request to go synthetic.""" + _off(monkeypatch) + assert agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH)) == "" -def test_resume_rejects_stale_agentx_epoch(monkeypatch): +@pytest.mark.parametrize("shell_value", [None, "", " "]) +def test_fresh_shell_resume_reexports_persisted_agentx_mode(monkeypatch, shell_value): + if shell_value is None: + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + else: + monkeypatch.setenv("HYPERLOOM_AGENTX", shell_value) + + try: + assert _restore_agentx_env_from_state(_St("agentx", AGENTX_MEASUREMENT_EPOCH)) is True + assert os.environ["HYPERLOOM_AGENTX"] == "1" + finally: + # The helper deliberately writes os.environ directly. When the key was + # initially absent, monkeypatch.delenv has no undo entry for that new + # write, so clean it explicitly to keep later tests mode-neutral. + if shell_value is None: + os.environ.pop("HYPERLOOM_AGENTX", None) + + +def test_fresh_shell_resume_restores_native_agentx_runtime_pins(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + for name in pins: + # A blank value models an omitted fresh-shell export while preserving + # monkeypatch's teardown record for the helper's direct env writes. + monkeypatch.setenv(name, "") + + assert _restore_agentx_runtime_pins_from_state(state) == pins + assert {name: os.environ[name] for name in pins} == pins + + +def test_prebaseline_resume_restores_seed_time_agentx_runtime_pins(monkeypatch, tmp_path): + _blank_agentx_runtime_pins(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + pins = { + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm.sh", + "INFERENCEX_PATH": str(inferencex), + "HYPERLOOM_IMAGE": "rocm/agentx:seeded", + "AGENTX_MODE": "canonical", + "AGENTX_FAILED_REQUEST_THRESHOLD": "0.1", + } + state = _St( + "agentx", + AGENTX_MEASUREMENT_EPOCH, + agentx_runtime_pins=pins, + active_inferencex_path=str(inferencex), + ) + for name in pins: + monkeypatch.setenv(name, "") + + assert _restore_agentx_runtime_pins_from_state(state) == pins + assert {name: os.environ[name] for name in pins} == pins + + +def test_resume_ignores_generic_active_framework_checkout(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + promoted = tmp_path / "InferenceX-promoted" + promoted.mkdir() + state.active_inferencex_path = str(promoted) + state.agentx_runtime_pins = dict(pins) + for name in pins: + monkeypatch.setenv(name, "") + + restored = _restore_agentx_runtime_pins_from_state(state) + + assert restored["INFERENCEX_PATH"] == str((tmp_path / "InferenceX").resolve()) + assert os.environ["INFERENCEX_PATH"] == str((tmp_path / "InferenceX").resolve()) + + +def test_prebaseline_resume_restores_selector_and_resolved_contract( + monkeypatch, + tmp_path, +): + _blank_agentx_runtime_pins(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + pins = { + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm.sh", + "INFERENCEX_PATH": str(inferencex), + "HYPERLOOM_IMAGE": "rocm/agentx:seeded", + "AGENTX_RECIPE": "glm5-agentic", + "AGENTX_SELECTOR": '{"ep":4,"tp":4}', + "AGENTX_MODE": "canonical", + "AGENTX_FAILED_REQUEST_THRESHOLD": "0.1", + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": "a" * 64, + "HYPERLOOM_AGENTX_GPU_COUNT": "4", + } + state = _St( + "agentx", + AGENTX_MEASUREMENT_EPOCH, + agentx_runtime_pins=pins, + ) + for name in pins: + monkeypatch.setenv(name, "") + + assert _restore_agentx_runtime_pins_from_state(state) == pins + assert {name: os.environ[name] for name in pins} == pins + + +def test_resume_ignores_preflight_generated_path_when_operator_did_not_pin_one( + monkeypatch, + tmp_path, +): + state, pins = _saved_native_baseline(tmp_path) + auto_checkout = tmp_path / "auto-cloned-InferenceX" + auto_checkout.mkdir() + for name in pins: + monkeypatch.setenv(name, "") + monkeypatch.setenv("INFERENCEX_PATH", str(auto_checkout)) + + restored = _restore_agentx_runtime_pins_from_state( + state, + operator_pins={name: "" for name in pins}, + ) + + assert restored["INFERENCEX_PATH"] == pins["INFERENCEX_PATH"] + assert os.environ["INFERENCEX_PATH"] == pins["INFERENCEX_PATH"] + + +def test_resume_accepts_equivalent_saved_inferencex_checkout(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + for name, value in pins.items(): + monkeypatch.setenv(name, value) + checkout = tmp_path / "InferenceX" + monkeypatch.setenv("INFERENCEX_PATH", str(checkout / ".." / checkout.name)) + + expected = str(checkout.resolve()) + assert _restore_agentx_runtime_pins_from_state(state) == { + "INFERENCEX_PATH": expected, + } + assert os.environ["INFERENCEX_PATH"] == expected + + +def test_resume_rejects_native_agentx_runtime_pin_conflict(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + for name in pins: + monkeypatch.setenv(name, "") + monkeypatch.setenv("HYPERLOOM_IMAGE", "rocm/agentx:different") + + with pytest.raises(ValueError, match="HYPERLOOM_IMAGE=.*conflicts"): + _restore_agentx_runtime_pins_from_state(state) + assert os.environ["AGENTX_MODEL_ID"] == "" + assert os.environ["AGENTX_SERVER_SCRIPT"] == "" + assert os.environ["INFERENCEX_PATH"] == "" + + +def test_resume_rejects_failed_request_threshold_drift(monkeypatch, tmp_path): + state, pins = _saved_native_baseline(tmp_path) + for name in pins: + monkeypatch.setenv(name, "") + monkeypatch.setenv("AGENTX_FAILED_REQUEST_THRESHOLD", "1") + + with pytest.raises( + ValueError, + match="AGENTX_FAILED_REQUEST_THRESHOLD=.*conflicts", + ): + _restore_agentx_runtime_pins_from_state(state) + + +def test_prebaseline_resume_rejects_unpinned_selector_pollution( + monkeypatch, + tmp_path, +): + _blank_agentx_runtime_pins(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + pins = { + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm.sh", + "INFERENCEX_PATH": str(inferencex), + "HYPERLOOM_IMAGE": "rocm/agentx:seeded", + "AGENTX_MODE": "canonical", + "AGENTX_FAILED_REQUEST_THRESHOLD": "0.1", + } + state = _St( + "agentx", + AGENTX_MEASUREMENT_EPOCH, + agentx_runtime_pins=pins, + ) + for name in pins: + monkeypatch.setenv(name, "") + monkeypatch.setenv("AGENTX_SELECTOR", '{"kv_offloading":"none"}') + + with pytest.raises(ValueError, match="AGENTX_SELECTOR=.*conflicts"): + _restore_agentx_runtime_pins_from_state(state) + + +def test_synthetic_resume_does_not_read_agentx_runtime_pins(monkeypatch, tmp_path): + missing = tmp_path / "does-not-exist.yaml" + monkeypatch.setenv("AGENTX_MODEL_ID", "") + state = _St("synthetic", baseline_config_path=str(missing)) + + assert _restore_agentx_runtime_pins_from_state(state) == {} + assert os.environ["AGENTX_MODEL_ID"] == "" + + +@pytest.mark.parametrize("epoch", [0, AGENTX_MEASUREMENT_EPOCH + 1]) +def test_resume_rejects_stale_agentx_epoch(monkeypatch, epoch): """Same knobs, different workload: the old numbers cannot anchor.""" _on(monkeypatch) - reason = agentx_state_is_stale(_St("agentx", AGENTX_MEASUREMENT_EPOCH - 1)) + reason = agentx_state_is_stale(_St("agentx", epoch)) assert "epoch" in reason @@ -237,7 +1052,7 @@ def test_verdict_gate_is_inert_on_the_synthetic_path(monkeypatch): def test_verdict_gate_spares_scriptable_runs_under_agentx(monkeypatch): """A scriptable framework skips the aiperf switch entirely.""" _on(monkeypatch) - assert _valid(_measurement()) is True + assert _valid(_measurement(framework="xdit", quality_gate={"passed": True})) is True # --- inner Magpie timeout follows the AgentX cap ------------------------------ @@ -251,10 +1066,11 @@ def test_agentx_switch_does_not_resolve_launch_timeout(monkeypatch): apply_agentx_switch, ) - bench = {"framework": "vllm", "model": "/models/x", "timeout_seconds": 7200} + bench = {"framework": "vllm", "model": "/models/x", "timeout_seconds": 7200, "agentx": "enable"} apply_agentx_switch(bench) assert bench["timeout_seconds"] == 7200 - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == "single_node/agentic/test_fp4_mi300x_vllm_mtp.sh" + assert bench["agentx"] == "enable" def test_agentx_switch_leaves_the_inner_timeout_alone_without_agentx(monkeypatch): @@ -479,3 +1295,127 @@ def test_the_default_grid_never_re_derives_a_grace(monkeypatch, tmp_path): envs = yaml.safe_load(cfg_path.read_text(encoding="utf-8"))["benchmark"]["envs"] assert "AGENTX_WARMUP_GRACE_PERIOD" not in envs assert envs["CONC"] == "128" + + +@pytest.mark.parametrize("framework", ["sglang", "vllm", "atom"]) +def test_legacy_agentx_keeps_optimizer_launch_options_without_native_pins(monkeypatch, framework): + _blank_agentx_runtime_pins(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", "1") + _preflight_agentx_backend( + argparse.Namespace(framework=framework, nodes=2, enable_conc_sweep=True), _St("agentx", 1) + ) + + +def test_legacy_resume_preserves_epoch_one_and_does_not_require_native_pins(monkeypatch, native_config): + from hyperloom.inference_optimizer.cli import _enforce_agentx_resume_workload + + _blank_agentx_runtime_pins(monkeypatch) + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + monkeypatch.setenv("HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", "stale-native-pin") + monkeypatch.setenv("HYPERLOOM_AGENTX_GPU_COUNT", "4") + state = _St("agentx", 1, baseline_config_path="/old-session/missing-baseline.yaml") + assert agentx_state_is_stale(state) == "" + assert _restore_agentx_runtime_pins_from_state(state) == {} + assert "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT" not in os.environ + assert "HYPERLOOM_AGENTX_GPU_COUNT" not in os.environ + args = argparse.Namespace(tp=8, ep=4, conc=16, precision="fp8") + _enforce_agentx_resume_workload(args, state) + assert args.conc == 16 + + +def test_native_resume_rejects_epoch_one_measurements(monkeypatch, native_config): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + state = _St("agentx", 1, baseline_config_path=str(native_config)) + assert "epoch 1" in agentx_state_is_stale(state) + + +def test_native_preflight_requires_pins_but_legacy_resume_ignores_fresh_native_config(monkeypatch, native_config): + _blank_agentx_runtime_pins(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.delenv("HYPERLOOM_BENCHMARK_BACKEND", raising=False) + args = argparse.Namespace(framework="sglang", nodes=1) + with pytest.raises(SystemExit): + _preflight_agentx_backend(args) + _preflight_agentx_backend(args, _St("agentx", 1)) + + +@pytest.mark.parametrize("ray_override", [None, "1"]) +def test_accepted_native_resume_restores_direct_execution_identity(monkeypatch, tmp_path, ray_override): + from hyperloom.common.agentx_mode import native_agentx_session + from hyperloom.orchestrator.actions.executors import _ray_backend + from hyperloom.orchestrator.actions.executors.profile import ProfileExecutor + + state, pins = _saved_native_baseline(tmp_path) + _blank_agentx_runtime_pins(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "") + monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False) + monkeypatch.setenv("INFERENCE_OPTIMIZER_NODES", "1") + if ray_override is None: + monkeypatch.delenv("INFERENCE_OPTIMIZER_RAY_EXEC", raising=False) + else: + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", ray_override) + _configure_benchmark_config(argparse.Namespace(resume_from=str(tmp_path), benchmark_config=None)) + assert "HYPERLOOM_BENCHMARK_CONFIG" not in os.environ + + _restore_agentx_env_from_state(state) + assert _restore_agentx_runtime_pins_from_state(state) == pins + + assert os.environ["HYPERLOOM_BENCHMARK_CONFIG"] == state.baseline_config_path + assert native_agentx_session() is True + assert _ray_backend._should_use_ray_backend() is False + assert _ray_backend.ray_gpu_specialist_exec_enabled() is False + assert ProfileExecutor()._resolve_default_config().name.startswith("profile_") + + +@pytest.mark.parametrize( + "backend, client", [("", "aiperf"), ("legacy", "aiperf"), ("aiperf", "aiperf"), ("mlperf", "mlperf")] +) +@pytest.mark.parametrize("raw_mode", ["", "1"]) +def test_bare_resume_restores_saved_legacy_client_without_upgrading_epoch(monkeypatch, backend, client, raw_mode): + from hyperloom.common.agentx_workload import agentx_client_script + + monkeypatch.setenv("HYPERLOOM_AGENTX", raw_mode) + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "") + state = _St("agentx", 1, backend) + assert agentx_state_is_stale(state) == "" + assert _restore_agentx_env_from_state(state) is (raw_mode != "1") + assert os.environ["HYPERLOOM_AGENTIC_BACKEND"] == client + assert agentx_client_script() == ("mlperf_agentic_client.sh" if client == "mlperf" else "aiperf_client.sh") + assert (state.agentx_epoch, state.agentx_backend) == (1, backend) + assert agentx_state_is_stale(state) == "" + + +@pytest.mark.parametrize("epoch", [2, 3, 4]) +@pytest.mark.parametrize("backend", ["", "native"]) +def test_saved_native_epoch_infers_missing_backend_and_rejects_mlperf(monkeypatch, epoch, backend): + from hyperloom.common.agentx_mode import managed_native_agentx_session, native_agentx_session + + monkeypatch.setenv("HYPERLOOM_AGENTX", "") + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "") + state = _St("agentx", epoch, backend) + assert agentx_state_is_stale(state) == "" + _restore_agentx_env_from_state(state) + assert native_agentx_session(state) + assert managed_native_agentx_session(state) is (epoch == 4) + assert state.agentx_epoch == epoch + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf") + assert "backend" in agentx_state_is_stale(state) + + +def test_fresh_mlperf_does_not_require_native_pins_but_conflicts_with_native_yaml(monkeypatch, tmp_path, capsys): + _blank_agentx_runtime_pins(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_BACKEND", "") + args = argparse.Namespace(framework="sglang", nodes=1, enable_conc_sweep=True) + _preflight_agentx_backend(args) + source = tmp_path / "native.yaml" + source.write_text("benchmark:\n agentx: enable\n") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + with pytest.raises(SystemExit) as exc: + _preflight_agentx_backend(args) + assert exc.value.code == 2 + assert "MLPerf client backend" in capsys.readouterr().err diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_fresh_source.py b/src/hyperloom/inference_optimizer/tests/test_agentx_fresh_source.py new file mode 100644 index 0000000000..dfec0fc6ea --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_fresh_source.py @@ -0,0 +1,367 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Public AgentX launches resolve a native recipe without model-specific defaults.""" + +from __future__ import annotations + +import argparse +import copy +import hashlib +import json +import os +from pathlib import Path + +import pytest +import yaml + +from hyperloom.inference_optimizer import cli +from hyperloom.inference_optimizer.cli.agentx_source import prepare_native_agentx_source + + +@pytest.fixture(autouse=True) +def fresh_launch_env(monkeypatch): + original = dict(os.environ) + + def managed(name): + return name.startswith(("AGENTX_", "HYPERLOOM_AGENTX", "HYPERLOOM_BENCHMARK_CONFIG")) or name in { + "INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", + "MODEL_PATH", + "HYPERLOOM_IMAGE", + } + + for name in tuple(os.environ): + if managed(name): + monkeypatch.setenv(name, "") + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + yield + source = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "") + if source and Path(source).name.startswith("hyperloom-agentx-"): + Path(source).unlink(missing_ok=True) + for name in tuple(os.environ): + if managed(name): + if name in original: + os.environ[name] = original[name] + else: + os.environ.pop(name) + + +def _args(**updates): + values = dict( + resume_from=None, + benchmark_config=None, + model="Qwen/Qwen3-32B", + framework="vllm", + precision="bf16", + gpu_type="mi300x", + conc=8, + tp=None, + ep=None, + ) + values.update(updates) + return argparse.Namespace(**values) + + +@pytest.mark.parametrize( + "model,framework,runner,precision", + [ + ("Qwen/Qwen3-32B", "vllm", "mi300x", "bf16"), + ("moonshotai/Kimi-K2-Instruct", "sglang", "mi355x", "fp8"), + ], +) +def test_fresh_switch_builds_native_source_and_resolver_owns_launcher( + monkeypatch, tmp_path, model, framework, runner, precision +): + args = _args(model=model, framework=framework, gpu_type=runner, precision=precision) + assert cli._configure_benchmark_config(args) + assert prepare_native_agentx_source(args) + source = Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]) + benchmark = yaml.safe_load(source.read_text())["benchmark"] + assert benchmark["model"] == model + assert benchmark["framework"] == framework + assert benchmark["runner_type"] == runner + assert benchmark["precision"] == precision + assert benchmark["agentx"]["launch_overrides"] == {"version": 1} + assert benchmark["envs"]["CONC"] == 8 + assert "benchmark_script" not in benchmark + assert os.environ["HYPERLOOM_BENCHMARK_CONFIG_SHA256"] == hashlib.sha256(source.read_bytes()).hexdigest() + from hyperloom.inference_optimizer.agentx import native + + launcher = f"single_node/agentic/{framework}_resolved_model.sh" + + def resolve(benchmark, *, inferencex_path): + assert "benchmark_script" not in benchmark + assert benchmark["agentx"]["launch_overrides"] == {"version": 1} + return { + "benchmark": dict(benchmark, benchmark_script=launcher), + "recipe": "unique-recipe", + "config_file": "configs/amd-master.yaml", + "entry": {"image": "image@sha256:abc"}, + "topology": {"gpu_count": 4, "ep": 4, "conc": 8, "recipe_fingerprint": "a" * 64}, + } + + monkeypatch.setattr(native, "preview_native_recipe", resolve) + monkeypatch.setattr( + native, "native_execution_identity", lambda **kwargs: {"static_execution_fingerprint": "b" * 64} + ) + monkeypatch.setenv("INFERENCEX_PATH", str(tmp_path)) + assert cli._finalize_benchmark_config(args) + assert os.environ["AGENTX_SERVER_SCRIPT"] == launcher + assert args.tp == 4 + assert args.ep == 4 + + +@pytest.mark.parametrize("path", ["/runtime/bin:/usr/bin", "/opt/venv/bin:/runtime/bin:/usr/bin"]) +def test_cli_recipe_identity_survives_materialization_and_rejects_runtime_drift(monkeypatch, tmp_path, path): + from hyperloom.inference_optimizer.agentx import native + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256, native_workload_fingerprint + from hyperloom.orchestrator.actions.executors._workload_envs import materialize_config_with_envs + + monkeypatch.setenv("PATH", path) + monkeypatch.setenv("PYTHONPATH", "/runtime/packages") + monkeypatch.setenv("LD_LIBRARY_PATH", "/runtime/lib") + monkeypatch.setenv("LIBRARY_PATH", "/runtime/link") + monkeypatch.setenv("ROCR_VISIBLE_DEVICES", "0,1,2,3") + monkeypatch.setenv("TP", "4") + monkeypatch.setenv("EP", "4") + monkeypatch.setenv("CONC", "8") + monkeypatch.setenv("INFERENCEX_PATH", str(tmp_path)) + + def resolve(benchmark, *, inferencex_path): + current = copy.deepcopy(benchmark) + current["benchmark_script"] = "srt_agentic.sh" + current["docker_image"] = "fixture/image:fixed" + current["timeout_seconds"] = 7200 + server_env = { + key: current["envs"][key] + for key in ("PATH", "PYTHONPATH", "LD_LIBRARY_PATH", "LIBRARY_PATH") + if key in current["envs"] + } + fingerprint = canonical_sha256(server_env) + return { + "benchmark": current, + "recipe": "fixture-recipe", + "config_file": "configs/amd-master.yaml", + "entry": {"image": current["docker_image"], "recipe-fingerprint": fingerprint}, + "topology": { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "gpu_count": 4, + "ep": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": fingerprint, + }, + } + + monkeypatch.setattr(native, "preview_native_recipe", resolve) + monkeypatch.setattr( + native, + "native_execution_identity", + lambda **kwargs: { + "static_execution_fingerprint": "b" * 64, + "workload_fingerprint": native_workload_fingerprint(kwargs["resolved_benchmark"], "b" * 64), + }, + ) + args = _args() + cli._configure_benchmark_config(args) + prepare_native_agentx_source(args) + cli._finalize_benchmark_config(args) + source = Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]) + result = materialize_config_with_envs(source, tmp_path / "baseline", agentx_mode=True, native_agentx_mode=True) + materialized = yaml.safe_load(result.read_text())["benchmark"] + assert materialized["envs"]["PATH"] == "/opt/venv/bin:/runtime/bin:/usr/bin" + assert ( + materialized["workload_spec"]["recipe"]["recipe_fingerprint"] + == os.environ["HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT"] + ) + candidate = materialize_config_with_envs( + result, + tmp_path / "candidate", + extra_server_args="--disable-cuda-graph", + agentx_mode=True, + native_agentx_mode=True, + ) + candidate_benchmark = yaml.safe_load(candidate.read_text())["benchmark"] + assert "AGENTX_PHASE_WAIT_TIMEOUT_S" not in candidate_benchmark["envs"] + assert native_workload_fingerprint(candidate_benchmark, "b" * 64) == native_workload_fingerprint( + materialized, + "b" * 64, + ) + assert candidate_benchmark["agentx"]["launch_overrides"]["append_args"] == ["--disable-cuda-graph"] + + monkeypatch.setenv("PYTHONPATH", "/another/runtime") + with pytest.raises(ValueError, match="recipe changed after session finalization"): + materialize_config_with_envs(source, tmp_path / "changed", agentx_mode=True, native_agentx_mode=True) + + +def test_local_checkpoint_needs_explicit_canonical_identity(monkeypatch, tmp_path): + args = _args(model=str(tmp_path)) + with pytest.raises(ValueError, match="canonical model id"): + prepare_native_agentx_source(args) + monkeypatch.setenv("AGENTX_MODEL_ID", "Qwen/Qwen3-32B") + assert prepare_native_agentx_source(args) + benchmark = yaml.safe_load(Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text())["benchmark"] + assert benchmark["model"] == "Qwen/Qwen3-32B" + assert benchmark["envs"]["MODEL_PATH"] == str(tmp_path) + + +def test_unknown_gpu_is_not_replaced_with_example_gpu(): + with pytest.raises(ValueError, match="detected GPU"): + prepare_native_agentx_source(_args(gpu_type=None)) + + +def test_fresh_source_preserves_operator_file_and_selectors(tmp_path): + source = tmp_path / "source.yaml" + source.write_text( + yaml.safe_dump( + { + "benchmark": { + "model": "Qwen/Qwen3-32B", + "agentx": {"recipe": "explicit-recipe", "selector": {"tp": 4}, "concurrency": 8}, + } + } + ) + ) + original = source.read_bytes() + args = _args(benchmark_config=str(source)) + cli._configure_benchmark_config(args) + prepare_native_agentx_source(args) + generated = yaml.safe_load(Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text())["benchmark"] + assert source.read_bytes() == original + assert generated["agentx"]["recipe"] == "explicit-recipe" + assert generated["agentx"]["selector"] == {"tp": 4} + assert generated["agentx"]["launch_overrides"] == {"version": 1} + + +def test_unsupported_recipe_does_not_fall_back_to_legacy(monkeypatch, tmp_path): + args = _args() + prepare_native_agentx_source(args) + from hyperloom.inference_optimizer.agentx import native + + def ambiguous(*args, **kwargs): + raise ValueError("multiple AgentX recipes: choose agentx.recipe") + + monkeypatch.setattr(native, "preview_native_recipe", ambiguous) + monkeypatch.setenv("INFERENCEX_PATH", str(tmp_path)) + with pytest.raises(ValueError, match="multiple AgentX recipes"): + cli._finalize_benchmark_config(args) + assert "aiperf_client.sh" not in Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text() + + +def test_resume_does_not_generate_new_source(): + assert not prepare_native_agentx_source(_args(resume_from="existing-session")) + + +def test_fresh_entry_clears_previous_session_before_mode_selection(monkeypatch, tmp_path): + from hyperloom.common.agentx_mode import native_agentx_optimization_session + + (tmp_path / "state.json").write_text(json.dumps({"benchmark_mode": "agentx", "agentx_epoch": 1})) + monkeypatch.setenv("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", str(tmp_path)) + assert not native_agentx_optimization_session() + assert cli._configure_benchmark_config(_args()) + assert "INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR" not in os.environ + assert native_agentx_optimization_session() + + +@pytest.mark.parametrize("epoch", [1, 3]) +def test_resume_preflight_uses_saved_mode_and_restores_previous_pointer(monkeypatch, tmp_path, epoch): + from hyperloom.common.agentx_mode import native_agentx_session + + session = tmp_path / "resumed" + session.mkdir() + (session / "state.json").write_text(json.dumps({"benchmark_mode": "agentx", "agentx_epoch": epoch})) + previous = str(tmp_path / "previous-session") + monkeypatch.setenv("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", previous) + + def preflight(args): + assert native_agentx_session() is (epoch >= 2) + return {"provider": "test"} + + monkeypatch.setattr(cli, "_preflight", preflight) + assert cli._preflight_for_session(_args(resume_from=str(session))) == {"provider": "test"} + assert os.environ["INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR"] == previous + + +def test_source_preserves_explicit_launch_request(tmp_path): + source = tmp_path / "source.yaml" + request = {"version": 1, "append_args": ["--mem-fraction-static", "0.8"], "env": {"SGLANG_USE_AITER": "1"}} + source.write_text( + yaml.safe_dump({"benchmark": {"model": "Qwen/Qwen3-32B", "agentx": {"launch_overrides": request}}}) + ) + args = _args(benchmark_config=str(source)) + cli._configure_benchmark_config(args) + prepare_native_agentx_source(args) + generated = yaml.safe_load(Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text())["benchmark"] + assert generated["agentx"]["launch_overrides"] == request + + +def test_custom_model_forwards_explicit_topology_context_image_and_local_checkpoint(monkeypatch, tmp_path): + monkeypatch.setenv("HYPERLOOM_IMAGE", "operator/image@sha256:abc") + monkeypatch.setenv("AGENTX_MODEL_ID", "operator/custom-model") + args = _args(model=str(tmp_path), tp=4, ep=2, max_model_len=32768) + prepare_native_agentx_source(args) + benchmark = yaml.safe_load(Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text())["benchmark"] + assert benchmark["docker_image"] == "operator/image@sha256:abc" + assert benchmark["model"] == "operator/custom-model" + assert benchmark["envs"] == { + "TP": 4, + "EP_SIZE": 2, + "CONC": 8, + "MAX_MODEL_LEN": 32768, + "AGENTX_MODEL_ID": "operator/custom-model", + "MODEL_PATH": str(tmp_path), + } + + +def test_custom_source_topology_survives_unspecified_cli_defaults(tmp_path): + source = tmp_path / "custom.yaml" + source.write_text( + yaml.safe_dump( + { + "benchmark": { + "model": "operator/custom-model", + "docker_image": "operator/custom-image", + "agentx": {"enabled": True}, + "envs": {"TP": 4, "EP_SIZE": 2, "MAX_MODEL_LEN": 32768}, + } + } + ) + ) + args = _args(benchmark_config=str(source), model=None, max_model_len=None) + cli._configure_benchmark_config(args) + prepare_native_agentx_source(args) + benchmark = yaml.safe_load(Path(os.environ["HYPERLOOM_BENCHMARK_CONFIG"]).read_text())["benchmark"] + assert benchmark["docker_image"] == "operator/custom-image" + assert benchmark["envs"]["TP"] == 4 + assert benchmark["envs"]["EP_SIZE"] == 2 + assert benchmark["envs"]["MAX_MODEL_LEN"] == 32768 + + +def test_conflicting_explicit_image_fails_before_recipe_resolution(monkeypatch, tmp_path): + source = tmp_path / "custom.yaml" + source.write_text(yaml.safe_dump({"benchmark": {"model": "operator/custom-model", "docker_image": "source/image"}})) + args = _args(benchmark_config=str(source), model=None) + cli._configure_benchmark_config(args) + monkeypatch.setenv("HYPERLOOM_IMAGE", "operator/different-image") + with pytest.raises(ValueError, match="HYPERLOOM_IMAGE conflicts"): + prepare_native_agentx_source(args) + + +@pytest.mark.parametrize( + "name", + ["AIPERF_BIN", "AGENTIC_CONCURRENCY", "AGENTX_DATASET", "AGENTX_WARMUP_REQUESTS_PER_LANE", "WEKA_LOADER_OVERRIDE"], +) +@pytest.mark.parametrize("value", ["private-value", 0, None, " "]) +def test_native_yaml_replay_controls_are_rejected_when_set(tmp_path, name, value): + source = tmp_path / "source.yaml" + source.write_text(yaml.safe_dump({"benchmark": {"agentx": "enable", "envs": {name: value}}})) + args = _args(benchmark_config=str(source)) + if value is None or not str(value).strip(): + assert cli._configure_benchmark_config(args) + else: + with pytest.raises(ValueError, match=name) as error: + cli._configure_benchmark_config(args) + assert "benchmark.envs" in str(error.value) + assert "private-value" not in str(error.value) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_geak_proxy.py b/src/hyperloom/inference_optimizer/tests/test_agentx_geak_proxy.py new file mode 100644 index 0000000000..8427b8f4e6 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_geak_proxy.py @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""GEAK receives native tunables but never inherits canonical performance credit.""" + +import shlex + +import pytest + +from hyperloom.inference_optimizer.agentx.geak_proxy import seed_native_geak_proxy +from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + + +@pytest.mark.parametrize( + ("framework", "argv"), + [ + ("sglang", ["/runtime/bin/python", "-m", "sglang.launch_server", "--model-path", "/models/model"]), + ("vllm", ["/runtime/bin/vllm", "serve", "/models/model"]), + ], +) +def test_geak_effective_config_receives_observed_tunables_but_owns_proxy_endpoint(framework, argv): + flags = ["--kv-cache-dtype", "fp8", "--json-option", '{"key": "two words"}'] + launch = { + "effective_argv": [*argv, "--port", "8123", "--tensor-parallel-size", "4", *flags], + "runtime_environment": { + "PYTHONPATH": "/patched/runtime", + "SGLANG_USE_AITER": "1", + "HIP_VISIBLE_DEVICES": "0,1", + }, + } + launch["evidence_sha256"] = canonical_sha256(launch) + handoff = { + "framework": framework, + "raw_baseline_tput": 200.0, + "baseline_env_spec": {"config": {"extra_server_args": "--stale-delta"}, "source_snapshots": [{"id": "source"}]}, + } + seed_native_geak_proxy(handoff, benchmark={}, measurement={"agentx_server_launch": launch}) + config = handoff["baseline_env_spec"]["config"] + assert shlex.split(config["server_launch_flags"]) == flags + assert config["extra_server_args"] == "" + assert handoff["accepted_flags"] == "" + assert config["extra_envs"] == {"PYTHONPATH": "/patched/runtime", "SGLANG_USE_AITER": "1", "PATH": "/runtime/bin"} + assert handoff["baseline_env_spec"]["source_snapshots"] == [{"id": "source"}] + assert handoff["native_server_launch"] == launch + assert handoff["raw_baseline_tput"] == handoff["orchestrator_best_tput_same_config"] == 0.0 + assert handoff["same_config_reference_verification_status"] == "unverified_workload" + + +@pytest.mark.parametrize("evidence", [None, {}, {"effective_argv": ["python"], "evidence_sha256": "0" * 64}]) +def test_geak_rejects_missing_or_changed_native_launch_evidence(evidence): + with pytest.raises(ValueError, match="launch"): + seed_native_geak_proxy({}, benchmark={}, measurement={"agentx_server_launch": evidence}) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_managed_identity.py b/src/hyperloom/inference_optimizer/tests/test_agentx_managed_identity.py new file mode 100644 index 0000000000..d096ca6284 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_managed_identity.py @@ -0,0 +1,279 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Managed AgentX binds native recipes and nested dependencies without shell manifests.""" + +from __future__ import annotations + +import hashlib +import json +import subprocess +from types import SimpleNamespace +from copy import deepcopy + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx import identity, managed, native +from hyperloom.inference_optimizer.agentx.identity import canonical_sha256, validate_server_launch +from hyperloom.inference_optimizer.cli import preflight +from hyperloom.inference_optimizer.tests.test_agentx_native import _commit_all, _init_git_repo +from hyperloom.inference_optimizer.tests.test_agentx_optimization_identity import _candidate_artifacts + + +def _managed_checkout(tmp_path, *, nested=True, custom=False): + repository = tmp_path / "Inference X" + _init_git_repo(repository) + project = repository / "inferencex-e2e" if nested else repository + files = ["benchmarks/srt_agentic.sh", "benchmarks/benchmark_lib.sh", "configs/runners.yaml", "infx/client.py"] + if not custom: + files.append("configs/model.yaml") + for name in files: + path = project / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("# immutable recipe input\n") + dependency = project / "utils/aiperf" + _init_git_repo(dependency) + (dependency / "pyproject.toml").write_text("[project]\nname = 'aiperf'\n") + aiperf = _commit_all(dependency) + revision = _commit_all(repository) + spec = { + "version": 1, + "framework": "sglang", + "argv": ["python3", "-m", "sglang.launch_server"], + "env": {}, + "client_env": {}, + "source_files": { + str(project / name): hashlib.sha256((project / name).read_bytes()).hexdigest() for name in files + }, + "client_revision": revision, + "aiperf_revision": aiperf, + } + benchmark = { + "framework": "sglang", + "benchmark_script": "srt_agentic.sh", + "agentx": {"enabled": True, "launch_overrides": {"version": 1}, "resolved": {"server-launch-spec": spec}}, + } + arguments = { + "inferencex_path": repository, + "benchmark_script": "srt_agentic.sh", + "config_file": "magpie:custom" if custom else "configs/model.yaml", + "resolved_benchmark": benchmark, + "expected_ref": revision, + "magpie_execution": {"source_commit": "a" * 40, "fingerprint": "b" * 64}, + "expected_magpie_ref": "a" * 40, + "launch_env": {}, + } + return project, arguments + + +@pytest.mark.parametrize("nested", [False, True]) +@pytest.mark.parametrize("custom", [False, True]) +def test_managed_identity_accepts_both_project_roots_and_custom_recipe(tmp_path, nested, custom): + project, arguments = _managed_checkout(tmp_path, nested=nested, custom=custom) + repository_identity = native.native_execution_identity(**arguments) + arguments["inferencex_path"] = project + project_identity = native.native_execution_identity(**arguments) + assert repository_identity == project_identity + assert project_identity["execution_owner"] == "magpie" + assert project_identity["launcher"] == "benchmarks/srt_agentic.sh" + assert project_identity["config_file"] == arguments["config_file"] + assert project_identity["aiperf_commit"] == managed.aiperf_revision(project) + assert ( + native.resolve_native_launcher(inferencex_path=project, benchmark_script="srt_agentic.sh") + == project / "benchmarks/srt_agentic.sh" + ) + assert preflight._inferencex_checkout_ok(project, ref=arguments["expected_ref"], require_agentx_submodule=True) + assert preflight._inferencex_checkout_ok( + arguments["inferencex_path"], ref=arguments["expected_ref"], require_agentx_submodule=True + ) + + +@pytest.mark.parametrize( + "change", ["client", "untracked_python", "dependency", "gitlink", "spec_revision", "source_omission"] +) +def test_managed_identity_rejects_modified_execution_inputs(tmp_path, change): + project, arguments = _managed_checkout(tmp_path) + spec = arguments["resolved_benchmark"]["agentx"]["resolved"]["server-launch-spec"] + if change == "client": + (project / "benchmarks/srt_agentic.sh").write_text("# changed\n") + elif change == "untracked_python": + (project / "infx/injected.py").write_text("# untracked import input\n") + elif change == "dependency": + (project / "utils/aiperf/pyproject.toml").write_text("# modified harness\n") + elif change == "gitlink": + (project / "utils/aiperf/pyproject.toml").write_text("# other harness revision\n") + _commit_all(project / "utils/aiperf") + elif change == "spec_revision": + spec["client_revision"] = "d" * 40 + else: + spec["source_files"].pop(str(project / "benchmarks/srt_agentic.sh")) + with pytest.raises(ValueError): + native.native_execution_identity(**arguments) + + +def test_managed_candidate_changes_do_not_change_static_workload_identity(tmp_path): + _, arguments = _managed_checkout(tmp_path) + baseline = native.native_execution_identity(**arguments) + arguments["resolved_benchmark"]["agentx"]["launch_overrides"] = {"version": 1, "env": {"SGLANG_USE_AITER": "1"}} + candidate = native.native_execution_identity(**arguments) + assert candidate["static_execution_fingerprint"] == baseline["static_execution_fingerprint"] + assert candidate["workload_fingerprint"] == baseline["workload_fingerprint"] + assert candidate["execution_fingerprint"] != baseline["execution_fingerprint"] + + +@pytest.mark.parametrize("change", [None, "owner", "source", "spec", "argv", "unexpected_profile"]) +def test_managed_receipt_binds_magpie_recipe_before_accepting_candidate(tmp_path, monkeypatch, change): + monkeypatch.setattr(identity, "_validate_managed_receipt", lambda *_args: []) + config, report = _candidate_artifacts(tmp_path) + evidence = report["agentx_metrics"]["server_launch"] + spec = {"version": 1, "argv": list(evidence["base_argv"]), "source_files": {"/recipe.yaml": "a" * 64}} + config["agentx"]["resolved"]["server-launch-spec"] = spec + evidence.update( + owner="magpie", server_spec_sha256=canonical_sha256(spec), recipe_source_files=deepcopy(spec["source_files"]) + ) + if change == "owner": + evidence["owner"] = "shell" + elif change == "source": + evidence["recipe_source_files"] = {} + elif change == "spec": + evidence["server_spec_sha256"] = "b" * 64 + elif change == "argv": + evidence["base_argv"] = evidence["effective_argv"] = ["unrelated-server"] + elif change == "unexpected_profile": + evidence["torch_profiler"] = {"enabled": True} + evidence["evidence_sha256"] = canonical_sha256( + {key: value for key, value in evidence.items() if key != "evidence_sha256"} + ) + errors = validate_server_launch(config, evidence) + assert bool(errors) is bool(change) + + +def test_managed_profiler_receipt_requires_magpie_validation(tmp_path, monkeypatch): + config, report = _candidate_artifacts(tmp_path) + evidence = report["agentx_metrics"]["server_launch"] + spec = {"version": 1, "argv": list(evidence["base_argv"]), "source_files": {}} + config["agentx"]["resolved"]["server-launch-spec"] = spec + config["profiler"] = {"torch_profiler": {"enabled": True}} + evidence.update(owner="magpie", recipe_source_files={}, torch_profiler={"capture_id": "a" * 32}) + evidence["evidence_sha256"] = canonical_sha256( + {key: value for key, value in evidence.items() if key != "evidence_sha256"} + ) + assert "server_launch_profile_workspace_missing" in validate_server_launch(config, evidence) + + +def _diagnostic_arguments(tmp_path): + project, arguments = _managed_checkout(tmp_path) + accepted = arguments["resolved_benchmark"] + accepted["profiler"] = {"torch_profiler": {"enabled": False}} + execution = native.native_execution_identity(**arguments) + accepted["workload_spec"] = {"execution": execution} + parent = tmp_path / "accepted.yaml" + parent.write_text(yaml.safe_dump({"benchmark": accepted})) + diagnostic = deepcopy(accepted) + diagnostic["profiler"] = {"torch_profiler": {"enabled": True, "num_profiles": 3, "num_steps": 20}} + diagnostic["workload_spec"]["profile_parent"] = { + "materialized_config": str(parent), + "config_sha256": hashlib.sha256(parent.read_bytes()).hexdigest(), + "workload_fingerprint": execution["workload_fingerprint"], + } + arguments["resolved_benchmark"] = diagnostic + arguments["launch_env"] = {"HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT": execution["workload_fingerprint"]} + return arguments, execution, parent + + +def test_managed_diagnostics_retain_verified_parent_identity_and_bind_candidate_overrides(tmp_path): + arguments, accepted_execution, _ = _diagnostic_arguments(tmp_path) + diagnostic_execution = native.native_execution_identity(**arguments) + assert diagnostic_execution == accepted_execution + diagnostic = arguments["resolved_benchmark"] + assert diagnostic["profiler"]["torch_profiler"]["enabled"] is True + diagnostic["agentx"]["launch_overrides"] = {"version": 1, "env": {"SGLANG_USE_AITER": "1"}} + candidate_execution = native.native_execution_identity(**arguments) + assert candidate_execution["workload_fingerprint"] == accepted_execution["workload_fingerprint"] + assert candidate_execution["static_execution_fingerprint"] == accepted_execution["static_execution_fingerprint"] + assert candidate_execution["execution_fingerprint"] != accepted_execution["execution_fingerprint"] + + +@pytest.mark.parametrize("change", ["model", "concurrency", "parent", "session_pin", "profile_flag", "missing_parent"]) +def test_managed_diagnostic_derivation_cannot_change_parent_workload(tmp_path, change): + arguments, _, parent = _diagnostic_arguments(tmp_path) + diagnostic = arguments["resolved_benchmark"] + if change == "model": + diagnostic["model"] = "other/model" + elif change == "concurrency": + diagnostic["envs"] = {"CONC": 999} + elif change == "parent": + parent.write_text(parent.read_text() + "# changed after acceptance\n") + elif change == "session_pin": + arguments["launch_env"]["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] = "c" * 64 + elif change == "profile_flag": + diagnostic["profiler"]["torch_profiler"]["enabled"] = False + else: + diagnostic["workload_spec"].pop("profile_parent") + with pytest.raises(ValueError, match="profile|diagnostic"): + native.native_execution_identity(**arguments) + + +@pytest.mark.parametrize( + "framework, flag", + [ + ("vllm", "--profiler-config"), + ("sglang", "--enable-profile-cuda-graph"), + ("sglang", "--enable_shape_discovery_for_cuda_graph_profile"), + ], +) +def test_managed_replace_profiler_argv_uses_audited_magpie_contract(tmp_path, monkeypatch, framework, flag): + config, report = _candidate_artifacts(tmp_path) + evidence = report["agentx_metrics"]["server_launch"] + config["framework"] = evidence["framework"] = framework + config["profiler"] = {"torch_profiler": {"enabled": True}} + config["agentx"]["launch_overrides"] = {"version": 1, "replace_args": True} + spec = {"version": 1, "argv": evidence["base_argv"], "source_files": {}} + config["agentx"]["resolved"]["server-launch-spec"] = spec + module = "vllm.entrypoints.openai.api_server" if framework == "vllm" else "sglang.launch_server" + options = [flag, '{"profiler": "torch"}'] if framework == "vllm" else [flag] + evidence["base_argv"] = ["python3", "-m", module, *options, "--optional-candidate", "1"] + evidence["effective_argv"] = evidence["base_argv"][:-2] + evidence.update( + owner="magpie", recipe_source_files={}, overrides_sha256=canonical_sha256(config["agentx"]["launch_overrides"]) + ) + evidence["evidence_sha256"] = canonical_sha256({k: v for k, v in evidence.items() if k != "evidence_sha256"}) + calls = [] + + def verify(command, **kwargs): + payload = json.loads(kwargs["input"]) + assert payload["profile"] is True + assert payload["workspace"] == str(tmp_path) + assert payload["evidence"]["effective_argv"] == evidence["effective_argv"] + assert "_validate_magpie_execution_tree(package, commit)" in command[-1] + assert 'read_launch_evidence(config, Path(payload["workspace"]))' in command[-1] + calls.append(command) + return SimpleNamespace(returncode=0) + + monkeypatch.setattr(identity.subprocess, "run", verify) + assert validate_server_launch(config, evidence, workspace=tmp_path) == [] + assert len(calls) == 1 + + +@pytest.mark.parametrize("failure", [None, "nonzero", "timeout"]) +def test_managed_accepted_launch_uses_magpie_argv_validation_without_workspace(tmp_path, monkeypatch, failure): + config, report = _candidate_artifacts(tmp_path) + evidence = report["agentx_metrics"]["server_launch"] + spec = {"version": 1, "argv": list(evidence["base_argv"]), "source_files": {}} + config["agentx"]["resolved"]["server-launch-spec"] = spec + evidence.update(owner="magpie", recipe_source_files={}, server_spec_sha256=canonical_sha256(spec)) + evidence["evidence_sha256"] = canonical_sha256({k: v for k, v in evidence.items() if k != "evidence_sha256"}) + + def verify(command, **kwargs): + payload = json.loads(kwargs["input"]) + assert payload["profile"] is False + assert 'apply_launch_args(evidence["base_argv"]' in command[-1] + assert "_validate_golden_launch(" in command[-1] + if failure == "timeout": + raise subprocess.TimeoutExpired(command, 120) + return SimpleNamespace(returncode=int(failure is not None)) + + monkeypatch.setattr(identity.subprocess, "run", verify) + errors = validate_server_launch(config, evidence) + assert errors == (["server_launch_managed_verification_failed"] if failure else []) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_mode.py b/src/hyperloom/inference_optimizer/tests/test_agentx_mode.py new file mode 100644 index 0000000000..1098fcbf8d --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_mode.py @@ -0,0 +1,225 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native AgentX selection preserves the legacy CLI and persisted sessions.""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +from types import SimpleNamespace + +import pytest +import yaml + +from hyperloom.common.agentx_mode import ( + config_enables_native_agentx, + managed_native_agentx_session, + native_agentx_session, + native_agentx_optimization_session, +) +from hyperloom.inference_optimizer.cli.preflight import _native_agentx_preflight_requested + + +@pytest.mark.parametrize( + "selector, expected", [("enable", True), ({"enabled": True}, True), (False, False), (None, False)] +) +def test_fresh_native_selection_accepts_source_yaml_or_public_switch(tmp_path, selector, expected): + path = tmp_path / "benchmark.yaml" + path.write_text(yaml.safe_dump({"benchmark": {"agentx": selector}}), encoding="utf-8") + env = {"HYPERLOOM_BENCHMARK_CONFIG": str(path)} + assert native_agentx_session(env=env) is expected + assert native_agentx_session(env={"HYPERLOOM_AGENTX": "1"}) is True + + +@pytest.mark.parametrize("raw", ["[", "[]", "benchmark: []", "benchmark: {}"]) +def test_invalid_or_generic_yaml_does_not_select_native(tmp_path, raw): + path = tmp_path / "source.yaml" + path.write_text(raw, encoding="utf-8") + assert config_enables_native_agentx(path) is False + assert config_enables_native_agentx(tmp_path / "missing.yaml") is False + + +@pytest.mark.parametrize("mapping", [True, False]) +def test_saved_legacy_or_synthetic_state_outranks_ambient_native_config(tmp_path, mapping): + source = tmp_path / "native.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + env = {"HYPERLOOM_BENCHMARK_CONFIG": str(source), "HYPERLOOM_AGENTX": "1"} + for mode in ("agentx", "synthetic"): + fields = {"benchmark_mode": mode, "agentx_epoch": 1} + state = fields if mapping else SimpleNamespace(**fields) + assert native_agentx_session(state, env=env) is False + + +def test_profile_template_cannot_downgrade_native_session_identity(tmp_path): + source = tmp_path / "source.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + profile = tmp_path / "profile.yaml" + profile.write_text("benchmark:\n profiling: enable\n", encoding="utf-8") + state = SimpleNamespace( + benchmark_mode="agentx", baseline_config_path=str(profile), benchmark_source_config_path=str(source) + ) + assert native_agentx_session(state, env={"HYPERLOOM_BENCHMARK_CONFIG": str(profile)}) is True + state.benchmark_source_config_path = "" + state.agentx_epoch = 2 + assert native_agentx_session(state, env={}) is True + + +@pytest.mark.parametrize("epoch, expected", [(1, False), (2, True)]) +def test_preflight_uses_saved_identity_before_exporting_native_pins(tmp_path, monkeypatch, epoch, expected): + source = tmp_path / "source.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + monkeypatch.setenv("MAGPIE_REF", "") + monkeypatch.setenv("INFERENCEX_REF", "") + pins = {"MAGPIE_REF": "a" * 40, "INFERENCEX_REF": "b" * 40, "AGENTX_SERVER_SCRIPT": "native.sh"} + (tmp_path / "state.json").write_text( + json.dumps({"benchmark_mode": "agentx", "agentx_epoch": epoch, "agentx_runtime_pins": pins}), encoding="utf-8" + ) + assert _native_agentx_preflight_requested(argparse.Namespace(resume_from=str(tmp_path))) is expected + assert os.environ["MAGPIE_REF"] == (pins["MAGPIE_REF"] if expected else "") + + +def test_preflight_selects_native_for_fresh_switch_but_not_unreadable_resume(monkeypatch, tmp_path): + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", "native.sh") + assert _native_agentx_preflight_requested(argparse.Namespace()) is True + assert _native_agentx_preflight_requested(argparse.Namespace(resume_from=str(tmp_path))) is False + + +@pytest.mark.parametrize("selector", [None, False, "disable", "off", {"enabled": False}, {"enabled": "false"}]) +def test_generic_config_keeps_native_package_lazy_in_fresh_process(tmp_path, selector): + config = tmp_path / "generic.yaml" + benchmark = {"framework": "sglang"} + if selector is not None: + benchmark["agentx"] = selector + config.write_text(yaml.safe_dump({"benchmark": benchmark}), encoding="utf-8") + code = """ +import sys +from hyperloom.common.agentx_mode import native_agentx_session +from hyperloom.orchestrator.actions.executors._ray_backend import _should_use_ray_backend +assert not native_agentx_session() +_should_use_ray_backend() +assert not any(name.startswith('hyperloom.inference_optimizer.agentx') for name in sys.modules) +""" + env = dict(os.environ, HYPERLOOM_BENCHMARK_CONFIG=str(config), HYPERLOOM_AGENTX="0") + result = subprocess.run([sys.executable, "-c", code], env=env, capture_output=True, text=True) + assert result.returncode == 0, result.stdout + result.stderr + + +@pytest.mark.parametrize( + "epoch,native,optimize", [(1, False, False), (2, True, False), (3, True, True), (4, True, True)] +) +def test_saved_epoch_controls_subprocess_routing(tmp_path, epoch, native, optimize): + state = {"benchmark_mode": "agentx", "agentx_epoch": epoch} + (tmp_path / "state.json").write_text(json.dumps(state), encoding="utf-8") + env = {"INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR": str(tmp_path), "HYPERLOOM_AGENTX": "1"} + assert native_agentx_session(env=env) is native + assert native_agentx_optimization_session(env=env) is optimize + assert native_agentx_optimization_session(state, env={"HYPERLOOM_AGENTX": "1"}) is optimize + assert managed_native_agentx_session(env=env) is (epoch == 4) + assert managed_native_agentx_session(state, env={"HYPERLOOM_AGENTX": "1"}) is (epoch == 4) + + +def test_epoch_one_outranks_saved_foreign_config(tmp_path): + source = tmp_path / "foreign.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + state = {"benchmark_mode": "agentx", "agentx_epoch": 1, "baseline_config_path": str(source)} + assert not native_agentx_session(state, env={"HYPERLOOM_AGENTX": "1"}) + + +def test_fresh_agentx_optimization_has_no_extra_switch(): + assert native_agentx_optimization_session(env={"HYPERLOOM_AGENTX": "1"}) + assert not native_agentx_optimization_session(env={"HYPERLOOM_AGENTX": "0"}) + assert managed_native_agentx_session(env={"HYPERLOOM_AGENTX": "1"}) + assert not managed_native_agentx_session(env={"HYPERLOOM_AGENTX": "0"}) + + +@pytest.mark.parametrize( + "epoch,warm_replay,backend", [(1, True, "legacy"), (2, False, "native"), (3, True, "native"), (4, True, "native")] +) +def test_persisted_epoch_keeps_backend_and_optimization_contract(epoch, warm_replay, backend): + from hyperloom.orchestrator.state.shared_state import SharedState + + state = SharedState.from_dict({"benchmark_mode": "agentx", "agentx_epoch": epoch, "warm_replay_enabled": True}) + assert state.agentx_epoch == epoch + assert state.agentx_backend == backend + assert state.warm_replay_enabled is warm_replay + restored = SharedState.from_dict(state.to_dict()) + assert restored.agentx_backend == backend + assert restored.agentx_epoch == epoch + assert restored.warm_replay_enabled is warm_replay + + +@pytest.mark.parametrize("client, native", [("", True), ("aiperf", True), ("mlperf", False)]) +def test_fresh_client_selection_keeps_managed_default_and_explicit_mlperf(client, native): + env = {"HYPERLOOM_AGENTX": "1", "HYPERLOOM_AGENTIC_BACKEND": client} + assert native_agentx_session(env=env) is native + assert managed_native_agentx_session(env=env) is native + + +@pytest.mark.parametrize("backend", ["legacy", "aiperf", "mlperf"]) +def test_persisted_epoch_one_client_survives_load_and_subprocess_routing(tmp_path, backend): + from hyperloom.orchestrator.state.shared_state import SharedState + + state = SharedState.from_dict({"benchmark_mode": "agentx", "agentx_epoch": 1, "agentx_backend": backend}) + state.save(tmp_path) + restored = SharedState.load_or_init(tmp_path) + assert restored.agentx_epoch == 1 + assert restored.agentx_backend == backend + env = {"HYPERLOOM_AGENTX": "1", "INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR": str(tmp_path)} + assert not native_agentx_session(env=env) + assert not managed_native_agentx_session(env=env) + + +@pytest.mark.parametrize( + "mode", ["native-switch", "native-yaml", "native-resume", "legacy-resume", "synthetic", "mlperf"] +) +@pytest.mark.parametrize( + "name", + ["AIPERF_BIN", "AGENTIC_CONCURRENCY", "AGENTX_DATASET", "AGENTX_WARMUP_REQUESTS_PER_LANE", "WEKA_LOADER_OVERRIDE"], +) +def test_cli_preflight_rejects_legacy_replay_env_only_for_native_sessions(tmp_path, monkeypatch, capsys, mode, name): + from hyperloom.inference_optimizer.cli import preflight + + class CredentialsReached(Exception): + pass + + def credentials(): + raise CredentialsReached + + monkeypatch.setattr(preflight, "_pin_resumed_session_args", lambda args: None) + monkeypatch.setattr(preflight, "_load_dotenv_fallback", lambda: None) + monkeypatch.setattr(preflight, "_load_kernel_agent_env_fallback", lambda: None) + monkeypatch.setattr(preflight, "_derive_runtime_paths", lambda: None) + monkeypatch.setattr(preflight, "_validate_credentials", credentials) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1" if mode in {"native-switch", "legacy-resume", "mlperf"} else "0") + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf" if mode == "mlperf" else "aiperf") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", "") + monkeypatch.setenv("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", "") + monkeypatch.setenv(name, "private-override-value") + args = argparse.Namespace(resume_from=None) + if mode == "native-yaml": + config = tmp_path / "native.yaml" + config.write_text("benchmark:\n agentx: enable\n") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(config)) + if mode.endswith("resume"): + (tmp_path / "state.json").write_text( + json.dumps({"benchmark_mode": "agentx", "agentx_epoch": 4 if mode == "native-resume" else 1}) + ) + args.resume_from = str(tmp_path) + if mode.startswith("native"): + with pytest.raises(SystemExit) as error: + preflight._preflight(args) + assert error.value.code == 2 + message = capsys.readouterr().err + assert name in message and "environment" in message + assert "private-override-value" not in message + else: + with pytest.raises(CredentialsReached): + preflight._preflight(args) + assert os.environ[name] == "private-override-value" diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_native.py b/src/hyperloom/inference_optimizer/tests/test_agentx_native.py new file mode 100644 index 0000000000..e37eec938e --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_native.py @@ -0,0 +1,1036 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native Magpie AgentX launcher validation tests.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx import native as native_agentx +from hyperloom.inference_optimizer.agentx.native import ( + native_agentx_enabled, + resolve_native_launcher, + resolve_native_recipe, + validate_native_launcher_name, +) + + +class _FakeDistribution: + def __init__(self, direct_url: dict | None, package_root: Path): + self._direct_url = direct_url + self._package_root = package_root + + def read_text(self, name: str) -> str | None: + assert name == "direct_url.json" + return json.dumps(self._direct_url) if self._direct_url is not None else None + + def locate_file(self, name: str) -> Path: + assert name == "Magpie" + return self._package_root + + +def _magpie_identity_namespace( + direct_url: dict | None, + *, + distribution_root: Path | None = None, +) -> dict[str, object]: + namespace: dict[str, object] = {} + exec(native_agentx._MAGPIE_SOURCE_IDENTITY_CODE, namespace) + + def _distribution(name: str): + if name != "magpie-eval": + pytest.fail(f"unexpected distribution: {name}") + if distribution_root is None: + raise namespace["PackageNotFoundError"] + return _FakeDistribution(direct_url, distribution_root) + + namespace["distribution"] = _distribution + return namespace + + +def _magpie_identity_resolver( + direct_url: dict | None, + *, + distribution_root: Path | None = None, +): + return _magpie_identity_namespace( + direct_url, + distribution_root=distribution_root, + )["_resolve_magpie_source_identity"] + + +def _init_git_repo(path: Path) -> str: + path.mkdir(parents=True, exist_ok=True) + subprocess.run(["git", "init", "-q", str(path)], check=True) + subprocess.run(["git", "-C", str(path), "config", "user.email", "test@example.com"], check=True) + subprocess.run(["git", "-C", str(path), "config", "user.name", "Hyperloom test"], check=True) + marker = path / "marker" + marker.write_text("outer\n", encoding="utf-8") + subprocess.run(["git", "-C", str(path), "add", "marker"], check=True) + subprocess.run(["git", "-C", str(path), "commit", "-qm", "fixture"], check=True) + return subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def _commit_all(path: Path, message: str = "fixture update") -> str: + subprocess.run(["git", "-C", str(path), "add", "."], check=True) + subprocess.run(["git", "-C", str(path), "commit", "-qm", message], check=True) + return subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def test_native_launcher_uses_the_pinned_upstream_manifest_and_rejects_edited_mapping(tmp_path, monkeypatch): + root = tmp_path / "InferenceX" + _init_git_repo(root) + script = "single_node/agentic/model.sh" + launcher = root / "benchmarks" / script + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/bash\nexit 0\n") + manifest = root / "configs" / "agentx-launchers.json" + manifest.parent.mkdir() + payload = { + "version": 1, + "recipes": {"model-recipe": {"benchmark_script": script, "launch_overrides_version": 1}}, + } + manifest.write_text(json.dumps(payload)) + commit = _commit_all(root) + monkeypatch.setattr(native_agentx, "_NATIVE_LAUNCH_CONTRACT_REFS", frozenset({commit})) + identity = native_agentx.validate_native_recipe_launcher( + inferencex_path=root, recipe="model-recipe", benchmark_script=script + ) + assert identity["inferencex_commit"] == commit + assert identity["launcher"] == script + + with pytest.raises(ValueError, match="requires audited launcher"): + native_agentx.validate_native_recipe_launcher( + inferencex_path=root, recipe="model-recipe", benchmark_script="single_node/agentic/other.sh" + ) + payload["recipes"]["model-recipe"]["benchmark_script"] = "single_node/agentic/other.sh" + manifest.write_text(json.dumps(payload)) + with pytest.raises(ValueError, match="manifest changed"): + native_agentx.validate_native_recipe_launcher( + inferencex_path=root, recipe="model-recipe", benchmark_script=script + ) + + +def test_magpie_wheel_identity_prefers_direct_url_over_unrelated_outer_git(tmp_path: Path): + outer = tmp_path / "hyperloom" + outer_commit = _init_git_repo(outer) + package_root = outer / ".venv" / "lib" / "python3.12" / "site-packages" / "Magpie" + package_root.mkdir(parents=True) + pinned_magpie_commit = "a" * 40 + resolve = _magpie_identity_resolver( + { + "url": "https://github.com/AMD-AGI/Magpie.git", + "vcs_info": { + "vcs": "git", + "commit_id": pinned_magpie_commit, + "requested_revision": pinned_magpie_commit, + }, + }, + distribution_root=package_root, + ) + + commit, url = resolve(package_root) + + assert commit == pinned_magpie_commit + assert commit != outer_commit + assert url == "https://github.com/AMD-AGI/Magpie.git" + + +def test_magpie_wheel_without_vcs_provenance_does_not_claim_outer_git(tmp_path: Path): + outer = tmp_path / "hyperloom" + _init_git_repo(outer) + package_root = outer / ".venv" / "lib" / "python3.12" / "site-packages" / "Magpie" + package_root.mkdir(parents=True) + resolve = _magpie_identity_resolver(None, distribution_root=package_root) + + assert resolve(package_root) == ("", "") + + +def test_magpie_source_checkout_uses_its_own_git_top_level(tmp_path: Path): + checkout = tmp_path / "Magpie-source" + _init_git_repo(checkout) + package_root = checkout / "Magpie" + package_root.mkdir() + (package_root / "__init__.py").write_text("# fixture\n", encoding="utf-8") + expected = _commit_all(checkout) + resolve = _magpie_identity_resolver(None) + + assert resolve(package_root) == (expected, checkout.resolve().as_uri()) + + +def test_magpie_editable_install_validates_declared_source_checkout(tmp_path: Path): + checkout = tmp_path / "Magpie-editable" + _init_git_repo(checkout) + package_root = checkout / "Magpie" + package_root.mkdir() + (package_root / "__init__.py").write_text("# fixture\n", encoding="utf-8") + expected = _commit_all(checkout) + resolve = _magpie_identity_resolver( + {"url": checkout.resolve().as_uri(), "dir_info": {"editable": True}}, + distribution_root=package_root, + ) + + assert resolve(package_root) == (expected, checkout.resolve().as_uri()) + + +def test_magpie_wheel_identity_rejects_distribution_import_mismatch(tmp_path: Path): + imported_root = tmp_path / "imported" / "Magpie" + imported_root.mkdir(parents=True) + distribution_root = tmp_path / "installed" / "Magpie" + distribution_root.mkdir(parents=True) + commit = "a" * 40 + resolve = _magpie_identity_resolver( + { + "url": "https://github.com/AMD-AGI/Magpie.git", + "vcs_info": {"vcs": "git", "commit_id": commit}, + }, + distribution_root=distribution_root, + ) + + with pytest.raises(RuntimeError, match="does not own the imported Magpie"): + resolve(imported_root) + + +def test_magpie_source_identity_rejects_dirty_package_tree(tmp_path: Path): + checkout = tmp_path / "Magpie-source" + _init_git_repo(checkout) + package_root = checkout / "Magpie" + package_root.mkdir() + module = package_root / "__init__.py" + module.write_text("# fixture\n", encoding="utf-8") + _commit_all(checkout) + module.write_text("# tampered\n", encoding="utf-8") + resolve = _magpie_identity_resolver(None) + + with pytest.raises(RuntimeError, match="clean Magpie/ source tree"): + resolve(package_root) + + +def test_magpie_source_identity_ignores_generated_pycache(tmp_path: Path): + checkout = tmp_path / "Magpie-source" + _init_git_repo(checkout) + package_root = checkout / "Magpie" + package_root.mkdir() + (package_root / "__init__.py").write_text("# fixture\n", encoding="utf-8") + expected = _commit_all(checkout) + cache = package_root / "__pycache__" + cache.mkdir() + (cache / "__init__.cpython-313.pyc").write_bytes(b"generated") + resolve = _magpie_identity_resolver(None) + + assert resolve(package_root) == (expected, checkout.resolve().as_uri()) + + +def test_magpie_published_execution_tree_pin_and_single_file_tamper(tmp_path: Path): + namespace = _magpie_identity_namespace(None) + package_root = tmp_path / "Magpie" + (package_root / "nested").mkdir(parents=True) + (package_root / "__init__.py").write_text("VALUE = 1\n", encoding="utf-8") + script = package_root / "nested" / "launch.sh" + script.write_text("#!/bin/sh\n", encoding="utf-8") + (package_root / "config.yaml").write_text("enabled: true\n", encoding="utf-8") + (package_root / "mcp").mkdir() + (package_root / "mcp" / "config.json").write_text("{}\n", encoding="utf-8") + (package_root / "__pycache__").mkdir() + (package_root / "__pycache__" / "ignored.py").write_text("tamper = True\n", encoding="utf-8") + + identify = namespace["_magpie_execution_tree_identity"] + validate = namespace["_validate_magpie_execution_tree"] + commit = "f" * 40 + pristine = identify(package_root) + namespace["_AUDITED_MAGPIE_EXECUTION_TREES"][commit] = pristine + + assert validate(package_root, commit) == pristine + script.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8") + with pytest.raises(RuntimeError, match="published execution tree differs"): + validate(package_root, commit) + + +def test_magpie_pinned_published_tree_hash_matches_independent_wheel_audit(): + namespace = _magpie_identity_namespace(None) + + assert namespace["_AUDITED_MAGPIE_EXECUTION_TREES"] == { + "5d6aaa29f2ee78e53ec35c8bc2e082d8a10c47f0": { + "file_count": 79, + "tree_sha256": ("afea9b95e992e0b715c955f5e72a7e92aca48d2c171ddff0e3d3e9082406e689"), + }, + "a3339dc2776ee0c977fb3313fe89f56da7a91555": { + "file_count": 79, + "tree_sha256": "113f880b18ccd3ec26e6a432fcdf06a0c365520d51c3ed3c46069d33d7f07e93", + }, + "d80eb4d3dad7fabe01ce81d049e2983adf2c86dd": { + "file_count": 87, + "tree_sha256": "4c827c3d4b6bb7f756964fdecaeb43b255470ee3285f5dc89d9d320c7e9164dc", + }, + "cf805323f0a6db6c63a0aa4ffcce7acf4e830ea6": { + "file_count": 87, + "tree_sha256": "65b4acccbe70c05c033f75a5054344bd2fead13dc81878f343427c6c04142e4a", + }, + "311d56429a3aadc3cfe1e1e0e2e533482d224cf0": { + "file_count": 87, + "tree_sha256": "f8be61cf4566f4daaf3c2463a600e49b11d97bade6632a7bb5b99050b4ce5656", + }, + "938bfd7a9bc1f3970f3fbce8b65ad88a8e0958f5": { + "file_count": 87, + "tree_sha256": "af43bc3fcbe768b30b87cd5a23ff6f4d36e0e865a8db7375ca4e205970990768", + }, + "63e3b19bee66700dbb012c538f121c9de0d86a2a": { + "file_count": 87, + "tree_sha256": "624a89b8e74adda9ab95e45d692cfb58a267b7e2d4de9a96076db24626d49e20", + }, + "c5c80698fef1b89cc6882264b80d5b306d4e9328": { + "file_count": 87, + "tree_sha256": "1f475d413e7af20204da8b8abcd06a6a69916d3960f175c72d627663ce33d94f", + }, + "658562345ad1a7e5a617e3631f3acfcec0eade4a": { + "file_count": 86, + "tree_sha256": "514ca3691163bb304cef31d392d9c10bbb5a3b3570b219f95ed12c6aeae29c69", + }, + "d72965776df5416dad063c00237f6e389b841162": { + "file_count": 81, + "tree_sha256": "84dd7920ff992571c7dc6d485a4680ac3fc2afe7b6dde606dbd754d545a5729c", + }, + } + assert namespace["_MAGPIE_UNPUBLISHED_PATHS"] == frozenset({"mcp/config.json"}) + + +def test_resolver_and_preflight_health_share_magpie_identity_guard(): + from hyperloom.inference_optimizer.magpie_health import magpie_health_code + + guard = native_agentx._MAGPIE_SOURCE_IDENTITY_CODE + assert native_agentx._RESOLVER_CODE.startswith(guard) + assert magpie_health_code(native_agentx=True).startswith(guard) + + +@pytest.mark.parametrize( + "custom_ref", + ["v0.2.0", "release/custom-magpie", "f" * 40], +) +def test_generic_magpie_health_keeps_custom_refs_on_importability_contract( + custom_ref: str, + monkeypatch: pytest.MonkeyPatch, +): + """A generic Magpie ref must not be rejected by native-only provenance policy.""" + from hyperloom.inference_optimizer.magpie_health import magpie_health_code + + code = magpie_health_code(native_agentx=False) + + assert code.strip() == "import Magpie" + assert "_validate_magpie_execution_tree" not in code + assert "AgentXConfig" not in code + monkeypatch.setitem(sys.modules, "Magpie", ModuleType("Magpie")) + monkeypatch.setattr(sys, "argv", ["magpie-health", custom_ref]) + exec(code, {}) + + +def test_native_launch_config_hash_binds_forwarded_environment() -> None: + base = { + "framework": "sglang", + "envs": {"CONC": 8, "PYTHONPATH": "/candidate/a"}, + "agentx": {"enabled": True}, + } + reordered = { + "agentx": {"enabled": True}, + "envs": {"PYTHONPATH": "/candidate/a", "CONC": 8}, + "framework": "sglang", + } + changed = {**base, "envs": {**base["envs"], "PYTHONPATH": "/candidate/b"}} + + assert native_agentx._canonical_benchmark_sha256(base) == native_agentx._canonical_benchmark_sha256(reordered) + assert native_agentx._canonical_benchmark_sha256(base) != native_agentx._canonical_benchmark_sha256(changed) + + +@pytest.mark.parametrize("value", [True, "enable", "enabled", {"enabled": True}]) +def test_native_agentx_enabled_values(value): + assert native_agentx_enabled(value) is True + + +@pytest.mark.parametrize("value", [False, "disabled", None, {"enabled": False}]) +def test_native_agentx_disabled_values(value): + assert native_agentx_enabled(value) is False + + +def test_validate_native_launcher_accepts_explicit_agentic_script(): + path = validate_native_launcher_name("single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh") + assert path.parts == ("single_node", "agentic", "dsv4_fp4_mi355x_sglang_mtp.sh") + + +@pytest.mark.parametrize( + "script", + ["", "/tmp/run.sh", "../run.sh", "sglang_mi355x.sh", "single_node/fixed_seq_len/run.sh"], +) +def test_validate_native_launcher_rejects_non_agentic_paths(script): + with pytest.raises(ValueError, match="single_node/agentic"): + validate_native_launcher_name(script) + + +def test_resolve_native_launcher_requires_existing_file(tmp_path: Path): + rel = "single_node/agentic/run.sh" + launcher = tmp_path / "benchmarks" / rel + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/usr/bin/env bash\n", encoding="utf-8") + + assert resolve_native_launcher(inferencex_path=tmp_path, benchmark_script=rel) == launcher + + +def test_resolve_native_launcher_does_not_modify_upstream(tmp_path: Path): + rel = "single_node/agentic/run.sh" + launcher = tmp_path / "benchmarks" / rel + launcher.parent.mkdir(parents=True) + original = "#!/usr/bin/env bash\nSGLANG_CMD=(python3 -m sglang.launch_server)\n" + launcher.write_text(original, encoding="utf-8") + + resolve_native_launcher(inferencex_path=tmp_path, benchmark_script=rel) + + assert launcher.read_text(encoding="utf-8") == original + + +def test_resolve_native_launcher_rejects_missing_file(tmp_path: Path): + with pytest.raises(FileNotFoundError, match="does not exist"): + resolve_native_launcher( + inferencex_path=tmp_path, + benchmark_script="single_node/agentic/missing.sh", + ) + + +@pytest.mark.parametrize( + "suffix", + ["Inference X", "InferenceX;touch-pwned", "InferenceX$(id)", "InferenceX\nnext"], +) +def test_native_boundaries_reject_shell_unsafe_checkout_paths(tmp_path: Path, suffix: str): + unsafe = tmp_path / suffix + + with pytest.raises(ValueError, match="shell-unsafe"): + resolve_native_launcher( + inferencex_path=unsafe, + benchmark_script="single_node/agentic/run.sh", + ) + with pytest.raises(ValueError, match="shell-unsafe"): + native_agentx.preview_native_recipe( + {"framework": "sglang", "agentx": {"enabled": True}}, + inferencex_path=unsafe, + ) + + +def test_native_launch_environment_identity_binds_effective_audited_controls() -> None: + benchmark = {"envs": {"SPEC_NUM_TOKENS": 2}} + base_env = { + "PATH": "/usr/bin", + "LD_LIBRARY_PATH": "/opt/rocm/lib", + "SCHEDULER_RECV_INTERVAL": "30", + "SPEC_NUM_TOKENS": "99", + "HICACHE_RATIO": "1.5", + "PYTHONHOME": "/opt/python-a", + "HF_TOKEN": "secret-a", + "HF_HOME": "/cache/a", + "LD_PRELOAD": "/tmp/blocked-a.so", + "AIPERF_BIN": "/tmp/untrusted-aiperf", + } + original = native_agentx.native_launch_environment_identity(benchmark, launch_env=base_env) + + assert { + "HICACHE_RATIO", + "LD_LIBRARY_PATH", + "PATH", + "PYTHONHOME", + "SCHEDULER_RECV_INTERVAL", + "SPEC_NUM_TOKENS", + }.issubset(original["names"]) + assert "HF_TOKEN" not in original["names"] + assert "HF_HOME" not in original["names"] + assert "LD_PRELOAD" not in original["names"] + assert "AIPERF_BIN" not in original["names"] + + for name, changed_value in ( + ("SCHEDULER_RECV_INTERVAL", "31"), + ("LD_LIBRARY_PATH", "/opt/rocm/other-lib"), + ("HICACHE_RATIO", "0.5"), + ("PYTHONHOME", "/opt/python-b"), + ): + changed_env = {**base_env, name: changed_value} + changed = native_agentx.native_launch_environment_identity(benchmark, launch_env=changed_env) + assert changed["sha256"] != original["sha256"] + + # Magpie's benchmark.envs overlay is authoritative, and generic child-env + # scrubbing happens before identity selection. + assert ( + native_agentx.native_launch_environment_identity( + benchmark, + launch_env={**base_env, "SPEC_NUM_TOKENS": "100"}, + ) + == original + ) + assert ( + native_agentx.native_launch_environment_identity( + benchmark, + launch_env={**base_env, "HF_TOKEN": "secret-b", "LD_PRELOAD": "/tmp/blocked-b.so"}, + ) + == original + ) + + +def test_native_launch_environment_scrubs_exported_bash_functions() -> None: + """An inherited function must not override commands in Magpie's ``bash -c`` launcher.""" + from hyperloom.common.env_safety import scrub_benchmark_process_env + + function_key = "BASH_FUNC_hyperloom_agentx_probe%%" + clean_env = {"PATH": "/usr/bin:/bin"} + poisoned_env = { + **clean_env, + function_key: "() { printf 'injected\\n'; }", + } + + assert native_agentx.native_launch_environment_identity({}, launch_env=poisoned_env) == ( + native_agentx.native_launch_environment_identity({}, launch_env=clean_env) + ) + + child_env = scrub_benchmark_process_env(dict(poisoned_env)) + native_agentx.scrub_native_agentx_ambient_env(child_env) + assert function_key not in child_env + + probe = subprocess.run( + ["/bin/bash", "-c", "type hyperloom_agentx_probe"], + env=child_env, + capture_output=True, + text=True, + check=False, + ) + assert probe.returncode != 0 + assert "function" not in f"{probe.stdout}\n{probe.stderr}".lower() + + +@pytest.mark.parametrize( + "name", + [ + "SCHEDULER_RECV_INTERVAL", + "SPEC_NUM_TOKENS", + "LD_LIBRARY_PATH", + "GPU_MEM_UTIL", + "MAX_NUM_SEQS", + "MAX_CUDAGRAPH_CAPTURE_SIZE", + "SYNTHETIC_ACCEPT_LEN", + "L3_PER_RANK_GB", + "HICACHE_RATIO", + "HICACHE_WRITE_POLICY", + "HICACHE_IO_BACKEND", + "HICACHE_MEM_LAYOUT", + ], +) +def test_audited_launcher_ambient_knobs_are_execution_controls(name: str) -> None: + baseline = native_agentx.native_launch_environment_identity({}, launch_env={}) + changed = native_agentx.native_launch_environment_identity({}, launch_env={name: "test-value"}) + + assert name in changed["names"] + assert changed["sha256"] != baseline["sha256"] + + +def _make_inferencex_checkout(tmp_path: Path) -> tuple[Path, str, str]: + aiperf_source = tmp_path / "aiperf-source" + _init_git_repo(aiperf_source) + (aiperf_source / "pyproject.toml").write_text( + "[project]\nname = 'aiperf'\nversion = '0'\n", + encoding="utf-8", + ) + _commit_all(aiperf_source, "add package") + + checkout = tmp_path / "InferenceX" + _init_git_repo(checkout) + launcher_rel = "single_node/agentic/run.sh" + launcher = checkout / "benchmarks" / launcher_rel + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/sh\n", encoding="utf-8") + (checkout / "benchmarks" / "benchmark_lib.sh").write_text("#!/bin/sh\n", encoding="utf-8") + configs = checkout / "configs" + configs.mkdir() + (configs / "amd-master.yaml").write_text("models: []\n", encoding="utf-8") + (configs / "runners.yaml").write_text("runners: {}\n", encoding="utf-8") + subprocess.run( + [ + "git", + "-C", + str(checkout), + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "-q", + str(aiperf_source), + "utils/aiperf", + ], + check=True, + ) + head = _commit_all(checkout, "add native inputs") + return checkout, head, launcher_rel + + +def test_native_execution_identity_binds_and_rejects_runners_yaml( + tmp_path: Path, + monkeypatch, +): + checkout, head, launcher_rel = _make_inferencex_checkout(tmp_path) + launcher_sha256 = native_agentx._sha256_file(checkout / "benchmarks" / launcher_rel) + monkeypatch.setattr( + native_agentx, + "_NATIVE_LAUNCHER_MANIFEST", + {(head, "fixture-recipe"): (launcher_rel, launcher_sha256)}, + ) + magpie_commit = "a" * 40 + magpie_execution = { + "fingerprint": "b" * 64, + "source_commit": magpie_commit, + } + resolved_benchmark = {"framework": "sglang", "envs": {"TP": 1}} + + identity = native_agentx.native_execution_identity( + inferencex_path=checkout, + benchmark_script=launcher_rel, + config_file="configs/amd-master.yaml", + resolved_benchmark=resolved_benchmark, + expected_ref=head, + magpie_execution=magpie_execution, + expected_magpie_ref=magpie_commit, + launch_env={"SCHEDULER_RECV_INTERVAL": "30", "LD_LIBRARY_PATH": "/opt/rocm/lib"}, + ) + changed_ambient = native_agentx.native_execution_identity( + inferencex_path=checkout, + benchmark_script=launcher_rel, + config_file="configs/amd-master.yaml", + resolved_benchmark=resolved_benchmark, + expected_ref=head, + magpie_execution=magpie_execution, + expected_magpie_ref=magpie_commit, + launch_env={"SCHEDULER_RECV_INTERVAL": "31", "LD_LIBRARY_PATH": "/opt/rocm/lib"}, + ) + + runners = checkout / "configs" / "runners.yaml" + assert identity["runners_config_sha256"] == native_agentx._sha256_file(runners) + assert identity["static_execution_fingerprint"] == changed_ambient["static_execution_fingerprint"] + assert identity["launch_environment"] != changed_ambient["launch_environment"] + assert identity["execution_fingerprint"] != changed_ambient["execution_fingerprint"] + runners.write_text("runners:\n tampered: true\n", encoding="utf-8") + with pytest.raises(ValueError, match="clean pinned launcher/replay inputs"): + native_agentx.native_execution_identity( + inferencex_path=checkout, + benchmark_script=launcher_rel, + config_file="configs/amd-master.yaml", + resolved_benchmark=resolved_benchmark, + expected_ref=head, + magpie_execution=magpie_execution, + expected_magpie_ref=magpie_commit, + ) + + +@pytest.mark.parametrize("layout", [".", "inferencex-e2e"]) +def test_recipe_resolution_uses_benchmark_interpreter_and_persists_project_root(tmp_path, monkeypatch, layout): + from hyperloom.orchestrator.actions.executors import benchmark_backend + + project = tmp_path / layout + (project / "benchmarks").mkdir(parents=True) + (project / "benchmarks/srt_agentic.sh").write_text("#!/bin/bash\n") + (project / "benchmarks/benchmark_lib.sh").write_text("#!/bin/bash\n") + seen = {} + monkeypatch.setattr( + benchmark_backend, + "resolve_benchmark_interpreter", + lambda: "/opt/benchmark/bin/python", + ) + + def _run(cmd, **kwargs): + seen["cmd"] = cmd + seen["payload"] = json.loads(kwargs["input"]) + result = { + "benchmark": {"framework": "sglang", "envs": {}}, + "recipe": "recipe-a", + "config_file": "configs/amd-master.yaml", + "entry": {"tp": 1}, + } + return subprocess.CompletedProcess( + cmd, + 0, + stdout=native_agentx._RESOLVER_SENTINEL + json.dumps(result) + "\n", + stderr="", + ) + + monkeypatch.setattr(native_agentx.subprocess, "run", _run) + result = native_agentx._run_magpie_recipe_resolver( + {"framework": "sglang", "inferencex_path": str(tmp_path)}, + inferencex_path=tmp_path, + ) + + assert seen["cmd"][0] == "/opt/benchmark/bin/python" + assert seen["payload"]["inferencex_path"] == str(project) + assert seen["payload"]["benchmark"]["inferencex_path"] == str(project) + assert result["recipe"] == "recipe-a" + + +def _install_fake_magpie(monkeypatch, *, entry: dict): + def _resolve(benchmark, *, inferencex_path): + assert inferencex_path.is_dir() + result = dict(benchmark) + envs = dict(result.get("envs") or {}) + envs.update( + { + "TP": entry["tp"], + "PP_SIZE": entry.get("pp", 1), + "PCP_SIZE": entry.get("pcp-size", 1), + "RECIPE_FINGERPRINT": entry["recipe-fingerprint"], + } + ) + result["envs"] = envs + result["docker_image"] = entry["image"] + result.setdefault("gpu_selection", {"auto": True}) + return { + "benchmark": result, + "recipe": "recipe-a", + "config_file": "configs/amd-master.yaml", + "entry": dict(entry), + } + + monkeypatch.setattr(native_agentx, "_run_magpie_recipe_resolver", _resolve) + monkeypatch.setattr( + native_agentx, + "validate_native_recipe_launcher", + lambda **_kwargs: { + "inferencex_commit": "1" * 40, + "recipe": "recipe-a", + "launcher": "single_node/agentic/run.sh", + "launcher_sha256": "2" * 64, + }, + ) + monkeypatch.setattr( + native_agentx, + "native_execution_identity", + lambda **_kwargs: { + "static_execution_fingerprint": "3" * 64, + "execution_fingerprint": "3" * 64, + "inferencex_commit": "1" * 40, + "magpie_commit": "4" * 40, + }, + ) + + +def _recipe_entry(**overrides): + value = { + "tp": 4, + "pp": 1, + "pcp-size": 1, + "ep": 4, + "conc": 8, + "duration": 3600, + "image": "example/sglang:agentx", + "runner": "cluster:mi355x-amds", + "recipe-fingerprint": "a" * 64, + } + value.update(overrides) + return value + + +def test_resolve_native_recipe_persists_topology_and_provenance(tmp_path, monkeypatch): + entry = _recipe_entry() + _install_fake_magpie(monkeypatch, entry=entry) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "0,1,2,3"}, + "gpu_selection": {"auto": True}, + "workload_spec": {"kind": "agentx_trace_replay"}, + } + + topology = resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image=entry["image"], + ) + + assert topology == { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "ep": 4, + "gpu_count": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": "a" * 64, + } + assert benchmark["envs"]["RECIPE_FINGERPRINT"] == "a" * 64 + assert benchmark["workload_spec"]["recipe"]["name"] == "recipe-a" + assert benchmark["workload_spec"]["outer_image_config_pinned"] is True + + +def test_resolve_native_recipe_rejects_recipe_fingerprint_drift( + tmp_path, + monkeypatch, +): + _install_fake_magpie(monkeypatch, entry=_recipe_entry()) + monkeypatch.setenv( + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "b" * 64, + ) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "0,1,2,3"}, + } + + with pytest.raises(ValueError, match="recipe changed after session finalization"): + resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image="example/sglang:agentx", + ) + + +def test_resolve_native_recipe_rejects_outer_topology_mismatch(tmp_path, monkeypatch): + _install_fake_magpie(monkeypatch, entry=_recipe_entry(tp=8)) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "0,1,2,3"}, + } + + with pytest.raises(ValueError, match="--tp"): + resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image="example/sglang:agentx", + ) + + +@pytest.mark.parametrize("outer_image", [None, "wrong/image:tag"]) +def test_resolve_native_recipe_requires_matching_outer_image( + tmp_path, + monkeypatch, + outer_image, +): + _install_fake_magpie(monkeypatch, entry=_recipe_entry()) + monkeypatch.delenv("HYPERLOOM_IMAGE", raising=False) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "0,1,2,3"}, + } + + with pytest.raises(ValueError, match="outer image"): + resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image=outer_image, + ) + + +def test_resolve_native_recipe_honors_exact_manual_gpu_mask(tmp_path, monkeypatch): + _install_fake_magpie(monkeypatch, entry=_recipe_entry()) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "0,1,2,3"}, + "gpu_selection": {"auto": True}, + } + + resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image="example/sglang:agentx", + ) + + assert benchmark["gpu_selection"]["auto"] is False + + +def test_resolve_native_recipe_rejects_nonzero_physical_mask(tmp_path, monkeypatch): + _install_fake_magpie(monkeypatch, entry=_recipe_entry()) + benchmark = { + "framework": "sglang", + "model": "amd/model", + "run_mode": "local", + "runner_type": "mi355x", + "envs": {"CONC": 8, "ROCR_VISIBLE_DEVICES": "4,5,6,7"}, + } + + with pytest.raises(ValueError, match="must be exactly"): + resolve_native_recipe( + benchmark, + inferencex_path=tmp_path, + expected_gpu_count=4, + outer_image="example/sglang:agentx", + ) + + +def _native_grid_base(tmp_path: Path) -> Path: + base = tmp_path / "native.yaml" + base.write_text( + yaml.safe_dump( + { + "benchmark": { + "framework": "sglang", + "model": "amd/model", + "precision": "fp4", + "run_mode": "local", + "runner_type": "mi355x", + "agentx": { + "enabled": True, + "recipe": "recipe-a", + "resolved": {"tp": 4, "ep": 4, "conc": 8}, + }, + "benchmark_script": "single_node/agentic/run.sh", + "inferencex_path": str(tmp_path), + "envs": { + "AGENTX_MODEL_ID": "amd/model", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/run.sh", + "CONC": 8, + "ROCR_VISIBLE_DEVICES": "0,1,2,3", + "TP": 4, + }, + "workload_spec": { + "kind": "agentx_trace_replay", + "outer_image": "example/sglang:agentx", + "resolved_topology": { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "ep": 4, + "gpu_count": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": "a" * 64, + }, + }, + } + }, + sort_keys=False, + ), + encoding="utf-8", + ) + return base + + +@pytest.mark.parametrize( + ("variant_kwargs", "build_kwargs", "match"), + [ + ({"extra_server_args": "--foo 1"}, {}, "variant.extra_server_args"), + ({"extra_envs": {"USE_AITER": "1"}}, {}, "extra_envs=USE_AITER"), + ({"remove_args": ["--foo"]}, {}, "remove_args"), + ({}, {"base_extra_args": "--bar 2"}, "base_extra_args"), + ], +) +def test_native_grid_variant_rejects_unapplied_server_candidates( + tmp_path, + variant_kwargs, + build_kwargs, + match, +): + from hyperloom.orchestrator.actions.executors._grid_runner import ( + GridVariant, + _build_variant_yaml, + ) + + base = _native_grid_base(tmp_path) + variant = GridVariant(name="candidate", **variant_kwargs) + with pytest.raises(ValueError, match=match): + _build_variant_yaml( + base, + build_kwargs.get("base_extra_args", ""), + variant, + output_subdir=tmp_path / "out", + ) + + +def test_native_grid_rejects_concurrency_change_before_recipe_resolution( + tmp_path, + monkeypatch, +): + from hyperloom.orchestrator.actions.executors._grid_runner import ( + GridVariant, + _build_variant_yaml, + ) + + base = _native_grid_base(tmp_path) + monkeypatch.setattr( + native_agentx, + "resolve_native_recipe", + lambda *_args, **_kwargs: pytest.fail("a fixed-CONC mismatch must fail before recipe resolution"), + ) + + with pytest.raises(ValueError, match="concurrency is fixed"): + _build_variant_yaml( + base, + "", + GridVariant(name="conc16", extra_envs={"CONC": "16"}), + output_subdir=tmp_path / "out", + ) + + +@pytest.mark.parametrize( + "changed", + [ + {"tp": 2, "pp": 2}, + {"recipe_fingerprint": "b" * 64}, + {"ep": 2}, + ], +) +def test_native_grid_rejects_same_size_recipe_arm_changes( + tmp_path, + monkeypatch, + changed, +): + from hyperloom.orchestrator.actions.executors._grid_runner import ( + GridVariant, + _build_variant_yaml, + ) + + topology = { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "ep": 4, + "gpu_count": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": "a" * 64, + } + topology.update(changed) + monkeypatch.setattr( + native_agentx, + "resolve_native_recipe", + lambda *_args, **_kwargs: dict(topology), + ) + + with pytest.raises(ValueError, match="topology-changing rounds"): + _build_variant_yaml( + _native_grid_base(tmp_path), + "", + GridVariant(name="same-conc"), + output_subdir=tmp_path / "out", + ) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_optimization_identity.py b/src/hyperloom/inference_optimizer/tests/test_agentx_optimization_identity.py new file mode 100644 index 0000000000..74f3051e2d --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_optimization_identity.py @@ -0,0 +1,282 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native candidates must prove their launch without changing the replay workload.""" + +from __future__ import annotations + +import json +from copy import deepcopy +from types import SimpleNamespace + +import pytest + +from hyperloom.common.perf_metric import graded_axes_of, perf_snapshot_from_mapping, rounds_are_comparable +from hyperloom.inference_optimizer.agentx.identity import canonical_sha256, native_workload_fingerprint +from hyperloom.inference_optimizer.agentx.identity import validate_server_launch +from hyperloom.inference_optimizer.tests.test_benchmark_result import ( + _native_agentx_report, + _write_native_config_snapshot, +) +from hyperloom.orchestrator.actions.executors.benchmark_result import extract_benchmark_measurement +from hyperloom.orchestrator.state.shared_state import resolve_graded_comparison + + +def _write_candidate_config(workspace, config): + (workspace / "hyperloom-input.yaml").write_text(json.dumps({"benchmark": config})) + (workspace / "config.yaml").write_text( + json.dumps({key: value for key, value in config.items() if key != "workload_spec"}) + ) + + +def _candidate_artifacts(workspace, *, env=None): + workspace.mkdir(parents=True, exist_ok=True) + _write_native_config_snapshot(workspace) + config = json.loads((workspace / "config.yaml").read_text()) + overrides = {"version": 1, "env": env or {}} + config["agentx"]["launch_overrides"] = overrides + static = "b" * 64 + config["workload_spec"]["execution"] = { + "static_execution_fingerprint": static, + "workload_fingerprint": native_workload_fingerprint(config, static), + } + _write_candidate_config(workspace, config) + launch = { + "version": 1, + "framework": "sglang", + "base_argv": ["python3", "-m", "sglang.launch_server"], + "effective_argv": ["python3", "-m", "sglang.launch_server"], + "base_env": dict.fromkeys(env or {}), + "effective_env": env or {}, + "source_files": {}, + "resolved_executable": "/usr/bin/python3", + "runtime_environment": env or {}, + "overrides_sha256": canonical_sha256(overrides), + } + launch["evidence_sha256"] = canonical_sha256(launch) + (workspace / "agentx_server_launch.json").write_text(json.dumps(launch)) + report = _native_agentx_report() + report["agentx_metrics"]["server_launch"] = launch + return config, report + + +def test_native_server_candidates_retain_comparable_workload_and_distinct_provenance(tmp_path): + baseline_dir, candidate_dir = tmp_path / "baseline", tmp_path / "candidate" + _, baseline_report = _candidate_artifacts(baseline_dir) + _, candidate_report = _candidate_artifacts(candidate_dir, env={"SGLANG_USE_AITER": "1"}) + baseline = extract_benchmark_measurement( + baseline_report, workspace=baseline_dir, materialized_config_path=baseline_dir / "hyperloom-input.yaml" + ) + candidate = extract_benchmark_measurement( + candidate_report, workspace=candidate_dir, materialized_config_path=candidate_dir / "hyperloom-input.yaml" + ) + assert baseline["submission_valid"] is True + assert candidate["submission_valid"] is True + assert candidate["agentx_workload_fingerprint"] == baseline["agentx_workload_fingerprint"] + assert candidate["agentx_candidate_fingerprint"] != baseline["agentx_candidate_fingerprint"] + persisted = graded_axes_of(baseline) + persisted["output_throughput"] = baseline["output_throughput"] + assert rounds_are_comparable(perf_snapshot_from_mapping(candidate), perf_snapshot_from_mapping(persisted)) + + +@pytest.mark.parametrize("change", ["model", "corpus", "concurrency", "dependencies", "environment"]) +def test_native_workload_identity_changes_when_fixed_inputs_change(tmp_path, change): + config, _ = _candidate_artifacts(tmp_path) + altered = deepcopy(config) + static = "b" * 64 + if change == "model": + altered["model"] = "different/model" + elif change == "corpus": + altered["agentx"]["recipe"] = "different-replay" + elif change == "concurrency": + altered["agentx"]["resolved"]["conc"] = 64 + elif change == "dependencies": + static = "c" * 64 + else: + altered["envs"] = {"MODEL_PATH": "/other/checkpoint"} + assert native_workload_fingerprint(altered, static) != native_workload_fingerprint(config, "b" * 64) + + +@pytest.mark.parametrize( + ("change", "error"), + [ + ("missing_report", "server_launch_missing"), + ("missing_artifact", "server_launch_artifact_missing"), + ("candidate", "server_launch_candidate_mismatch"), + ("env", "server_launch_environment_mismatch"), + ("source", "server_launch_source_mismatch"), + ("workload", "workload_fingerprint_invalid"), + ], +) +def test_native_candidate_without_matching_launch_evidence_cannot_be_promoted(tmp_path, change, error): + config, report = _candidate_artifacts(tmp_path, env={"SGLANG_USE_AITER": "1"}) + evidence = report["agentx_metrics"]["server_launch"] + if change == "missing_report": + report["agentx_metrics"].pop("server_launch") + elif change == "missing_artifact": + (tmp_path / "agentx_server_launch.json").unlink() + elif change == "candidate": + evidence["overrides_sha256"] = "c" * 64 + elif change == "env": + evidence["effective_env"] = {} + elif change == "source": + config["agentx"]["launch_overrides"]["source_files"] = {"/framework/kernel.py": "c" * 64} + else: + config["workload_spec"]["execution"]["workload_fingerprint"] = "c" * 64 + _write_candidate_config(tmp_path, config) + measurement = extract_benchmark_measurement( + report, workspace=tmp_path, materialized_config_path=tmp_path / "hyperloom-input.yaml" + ) + assert measurement["submission_valid"] is False + assert error in measurement["native_agentx_protocol_errors"] + assert "agentx_workload_fingerprint" not in measurement + + +def test_epoch_three_grading_rejects_missing_or_different_workload_identity(tmp_path, monkeypatch): + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _, report = _candidate_artifacts(tmp_path) + baseline = extract_benchmark_measurement( + report, workspace=tmp_path, materialized_config_path=tmp_path / "hyperloom-input.yaml" + ) + state = SimpleNamespace( + benchmark_mode="agentx", agentx_epoch=3, framework="sglang", baseline_perf=baseline, current_best={} + ) + candidate = {**baseline, "e2e_norm_intvty_p50": baseline["e2e_norm_intvty_p50"] * 1.1} + assert resolve_graded_comparison(state, candidate).verdict == "KEEP" + candidate["agentx_workload_fingerprint"] = "c" * 64 + assert resolve_graded_comparison(state, candidate).verdict == "REVERT" + candidate.pop("agentx_workload_fingerprint") + result = resolve_graded_comparison(state, candidate) + assert result.verdict == "REVERT" + assert result.degrade_reason == "native_workload_identity_missing" + + +def test_custom_native_result_binds_the_real_model_context_and_config(tmp_path): + config, report = _candidate_artifacts(tmp_path) + declared = {"native_context_length": 32768, "max_model_len": 16384, "model_config_sha256": "d" * 64} + resolved = config["agentx"]["resolved"] + resolved.update({"custom": True, **{key.replace("_", "-"): value for key, value in declared.items()}}) + config["workload_spec"]["execution"]["workload_fingerprint"] = native_workload_fingerprint(config, "b" * 64) + _write_candidate_config(tmp_path, config) + report["agentx_metrics"]["recipe"].update(custom_recipe=True, **declared) + raw_path = tmp_path / "inferencex_result.json" + raw = json.loads(raw_path.read_text()) + raw.update(custom_recipe=True, **declared) + raw_path.write_text(json.dumps(raw)) + measurement = extract_benchmark_measurement( + report, workspace=tmp_path, materialized_config_path=tmp_path / "hyperloom-input.yaml" + ) + assert measurement["submission_valid"] is True + + raw["max_model_len"] = 8192 + raw_path.write_text(json.dumps(raw)) + measurement = extract_benchmark_measurement( + report, workspace=tmp_path, materialized_config_path=tmp_path / "hyperloom-input.yaml" + ) + assert measurement["submission_valid"] is False + assert "max_model_len_mismatch" in measurement["native_agentx_protocol_errors"] + + +@pytest.mark.parametrize("change", ["missing", "candidate", "model", "drop_contract"]) +def test_native_snapshot_must_match_the_submitted_materialized_input(tmp_path, change): + config, report = _candidate_artifacts(tmp_path) + expected_path = tmp_path / "hyperloom-input.yaml" + if change == "missing": + expected_path = None + else: + snapshot = json.loads((tmp_path / "config.yaml").read_text()) + if change == "candidate": + snapshot["agentx"]["launch_overrides"]["env"]["SGLANG_USE_AITER"] = "1" + elif change == "model": + snapshot["model"] = "other/model" + else: + snapshot["agentx"].pop("launch_overrides") + (tmp_path / "config.yaml").write_text(json.dumps(snapshot)) + measurement = extract_benchmark_measurement(report, workspace=tmp_path, materialized_config_path=expected_path) + assert measurement["submission_valid"] is False + assert "agentx_workload_fingerprint" not in measurement + expected_error = "materialized_config_missing" if change == "missing" else "materialized_config_snapshot_mismatch" + assert expected_error in measurement["native_agentx_protocol_errors"] + + +@pytest.mark.parametrize("control", ["append", "remove", "replace", "unrequested"]) +def test_self_consistent_receipt_cannot_hide_ignored_argument_changes(tmp_path, control): + config, report = _candidate_artifacts(tmp_path) + overrides = config["agentx"]["launch_overrides"] + evidence = report["agentx_metrics"]["server_launch"] + evidence["base_argv"] += ["--model-path", "test/model", "--disable-cuda-graph"] + evidence["effective_argv"] = list(evidence["base_argv"]) + if control == "append": + overrides["append_args"] = ["--json-model-override-args", '{"key": "space value"}'] + elif control == "remove": + overrides["remove_args"] = ["--disable-cuda-graph"] + elif control == "replace": + overrides["replace_args"] = True + else: + evidence["effective_argv"].append("--unexpected-change") + evidence["overrides_sha256"] = canonical_sha256(overrides) + evidence["evidence_sha256"] = canonical_sha256( + {key: value for key, value in evidence.items() if key != "evidence_sha256"} + ) + assert validate_server_launch(config, evidence) == ["server_launch_arguments_mismatch"] + + +@pytest.mark.parametrize("change", ["missing_identity", "wrong_workload", "slower_interactivity"]) +def test_warm_native_replay_rejects_before_promoting_source_or_kernel(tmp_path, monkeypatch, change): + from hyperloom.inference_optimizer.tests.test_warm_replay import _make_coord, _replay_task + + _, report = _candidate_artifacts(tmp_path) + baseline = extract_benchmark_measurement( + report, workspace=tmp_path, materialized_config_path=tmp_path / "hyperloom-input.yaml" + ) + coord = _make_coord(tmp_path) + state = coord.shared_state + state.benchmark_mode = "agentx" + state.agentx_backend = "magpie" + state.agentx_epoch = 3 + state.baseline_perf = baseline + state.baseline_tput = baseline["output_throughput"] + result = {**baseline, "status": "succeeded", "output_throughput": state.baseline_tput * 1.2} + if change == "missing_identity": + result.pop("agentx_workload_fingerprint") + elif change == "wrong_workload": + result["agentx_workload_fingerprint"] = "d" * 64 + else: + result["e2e_norm_intvty_p50"] *= 0.9 + + def unexpected(*args, **kwargs): + pytest.fail("invalid native replay reached source/kernel promotion") + + monkeypatch.setattr(coord.phase_prelude, "_resolve_promoted_recipe_checkout", unexpected) + monkeypatch.setattr(coord.phase_prelude, "_book_combined_kernel_keep", unexpected) + coord.phase_prelude._settle_warm_replay(result, task=_replay_task(), recorder=None) + assert state.warm_replay_outcome["status"] == "drift" + assert not state.optimization_stack + assert not state.current_best + + +def test_warm_native_replay_keeps_interactivity_winner_and_its_launch_identity(tmp_path): + from hyperloom.inference_optimizer.tests.test_warm_replay import _make_coord, _replay_task + + _, report = _candidate_artifacts(tmp_path) + input_path = tmp_path / "hyperloom-input.yaml" + baseline = extract_benchmark_measurement(report, workspace=tmp_path, materialized_config_path=input_path) + coord = _make_coord(tmp_path) + state = coord.shared_state + state.benchmark_mode = "agentx" + state.agentx_backend = "magpie" + state.agentx_epoch = 3 + state.baseline_perf = baseline + state.baseline_tput = baseline["output_throughput"] + result = { + **baseline, + "status": "succeeded", + "materialized_config": str(input_path), + "e2e_norm_intvty_p50": baseline["e2e_norm_intvty_p50"] * 1.1, + } + coord.phase_prelude._settle_warm_replay(result, task=_replay_task(), recorder=None) + assert state.warm_replay_outcome["status"] == "reproduced" + assert state.optimization_stack + assert state.current_best["materialized_config"] == str(input_path) + assert state.current_best["agentx_server_launch"] == baseline["agentx_server_launch"] + assert state.current_best["agentx_workload_fingerprint"] == baseline["agentx_workload_fingerprint"] diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_repair.py b/src/hyperloom/inference_optimizer/tests/test_agentx_repair.py index f8b351712c..cba973f24a 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_repair.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_repair.py @@ -677,7 +677,7 @@ def run( uv_path.unlink(missing_ok=True) text = repair.install_script_path().read_text(encoding="utf-8") pip_gate = text[text.index("PIP_EXTRA=()") : text.index("# --- 1. inference_optimizer")] - block = text[text.index("# --- 2a. aiperf") : text.index("# --- 2b. Atomic-write")] + block = text[text.index("# --- 2a. aiperf") : text.index("# --- 3. InferenceX checkout:")] polluted_vars = ( "PYTHONHOME PYTHONPATH PYTHONUSERBASE PYTHONPLATLIBDIR __PYVENV_LAUNCHER__ " "PIP_TARGET PIP_PREFIX PIP_ROOT PIP_USER UV_SYSTEM_PYTHON UV_PYTHON " diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_runtime.py b/src/hyperloom/inference_optimizer/tests/test_agentx_runtime.py index c7731c8fc5..56e5ddb7ee 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_runtime.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_runtime.py @@ -5,6 +5,8 @@ from __future__ import annotations +import subprocess + import pytest import yaml @@ -33,6 +35,176 @@ def _cfg(tmp_path, script): return p +def _native_cfg(tmp_path, *, model_path: str = ""): + launcher = tmp_path / "benchmarks" / "single_node" / "agentic" / "model.sh" + launcher.parent.mkdir(parents=True) + launcher.write_text( + "#!/usr/bin/env bash\nSGLANG_CMD=(\n python3 -m sglang.launch_server\n)\n", + encoding="utf-8", + ) + cfg = tmp_path / "native.yaml" + cfg.write_text( + yaml.safe_dump( + { + "benchmark": { + "framework": "sglang", + "agentx": "enable", + "benchmark_script": "single_node/agentic/model.sh", + "envs": {"MODEL_PATH": model_path} if model_path else {}, + } + } + ), + encoding="utf-8", + ) + return cfg, launcher + + +def _pinned_native_cfg(tmp_path, monkeypatch): + cfg, _launcher = _native_cfg(tmp_path) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + benchmark = data["benchmark"] + benchmark.update( + { + "model": "amd/model", + "precision": "fp4", + "docker_image": "example/sglang:agentx", + "envs": {"CONC": 8}, + } + ) + topology = { + "tp": 4, + "pp": 1, + "pcp_size": 1, + "ep": 4, + "gpu_count": 4, + "conc": 8, + "duration_seconds": 3600, + "recipe_fingerprint": "a" * 64, + } + recipe = { + "name": "recipe-a", + "config_file": "configs/amd-master.yaml", + "recipe_fingerprint": "a" * 64, + "image": "example/sglang:agentx", + "runner": "cluster:mi355x-amds", + "model": "amd/model", + "model_prefix": "model", + "framework": "sglang", + "precision": "fp4", + "concurrency": 8, + "duration_seconds": 3600, + "launcher": "single_node/agentic/model.sh", + } + execution = { + "inferencex_commit": "d" * 40, + "magpie_commit": "c" * 40, + "static_execution_fingerprint": "b" * 64, + "execution_fingerprint": "b" * 64, + } + benchmark["workload_spec"] = { + "resolved_topology": dict(topology), + "recipe": dict(recipe), + "execution": dict(execution), + } + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + def _preview(current, *, inferencex_path): + current_conc = int(current.get("envs", {}).get("CONC", 8)) + current_image = str(current.get("docker_image") or recipe["image"]) + current_topology = {**topology, "conc": current_conc} + return { + "benchmark": dict(current), + "entry": { + "image": current_image, + "runner": recipe["runner"], + "model": recipe["model"], + "model-prefix": recipe["model_prefix"], + "framework": recipe["framework"], + "precision": recipe["precision"], + }, + "recipe": recipe["name"], + "config_file": recipe["config_file"], + "topology": current_topology, + "magpie_execution": {"fingerprint": "e" * 64, "source_commit": "c" * 40}, + } + + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.preview_native_recipe", + _preview, + ) + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.native_execution_identity", + lambda **_kwargs: dict(execution), + ) + env = { + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": "a" * 64, + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": "b" * 64, + "HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT": "b" * 64, + "HYPERLOOM_AGENTX_GPU_COUNT": "4", + "MAGPIE_REF": "c" * 40, + "INFERENCEX_REF": "d" * 40, + } + return cfg, env + + +def _pinned_profile_compat_cfg(tmp_path): + output_dir = tmp_path / "profile-output" + output_dir.mkdir() + checkout = output_dir / ".agentx-profile-inferencex" + benchmark_lib = checkout / "benchmarks" / "benchmark_lib.sh" + benchmark_lib.parent.mkdir(parents=True) + benchmark_lib.write_text("#!/bin/sh\n", encoding="utf-8") + subprocess.run(["git", "init", "-q", str(checkout)], check=True) + subprocess.run(["git", "-C", str(checkout), "add", "."], check=True) + subprocess.run( + [ + "git", + "-C", + str(checkout), + "-c", + "user.name=Hyperloom Test", + "-c", + "user.email=hyperloom@example.invalid", + "commit", + "-qm", + "profile fixture", + ], + check=True, + ) + head = subprocess.run( + ["git", "-C", str(checkout), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + config = output_dir / "baseline_config.with_envs.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "benchmark_script": "aiperf_client.sh", + "inferencex_path": str(checkout), + "envs": {"PROFILE": "1"}, + "profiler": {"torch_profiler": {"enabled": True}}, + "workload_spec": { + "kind": "agentx_trace_replay", + "harness": "hyperloom-profiler-compat", + }, + } + } + ), + encoding="utf-8", + ) + env = { + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": "a" * 64, + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": "b" * 64, + "HYPERLOOM_AGENTX_GPU_COUNT": "4", + "MAGPIE_REF": "c" * 40, + "INFERENCEX_REF": head, + } + return config, checkout, env + + def test_mlperf_client_skips_aiperf_preflight(tmp_path, monkeypatch): calls = {"deploy": 0, "preflight": 0, "mlperf": 0} monkeypatch.setattr(_DEPLOY, lambda d: calls.__setitem__("deploy", calls["deploy"] + 1)) @@ -65,6 +237,262 @@ def test_deploy_before_preflight_with_resolved_bin(tmp_path, monkeypatch): assert order == ["deploy", ("preflight", "/venv/bin/aiperf")] +def test_native_agentx_validates_launcher_without_modifying_checkout(tmp_path, monkeypatch): + calls = {"deploy": 0, "preflight": 0} + monkeypatch.setattr(_DEPLOY, lambda d: calls.__setitem__("deploy", calls["deploy"] + 1)) + monkeypatch.setattr(_CHECK, lambda b, **k: calls.__setitem__("preflight", calls["preflight"] + 1)) + cfg, launcher = _native_cfg(tmp_path) + original = launcher.read_bytes() + + assert runtime.maybe_prepare_agentx(env={}, inferencex_path=str(tmp_path), config_path=cfg) is True + + assert calls == {"deploy": 0, "preflight": 0} + assert launcher.read_bytes() == original + + +def test_native_agentx_uses_checkout_pinned_in_materialized_yaml(tmp_path): + cfg, launcher = _native_cfg(tmp_path) + original = launcher.read_bytes() + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + data["benchmark"]["inferencex_path"] = str(tmp_path) + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + assert runtime.maybe_prepare_agentx(env={}, inferencex_path="", config_path=cfg) is True + + assert launcher.read_bytes() == original + + +def test_native_agentx_scrubs_remote_model_id_from_child_environment(tmp_path): + cfg, _launcher = _native_cfg(tmp_path) + child = {"MODEL_PATH": "amd/GLM-5.2-MXFP4", "PATH": "/opt/venv/bin"} + + assert ( + runtime.maybe_prepare_agentx( + env=child, + inferencex_path=str(tmp_path), + config_path=cfg, + ) + is True + ) + + assert "MODEL_PATH" not in child + + +def test_native_agentx_keeps_materialized_local_model_path(tmp_path): + cfg, _launcher = _native_cfg(tmp_path, model_path="/models/glm") + child = {"MODEL_PATH": "amd/GLM-5.2-MXFP4"} + + assert ( + runtime.maybe_prepare_agentx( + env=child, + inferencex_path=str(tmp_path), + config_path=cfg, + ) + is True + ) + + assert child["MODEL_PATH"] == "/models/glm" + + +def test_pinned_native_agentx_revalidates_unchanged_materialization(tmp_path, monkeypatch): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + + assert ( + runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(tmp_path), + config_path=cfg, + ) + is True + ) + + +def test_pinned_native_agentx_rejects_forwarded_env_changed_after_acceptance( + tmp_path, + monkeypatch, +): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + benchmark = data["benchmark"] + benchmark["envs"]["PYTHONPATH"] = "/candidate/a" + saved_execution = benchmark["workload_spec"]["execution"] + saved_execution["launch_config_sha256"] = "f" * 64 + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + def _identity(**kwargs): + current = dict(saved_execution) + if kwargs["resolved_benchmark"]["envs"].get("PYTHONPATH") != "/candidate/a": + current["launch_config_sha256"] = "0" * 64 + current["execution_fingerprint"] = "1" * 64 + return current + + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.native_execution_identity", + _identity, + ) + assert ( + runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(tmp_path), + config_path=cfg, + ) + is True + ) + + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + data["benchmark"]["envs"]["PYTHONPATH"] = "/candidate/b" + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + with pytest.raises(ValueError, match="resolved BenchmarkConfig changed"): + runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(tmp_path), + config_path=cfg, + ) + + +def test_pinned_native_agentx_cannot_be_disabled_in_materialized_yaml(tmp_path, monkeypatch): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + data["benchmark"]["agentx"] = False + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + with pytest.raises(ValueError, match="requires benchmark.agentx"): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + +def test_pinned_agentx_profile_compatibility_path_is_explicitly_allowed( + tmp_path, + monkeypatch, +): + config, checkout, env = _pinned_profile_compat_cfg(tmp_path) + calls: list[object] = [] + monkeypatch.setattr(_DEPLOY, lambda path: calls.append(("deploy", path))) + monkeypatch.setattr(_RESOLVE, lambda _env: "/venv/bin/aiperf") + monkeypatch.setattr( + _CHECK, + lambda binary, **kwargs: calls.append(("preflight", binary, kwargs)), + ) + + assert runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(checkout), + config_path=config, + allow_profile_compat=True, + ) + + assert calls[0] == ("deploy", checkout / "benchmarks") + assert calls[1][0:2] == ("preflight", "/venv/bin/aiperf") + assert calls[1][2]["require_progress_api"] is True + + +def test_pinned_agentx_profile_compatibility_rejects_shell_unsafe_checkout_path( + tmp_path, +): + unsafe_root = tmp_path / "profile root;unsafe" + unsafe_root.mkdir() + config, checkout, env = _pinned_profile_compat_cfg(unsafe_root) + + with pytest.raises(ValueError, match="shell-unsafe"): + runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(checkout), + config_path=config, + allow_profile_compat=True, + ) + + +def test_pinned_agentx_generic_client_still_fails_closed(tmp_path): + config, checkout, env = _pinned_profile_compat_cfg(tmp_path) + data = yaml.safe_load(config.read_text(encoding="utf-8")) + data["benchmark"]["workload_spec"].pop("harness") + config.write_text(yaml.safe_dump(data), encoding="utf-8") + + with pytest.raises(ValueError, match="diagnostic harness identity"): + runtime.maybe_prepare_agentx( + env=env, + inferencex_path=str(checkout), + config_path=config, + allow_profile_compat=True, + ) + + +def test_pinned_native_agentx_requires_session_owned_execution_identity(tmp_path, monkeypatch): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + data["benchmark"]["workload_spec"].pop("execution") + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + with pytest.raises(ValueError, match="missing workload_spec.execution"): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + +def test_native_optimization_accepts_new_candidate_on_the_persisted_workload(tmp_path, monkeypatch): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + bench = data["benchmark"] + bench["agentx"] = { + "enabled": True, + "launch_overrides": {"version": 1, "append_args": ["--chunked-prefill-size", "8192"]}, + } + execution = bench["workload_spec"]["execution"] + execution["execution_fingerprint"] = "f" * 64 + execution["workload_fingerprint"] = "e" * 64 + env["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] = "e" * 64 + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.native_execution_identity", lambda **_kwargs: dict(execution) + ) + + assert runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) is True + + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.native_execution_identity", + lambda **_kwargs: {**execution, "execution_fingerprint": "d" * 64}, + ) + with pytest.raises(ValueError, match="persisted execution identity changed"): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.native_execution_identity", + lambda **_kwargs: {**execution, "workload_fingerprint": "d" * 64}, + ) + with pytest.raises(ValueError, match="BenchmarkConfig changed"): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + +def test_native_optimization_cannot_drop_the_launch_contract(tmp_path, monkeypatch): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + env["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] = "e" * 64 + with pytest.raises(ValueError, match="missing its launch contract"): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + +@pytest.mark.parametrize( + ("field", "value", "match"), + [ + ("CONC", 16, "resolved topology changed"), + ("docker_image", "attacker/image:latest", "recipe identity changed"), + ], +) +def test_pinned_native_agentx_rejects_recipe_inputs_changed_after_materialization( + tmp_path, + monkeypatch, + field, + value, + match, +): + cfg, env = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(cfg.read_text(encoding="utf-8")) + if field == "CONC": + data["benchmark"]["envs"][field] = value + else: + data["benchmark"][field] = value + cfg.write_text(yaml.safe_dump(data), encoding="utf-8") + + with pytest.raises(ValueError, match=match): + runtime.maybe_prepare_agentx(env=env, inferencex_path=str(tmp_path), config_path=cfg) + + def test_preflight_memoized_per_bin(tmp_path, monkeypatch): n = {"p": 0} monkeypatch.setattr(_DEPLOY, lambda d: None) @@ -197,6 +625,15 @@ def test_prepare_runtime_uses_materialized_agentx_script_without_env(tmp_path, m assert order == ["deploy", "preflight"] +def test_prepare_runtime_detects_native_agentx_without_ambient_env(tmp_path, monkeypatch): + monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False) + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + cfg, launcher = _native_cfg(tmp_path) + original = launcher.read_bytes() + assert prepare_agentx_runtime(env={}, inferencex_path=str(tmp_path), config_path=cfg) is None + assert launcher.read_bytes() == original + + def test_prepare_runtime_on_deploys_returns_none(tmp_path, monkeypatch): """ON deploys the client + preflights, returning None on success.""" monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False) diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_switch.py b/src/hyperloom/inference_optimizer/tests/test_agentx_switch.py index 34b03d801b..6772192782 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_switch.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_switch.py @@ -13,10 +13,16 @@ import yaml from hyperloom.common.env import EnvValueError +from hyperloom.inference_optimizer.agentx import native as native_agentx from hyperloom.orchestrator.actions.executors import _workload_envs as we _AGENTX_ENV_KEYS = ( "HYPERLOOM_AGENTX", + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT", + "HYPERLOOM_AGENTX_GPU_COUNT", + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_MODE", "HYPERLOOM_AGENTIC_BACKEND", "AGENTX_DATASET", "AGENTX_MAX_CTX", @@ -28,6 +34,7 @@ "WEKA_LOADER_OVERRIDE", "RUN_EVAL", "MODEL_PATH", + "TARGET_GPU_TYPE", "PORT", "MLPERF_AGENTIC_FLOW", "AGENTIC_NUM_TRAJECTORIES", @@ -35,12 +42,54 @@ "MLPERF_TOKENIZER_DIR", ) +_MODEL_ID = "acme/Test-Model" +_VLLM_LAUNCHER = "single_node/agentic/test_fp4_mi300x_vllm_mtp.sh" + + +@pytest.fixture(autouse=True) +def _resolved_recipe_stub(tmp_path, monkeypatch): + """Materialization tests isolate the switch from Magpie/InferenceX I/O.""" + monkeypatch.setenv("HYPERLOOM_IMAGE", "test/agentx:image") + + def _resolve(benchmark, *, expected_gpu_count, **_kwargs): + envs = benchmark.setdefault("envs", {}) + # Magpie persists the recipe's inner tensor parallelism. ``setdefault`` + # lets PP/PCP tests supply an inner TP smaller than the physical count. + envs.setdefault("TP", int(expected_gpu_count)) + expected_mask = ",".join(str(index) for index in range(int(expected_gpu_count))) + if str(envs.get("ROCR_VISIBLE_DEVICES") or "") != expected_mask: + raise ValueError(f"ROCR_VISIBLE_DEVICES must be exactly {expected_mask!r}") + benchmark.setdefault("gpu_selection", {})["auto"] = False + workload = benchmark.setdefault("workload_spec", {}) + workload["resolved_topology"] = { + "tp": int(expected_gpu_count), + "pp": 1, + "pcp_size": 1, + "ep": 1, + "gpu_count": int(expected_gpu_count), + "recipe_fingerprint": "a" * 64, + } + workload["recipe"] = { + "name": "test-agentx-recipe", + "recipe_fingerprint": "a" * 64, + "image": "test/agentx:image", + } + return workload["resolved_topology"] + + monkeypatch.setattr(native_agentx, "resolve_native_recipe", _resolve) + def _clear_env(monkeypatch): for k in _AGENTX_ENV_KEYS: monkeypatch.delenv(k, raising=False) +def _enable_native(monkeypatch, *, launcher: str = _VLLM_LAUNCHER) -> None: + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", _MODEL_ID) + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", launcher) + + def _write(path, **bench_extra): bench = {"framework": "vllm", "model": "/m", "envs": {}} bench.update(bench_extra) @@ -48,7 +97,18 @@ def _write(path, **bench_extra): return path +def _write_native(path, **bench_extra): + return _write(path, **{"model": "", "agentx": "enable", **bench_extra}) + + def _materialize(src, out, **kw): + source = yaml.safe_load(src.read_text(encoding="utf-8")) or {} + source_native = we._serialized_native_agentx_enabled((source.get("benchmark") or {}).get("agentx")) + env_native = os.environ.get("HYPERLOOM_AGENTX", "").strip().lower() in {"1", "true", "yes", "on"} + if source_native or env_native or kw.get("agentx_mode") is True: + inferencex = src.parent / "ix" + inferencex.mkdir(exist_ok=True) + kw.setdefault("inferencex_path", str(inferencex)) res = we.materialize_config_with_envs(src, out, **kw) return yaml.safe_load(res.read_text())["benchmark"] @@ -76,18 +136,55 @@ def test_switch_off_no_agentx_leakage(tmp_path, monkeypatch): # ── ON path: authoritative overwrite + env injection ───────────────────────── def test_switch_on_authoritative_overwrite(tmp_path, monkeypatch): _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") - src = _write(tmp_path / "base.yaml") + _enable_native(monkeypatch) + src = _write_native(tmp_path / "base.yaml") # gpu_type pre-pins vllm_mi300x.sh; the switch must overwrite it. bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == _VLLM_LAUNCHER + assert bench["agentx"] == "enable" + assert bench["model"] == _MODEL_ID + assert bench["gpu_selection"]["auto"] is False + assert "count" not in bench["gpu_selection"] + assert bench["envs"]["MODEL_PATH"] == "/m" assert "timeout_seconds" not in bench assert "AGENTX_PHASE_WAIT_TIMEOUT_S" not in bench["envs"] +def test_native_outer_gpu_count_does_not_overwrite_inner_recipe_tp( + tmp_path, + monkeypatch, +): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + monkeypatch.setenv("TP", "4") + monkeypatch.setenv("HYPERLOOM_AGENTX_GPU_COUNT", "4") + monkeypatch.setenv("ROCR_VISIBLE_DEVICES", "0,1,2,3") + src = _write_native( + tmp_path / "base.yaml", + agentx="enable", + envs={ + "TP": 2, + "CONC": 8, + "ROCR_VISIBLE_DEVICES": "0,1,2,3", + }, + ) + + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi300x", + model_path="/m", + ) + + assert bench["envs"]["TP"] == 2 + assert bench["workload_spec"]["resolved_topology"]["gpu_count"] == 4 + + def test_persisted_agentx_mode_switches_without_ambient_env(tmp_path, monkeypatch): _clear_env(monkeypatch) - src = _write(tmp_path / "base.yaml") + monkeypatch.setenv("AGENTX_MODEL_ID", _MODEL_ID) + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", _VLLM_LAUNCHER) + src = _write_native(tmp_path / "base.yaml") bench = _materialize( src, tmp_path / "out", @@ -95,45 +192,126 @@ def test_persisted_agentx_mode_switches_without_ambient_env(tmp_path, monkeypatc model_path="/m", agentx_mode=True, ) - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == _VLLM_LAUNCHER + assert bench["agentx"] == "enable" + + +def test_native_magpie_yaml_round_trips_without_duplicate_agentx_envs(tmp_path, monkeypatch): + _clear_env(monkeypatch) + native_config = { + "enabled": True, + "mode": "canonical", + "recipe": "test-agentx-recipe", + "selector": {"tp": 8}, + "failed_request_threshold": 0.05, + } + src = _write( + tmp_path / "native.yaml", + model=_MODEL_ID, + benchmark_script=_VLLM_LAUNCHER, + agentx=native_config, + ) + + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi300x", + model_path="/models/local", + agentx_mode=True, + ) + + assert bench["agentx"] == native_config + assert bench["model"] == _MODEL_ID + assert bench["benchmark_script"] == _VLLM_LAUNCHER + assert bench["envs"]["MODEL_PATH"] == "/models/local" def test_switch_on_injects_model_and_run_eval(tmp_path, monkeypatch): _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "true") - src = _write(tmp_path / "base.yaml") + _enable_native(monkeypatch) + src = _write_native(tmp_path / "base.yaml") bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/model/x") envs = bench["envs"] assert envs["RUN_EVAL"] == "false" - assert envs["MODEL"] == "/model/x" + assert envs["MODEL_PATH"] == "/model/x" + assert "MODEL" not in envs + assert bench["model"] == _MODEL_ID + + +def test_remote_model_id_does_not_become_a_relative_model_path(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", _VLLM_LAUNCHER) + remote_model = "amd/GLM-5.2-MXFP4" + src = _write_native(tmp_path / "base.yaml") + + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path=remote_model) + + assert bench["model"] == remote_model + assert "MODEL_PATH" not in bench["envs"] + + +def test_switch_on_pins_zero_based_gpu_mask_for_native_launcher(tmp_path, monkeypatch): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + monkeypatch.setenv("TP", "4") + monkeypatch.delenv("ROCR_VISIBLE_DEVICES", raising=False) + src = _write_native(tmp_path / "base.yaml") + + bench = _materialize(src, tmp_path / "out", gpu_type="mi355x", model_path="/m") + + assert bench["envs"]["TP"] == 4 + assert bench["envs"]["ROCR_VISIBLE_DEVICES"] == "0,1,2,3" + + +def test_switch_on_rejects_nonzero_outer_gpu_mask(tmp_path, monkeypatch): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + monkeypatch.setenv("TP", "4") + monkeypatch.setenv("ROCR_VISIBLE_DEVICES", "4,5,6,7") + src = _write_native(tmp_path / "base.yaml") + + with pytest.raises(ValueError, match="ROCR_VISIBLE_DEVICES"): + _materialize(src, tmp_path / "out", gpu_type="mi355x", model_path="/m") -def test_switch_on_passes_agentx_env(tmp_path, monkeypatch): +def test_native_agentx_keeps_physical_mi325_runner_identity(tmp_path, monkeypatch): _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "on") + _enable_native(monkeypatch, launcher="single_node/agentic/glm5.2_fp8_mi325x_mtp.sh") + monkeypatch.setenv("TARGET_GPU_TYPE", "mi325x") + src = _write_native(tmp_path / "base.yaml") + + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/models/glm") + + assert bench["runner_type"] == "mi325x" + assert bench["benchmark_script"] == "single_node/agentic/glm5.2_fp8_mi325x_mtp.sh" + + +def test_native_switch_rejects_legacy_corpus_and_client_overrides(tmp_path, monkeypatch): + _clear_env(monkeypatch) + _enable_native(monkeypatch) monkeypatch.setenv("AGENTX_DATASET", "semianalysis-cc-traces-weka-with-subagents") monkeypatch.setenv("AGENTX_NUM_ENTRIES", "8") monkeypatch.setenv("AIPERF_BIN", "/venv/bin/aiperf") - src = _write(tmp_path / "base.yaml") - bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") - envs = bench["envs"] - assert envs["AGENTX_DATASET"] == "semianalysis-cc-traces-weka-with-subagents" - assert envs["AGENTX_NUM_ENTRIES"] == "8" - assert envs["AIPERF_BIN"] == "/venv/bin/aiperf" + src = _write_native(tmp_path / "base.yaml") + with pytest.raises(ValueError, match="corpus selection belongs"): + _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") def test_switch_on_materializes_workload_spec(tmp_path, monkeypatch): """AgentX ON must stamp a self-describing workload_spec into the recipe.""" _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native(monkeypatch) + monkeypatch.setenv("AGENTX_MODEL_ID", "moonshotai/Kimi-K3") monkeypatch.setenv("CONC", "8") monkeypatch.setenv("AGENTX_DURATION", "3600") - src = _write(tmp_path / "base.yaml") + src = _write_native(tmp_path / "base.yaml") bench = _materialize(src, tmp_path / "out", gpu_type="mi355x", model_path="/models/Kimi-K3") spec = bench.get("workload_spec") or {} assert spec.get("kind") == "agentx_trace_replay" assert spec.get("client") == "aiperf" assert spec.get("scenario") == "inferencex-agentx-mvp" + assert spec.get("harness") == "magpie-native-agentx" assert spec.get("corpus") == "semianalysis_cc_traces_weka_062126" assert spec.get("duration_s") == 3600 assert spec.get("geak_loop_duration_s") == 900 @@ -142,6 +320,67 @@ def test_switch_on_materializes_workload_spec(tmp_path, monkeypatch): assert placeholder.get("note") +def test_home_relative_checkpoint_is_expanded_before_launch(tmp_path, monkeypatch): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + monkeypatch.setenv("HOME", str(tmp_path)) + src = _write_native(tmp_path / "base.yaml") + + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="~/checkpoint") + + assert bench["envs"]["MODEL_PATH"] == str(tmp_path / "checkpoint") + + +def test_home_relative_checkpoint_cannot_be_inferred_as_hf_identity(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", _VLLM_LAUNCHER) + monkeypatch.setenv("HOME", str(tmp_path)) + src = _write_native(tmp_path / "base.yaml") + + with pytest.raises(ValueError, match="AGENTX_MODEL_ID"): + _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="~/checkpoint") + + +def test_local_checkpoint_name_does_not_choose_the_agentx_corpus(tmp_path, monkeypatch): + """Corpus identity comes from the recipe model, never a local basename.""" + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "amd/GLM-5.2-MXFP4") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", _VLLM_LAUNCHER) + src = _write_native(tmp_path / "base.yaml") + + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi355x", + model_path="/models/checkpoint", + ) + + assert bench["envs"]["MODEL_PATH"] == "/models/checkpoint" + assert bench["workload_spec"]["canonical_corpus"] == "semianalysis_cc_traces_weka_062126" + + +def test_profile_compat_uses_recipe_model_for_corpus(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "amd/GLM-5.2-MXFP4") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", _VLLM_LAUNCHER) + src = _write_native(tmp_path / "profile.yaml", profiler={"torch_profiler": {"enabled": True}}) + + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi355x", + model_path="/models/checkpoint", + allow_agentx_profile_compat=True, + ) + + assert bench["envs"]["MODEL"] == "/models/checkpoint" + assert bench["envs"]["AGENTX_MODEL_ID"] == "amd/GLM-5.2-MXFP4" + assert bench["workload_spec"]["canonical_corpus"] == "semianalysis_cc_traces_weka_062126" + + def test_switch_off_omits_workload_spec(tmp_path, monkeypatch): _clear_env(monkeypatch) src = _write(tmp_path / "base.yaml") @@ -149,14 +388,14 @@ def test_switch_off_omits_workload_spec(tmp_path, monkeypatch): assert "workload_spec" not in bench -def test_switch_forwards_weka_loader_override(tmp_path, monkeypatch): - """Upstream's own corpus pin has no ``AGENTX_`` prefix.""" +def test_native_switch_rejects_weka_loader_override(tmp_path, monkeypatch): + """A native corpus override would not participate in recipe identity.""" _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native(monkeypatch) monkeypatch.setenv("WEKA_LOADER_OVERRIDE", "semianalysis_cc_traces_weka_062126") - src = _write(tmp_path / "base.yaml") - bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") - assert bench["envs"]["WEKA_LOADER_OVERRIDE"] == "semianalysis_cc_traces_weka_062126" + src = _write_native(tmp_path / "base.yaml") + with pytest.raises(ValueError, match="WEKA_LOADER_OVERRIDE"): + _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") def test_switch_off_does_not_leak_weka_loader_override(tmp_path, monkeypatch): @@ -196,18 +435,67 @@ def test_switch_only_serving_frameworks(tmp_path, monkeypatch): assert bench.get("benchmark_script") != "aiperf_client.sh" +def test_profile_keeps_phase_gated_compatibility_client(tmp_path, monkeypatch): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + src = _write_native( + tmp_path / "profile.yaml", + envs={"PROFILE": "1"}, + profiler={"torch_profiler": {"enabled": True}}, + ) + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi300x", + model_path="/model/x", + allow_agentx_profile_compat=True, + ) + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert bench["envs"]["MODEL"] == "/model/x" + assert bench["envs"]["AGENTX_SERVER_SCRIPT"] == "" + assert bench["workload_spec"]["harness"] == "hyperloom-profiler-compat" + + +@pytest.mark.parametrize( + "extra", + [ + {"profiler": {"system_profiler": {"enabled": True}}}, + {"profiler": {"tracelens": {"enabled": "true"}}}, + {"gap_analysis": {"enabled": True}}, + ], +) +def test_all_magpie_incompatible_profile_modes_use_compat_client(tmp_path, monkeypatch, extra): + _clear_env(monkeypatch) + _enable_native(monkeypatch) + src = _write_native(tmp_path / "profile.yaml", **extra) + + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi300x", + model_path="/model/x", + allow_agentx_profile_compat=True, + ) + + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert bench["workload_spec"]["harness"] == "hyperloom-profiler-compat" + + # ── Regression: the shared grid/baseline/profile rebuild path (E1 bug) ──────── def test_runtime_overrides_honor_agentx_on(monkeypatch): """apply_runtime_benchmark_overrides must apply the switch, else the gpu_type-derived synthetic script silently reverts a materialize-time swap (the exact defect E1 caught: run_grid rebuilt to vllm_mi300x.sh).""" _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native(monkeypatch) from hyperloom.orchestrator.actions.executors._benchmark_runtime import ( apply_runtime_benchmark_overrides, ) - bench = {"framework": "vllm"} + bench = {"framework": "vllm", "agentx": "enable"} apply_runtime_benchmark_overrides(bench, model_path="/m", gpu_type="mi300x") - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == _VLLM_LAUNCHER + assert bench["agentx"] == "enable" assert bench["envs"]["RUN_EVAL"] == "false" @@ -228,9 +516,70 @@ def test_runtime_overrides_preserve_materialized_agentx_without_env(monkeypatch) apply_runtime_benchmark_overrides, ) - bench = {"framework": "vllm", "benchmark_script": "aiperf_client.sh"} + bench = { + "framework": "vllm", + "agentx": "enable", + "benchmark_script": _VLLM_LAUNCHER, + "envs": { + "AGENTX_MODEL_ID": _MODEL_ID, + "AGENTX_SERVER_SCRIPT": _VLLM_LAUNCHER, + }, + } apply_runtime_benchmark_overrides(bench, model_path="/m", gpu_type="mi300x") - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == _VLLM_LAUNCHER + assert bench["agentx"] == "enable" + + +def test_sglang_agentx_keeps_native_launcher_through_runtime_rebuild(tmp_path, monkeypatch): + """AgentX must not enter the generic SGLang runner that disables radix caching.""" + from hyperloom.orchestrator.actions.executors._benchmark_runtime import ( + apply_runtime_benchmark_overrides, + ) + + _clear_env(monkeypatch) + launcher = "single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh" + source = _write( + tmp_path / "glm-agentx.yaml", + framework="sglang", + model="amd/GLM-5.2-MXFP4", + runner_type="mi355x", + agentx="enable", + benchmark_script=launcher, + envs={"CONC": 8}, + ) + benchmark = _materialize(source, tmp_path / "out", gpu_type="mi355x", model_path="/models/glm") + + apply_runtime_benchmark_overrides(benchmark, model_path="/models/glm", gpu_type="mi355x", conc=8) + + assert benchmark["agentx"] == "enable" + assert benchmark["benchmark_script"] == launcher + assert benchmark["envs"]["AGENTX_SERVER_SCRIPT"] == launcher + assert benchmark["envs"]["CONC"] == 8 + assert "--disable-radix-cache" not in benchmark["envs"].get("EXTRA_SGLANG_ARGS", "") + + +def test_runtime_round_concurrency_overrides_stale_agentx_config(monkeypatch): + _clear_env(monkeypatch) + from hyperloom.orchestrator.actions.executors._benchmark_runtime import ( + apply_runtime_benchmark_overrides, + ) + + bench = { + "framework": "vllm", + "agentx": {"enabled": True, "mode": "canonical", "concurrency": 8}, + "benchmark_script": _VLLM_LAUNCHER, + "envs": { + "CONC": 8, + "AGENTX_MODEL_ID": _MODEL_ID, + "AGENTX_SERVER_SCRIPT": _VLLM_LAUNCHER, + }, + } + + apply_runtime_benchmark_overrides(bench, model_path="/m", gpu_type="mi300x", conc=16) + + assert bench["envs"]["CONC"] == 16 + assert bench["workload_spec"]["concurrency"] == 16 + assert "concurrency" not in bench["agentx"] # ── A2: OFF path never imports the agentx package (lazy-import guarantee) ───── @@ -289,15 +638,15 @@ def test_agentx_package_importable_contrast(): assert "CONTRAST_OK" in r.stdout -# ── Finding 1: framework injected so the wrapper delegates to the right builtin ─ -def test_switch_on_injects_framework_for_delegation(tmp_path, monkeypatch): - """ON must inject ``benchmark.framework`` into ``envs.FRAMEWORK`` so aiperf_client.sh delegates to ``{framework}_{gpu}.sh``.""" +# ── Framework identity reaches the native launcher environment ─────────────── +def test_switch_on_injects_framework_for_native_launcher(tmp_path, monkeypatch): _clear_env(monkeypatch) - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") for fw in ("sglang", "vllm"): - src = _write(tmp_path / f"{fw}.yaml", framework=fw) + launcher = f"single_node/agentic/test_fp4_mi300x_{fw}_mtp.sh" + _enable_native(monkeypatch, launcher=launcher) + src = _write_native(tmp_path / f"{fw}.yaml", framework=fw) bench = _materialize(src, tmp_path / f"out_{fw}", gpu_type="mi300x", model_path="/m") - assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["benchmark_script"] == launcher assert bench["envs"]["FRAMEWORK"] == fw diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_switch_legacy.py b/src/hyperloom/inference_optimizer/tests/test_agentx_switch_legacy.py new file mode 100644 index 0000000000..efbc21c755 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_switch_legacy.py @@ -0,0 +1,234 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Compatibility contracts for the environment-selected AgentX client.""" + +from __future__ import annotations + +import pytest +import yaml + +from hyperloom.orchestrator.actions.executors import _workload_envs as we + +_AGENTX_ENV_KEYS = ( + "HYPERLOOM_AGENTX", + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_DATASET", + "AGENTX_MAX_CTX", + "AGENTX_NUM_ENTRIES", + "AGENTX_DURATION", + "AGENTX_WARMUP_DURATION", + "AGENTX_NUM_WARMUP_SESSIONS", + "AGENTX_KEEP_SERVER", + "AIPERF_BIN", + "WEKA_LOADER_OVERRIDE", + "RUN_EVAL", + "MODEL_PATH", +) + + +def _clear_env(monkeypatch): + for k in _AGENTX_ENV_KEYS: + monkeypatch.delenv(k, raising=False) + + +def _write(path, **bench_extra): + bench = {"framework": "vllm", "model": "/m", "envs": {}} + bench.update(bench_extra) + path.write_text(yaml.safe_dump({"benchmark": bench}), encoding="utf-8") + return path + + +def _materialize(src, out, **kw): + res = we.materialize_config_with_envs(src, out, **kw) + return yaml.safe_load(res.read_text())["benchmark"] + + +def test_switch_on_authoritative_overwrite(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", "vllm_custom_server.sh") + src = _write(tmp_path / "base.yaml") + # gpu_type pre-pins vllm_mi300x.sh; the switch must overwrite it. + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert bench["envs"]["AGENTX_SERVER_SCRIPT"] == "vllm_custom_server.sh" + assert "timeout_seconds" not in bench + assert "AGENTX_PHASE_WAIT_TIMEOUT_S" not in bench["envs"] + + +def test_persisted_agentx_mode_switches_without_ambient_env(tmp_path, monkeypatch): + _clear_env(monkeypatch) + src = _write(tmp_path / "base.yaml") + bench = _materialize( + src, + tmp_path / "out", + gpu_type="mi300x", + model_path="/m", + agentx_mode=True, + ) + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + + +def test_switch_on_injects_model_and_run_eval(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "true") + src = _write(tmp_path / "base.yaml") + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/model/x") + envs = bench["envs"] + assert envs["RUN_EVAL"] == "false" + assert envs["MODEL"] == "/model/x" + + +def test_switch_on_passes_agentx_env(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "on") + monkeypatch.setenv("AGENTX_DATASET", "semianalysis-cc-traces-weka-with-subagents") + monkeypatch.setenv("AGENTX_NUM_ENTRIES", "8") + monkeypatch.setenv("AIPERF_BIN", "/venv/bin/aiperf") + src = _write(tmp_path / "base.yaml") + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") + envs = bench["envs"] + assert envs["AGENTX_DATASET"] == "semianalysis-cc-traces-weka-with-subagents" + assert envs["AGENTX_NUM_ENTRIES"] == "8" + assert envs["AIPERF_BIN"] == "/venv/bin/aiperf" + + +def test_switch_on_materializes_workload_spec(tmp_path, monkeypatch): + """AgentX ON must stamp a self-describing workload_spec into the recipe.""" + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("CONC", "8") + monkeypatch.setenv("AGENTX_DURATION", "3600") + src = _write(tmp_path / "base.yaml") + bench = _materialize(src, tmp_path / "out", gpu_type="mi355x", model_path="/models/Kimi-K3") + spec = bench.get("workload_spec") or {} + assert spec.get("kind") == "agentx_trace_replay" + assert spec.get("client") == "aiperf" + assert spec.get("scenario") == "inferencex-agentx-mvp" + assert spec.get("corpus") == "semianalysis_cc_traces_weka_062126" + assert spec.get("duration_s") == 3600 + assert spec.get("geak_loop_duration_s") == 900 + assert spec.get("concurrency") == 8 + placeholder = spec.get("isl_osl_placeholder") or {} + assert placeholder.get("note") + + +def test_switch_forwards_weka_loader_override(tmp_path, monkeypatch): + """Upstream's own corpus pin has no ``AGENTX_`` prefix.""" + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("WEKA_LOADER_OVERRIDE", "semianalysis_cc_traces_weka_062126") + src = _write(tmp_path / "base.yaml") + bench = _materialize(src, tmp_path / "out", gpu_type="mi300x", model_path="/m") + assert bench["envs"]["WEKA_LOADER_OVERRIDE"] == "semianalysis_cc_traces_weka_062126" + + +def test_runtime_overrides_honor_agentx_on(monkeypatch): + """apply_runtime_benchmark_overrides must apply the switch, else the gpu_type-derived synthetic script silently reverts a materialize-time swap (the exact defect E1 caught: run_grid rebuilt to vllm_mi300x.sh).""" + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + from hyperloom.orchestrator.actions.executors._benchmark_runtime import ( + apply_runtime_benchmark_overrides, + ) + + bench = {"framework": "vllm"} + apply_runtime_benchmark_overrides(bench, model_path="/m", gpu_type="mi300x") + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert bench["envs"]["RUN_EVAL"] == "false" + + +def test_runtime_overrides_preserve_materialized_agentx_without_env(monkeypatch): + _clear_env(monkeypatch) + from hyperloom.orchestrator.actions.executors._benchmark_runtime import ( + apply_runtime_benchmark_overrides, + ) + + bench = {"framework": "vllm", "benchmark_script": "aiperf_client.sh"} + apply_runtime_benchmark_overrides(bench, model_path="/m", gpu_type="mi300x") + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + + +def test_switch_on_injects_framework_for_delegation(tmp_path, monkeypatch): + """ON must inject ``benchmark.framework`` into ``envs.FRAMEWORK`` so aiperf_client.sh delegates to ``{framework}_{gpu}.sh``.""" + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + for fw in ("sglang", "vllm"): + src = _write(tmp_path / f"{fw}.yaml", framework=fw) + bench = _materialize(src, tmp_path / f"out_{fw}", gpu_type="mi300x", model_path="/m") + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert bench["envs"]["FRAMEWORK"] == fw + + +@pytest.mark.parametrize("framework", ["sglang", "vllm"]) +def test_legacy_profile_still_patches_the_framework(tmp_path, monkeypatch, framework): + from hyperloom.orchestrator.actions.executors import _server_patcher + + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("HYPERLOOM_ENABLE_PATCH", "1") + monkeypatch.setattr(_server_patcher, "resolve_sglang_shape_mode", lambda: "patched") + patch_calls = [] + + def patch_framework(): + patch_calls.append(framework) + return True + + monkeypatch.setattr(we, f"ensure_{framework}_patched_for_tracelens", patch_framework) + src = _write( + tmp_path / "profile.yaml", + framework=framework, + envs={"PROFILE": "1"}, + profiler={"torch_profiler": {"enabled": True}}, + ) + + bench = _materialize(src, tmp_path / "out", model_path="/models/checkpoint") + + assert patch_calls == [framework] + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in bench + assert "harness" not in bench["workload_spec"] + assert bench["envs"]["HYPERLOOM_TRACELENS_PATCH_STATUS"] == "ok" + + +def test_legacy_materialization_keeps_server_candidates_and_replay_controls(tmp_path, monkeypatch): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_DURATION", "240") + monkeypatch.setenv("AGENTX_NUM_ENTRIES", "8") + src = _write(tmp_path / "base.yaml") + + bench = _materialize( + src, + tmp_path / "out", + extra_server_args="--max-num-seqs 16", + extra_envs={"VLLM_USE_V1": "1", "CONC": "4"}, + ) + + assert "agentx" not in bench + assert bench["benchmark_script"] == "aiperf_client.sh" + assert "--max-num-seqs 16" in bench["envs"]["EXTRA_VLLM_ARGS"] + assert bench["envs"]["VLLM_USE_V1"] == "1" + assert bench["workload_spec"]["concurrency"] == 4 + assert bench["workload_spec"]["duration_s"] == 240 + assert bench["workload_spec"]["num_entries"] == 8 + + +@pytest.mark.parametrize("disabled", [False, "disable", {"enabled": False}]) +def test_disabled_native_yaml_keeps_the_legacy_environment_contract(tmp_path, monkeypatch, disabled): + _clear_env(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + src = _write(tmp_path / "base.yaml", agentx=disabled) + + bench = _materialize(src, tmp_path / "out", model_path="/models/checkpoint") + + assert bench["agentx"] == disabled + assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["envs"]["MODEL"] == "/models/checkpoint" + assert "harness" not in bench["workload_spec"] diff --git a/src/hyperloom/inference_optimizer/tests/test_agentx_variant_timeout.py b/src/hyperloom/inference_optimizer/tests/test_agentx_variant_timeout.py index 4bbadf0baf..f201c47e04 100644 --- a/src/hyperloom/inference_optimizer/tests/test_agentx_variant_timeout.py +++ b/src/hyperloom/inference_optimizer/tests/test_agentx_variant_timeout.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""The benchmark launch synchronizes inner and outer caps for every workload.""" +"""Benchmark watchdog policy preserves fixed native workload identities.""" from __future__ import annotations @@ -12,6 +12,30 @@ from hyperloom.orchestrator.actions.executors._subprocess_kill import resolve_benchmark_timeouts +@pytest.mark.parametrize("agentx", [{"enabled": True}, {"enabled": True, "launch_overrides": {"version": 1}}]) +def test_native_watchdog_does_not_rewrite_fingerprinted_config(tmp_path, agentx): + from hyperloom.inference_optimizer.agentx.identity import native_workload_fingerprint, verified_workload_fingerprint + + benchmark = { + "framework": "sglang", + "agentx": agentx, + "timeout_seconds": 7200, + "envs": {"CONC": 8}, + } + fingerprint = native_workload_fingerprint(benchmark, "a" * 64) + benchmark["workload_spec"] = { + "execution": {"static_execution_fingerprint": "a" * 64, "workload_fingerprint": fingerprint} + } + path = tmp_path / "native.yaml" + path.write_text(yaml.safe_dump({"benchmark": benchmark})) + original = path.read_bytes() + + sync_benchmark_timeout(path, 7800.0) + + assert path.read_bytes() == original + assert verified_workload_fingerprint(yaml.safe_load(path.read_text())["benchmark"]) == fingerprint + + @pytest.mark.parametrize("declared", [1800, 2400, 36000]) def test_launch_replaces_stale_yaml_cap_with_invoking_policy(tmp_path, monkeypatch, declared): monkeypatch.setenv("INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC", "123.5") diff --git a/src/hyperloom/inference_optimizer/tests/test_atom_kernel_backend_default.py b/src/hyperloom/inference_optimizer/tests/test_atom_kernel_backend_default.py index 098190d27e..cbd391c5e2 100644 --- a/src/hyperloom/inference_optimizer/tests/test_atom_kernel_backend_default.py +++ b/src/hyperloom/inference_optimizer/tests/test_atom_kernel_backend_default.py @@ -22,7 +22,7 @@ import pytest -from hyperloom.inference_optimizer.cli import _apply_atom_auto_tighten +from hyperloom.inference_optimizer import cli def _args(**overrides: object) -> argparse.Namespace: @@ -51,7 +51,7 @@ def test_an_unset_backend_is_left_for_the_shared_default(monkeypatch): """Writing anything here would make atom resolve differently from every other framework.""" monkeypatch.delenv(_KEY, raising=False) - _apply_atom_auto_tighten(_args()) + cli._apply_atom_auto_tighten(_args()) assert _KEY not in os.environ @@ -61,7 +61,7 @@ def test_a_named_backend_survives_verbatim(monkeypatch, value): """Every spelling reaches the shared resolver unedited, including the blank ones.""" monkeypatch.setenv(_KEY, value) - _apply_atom_auto_tighten(_args()) + cli._apply_atom_auto_tighten(_args()) assert os.environ[_KEY] == value @@ -72,7 +72,7 @@ def test_the_shared_default_puts_atom_on_geak(monkeypatch): monkeypatch.delenv(_KEY, raising=False) - _apply_atom_auto_tighten(_args()) + cli._apply_atom_auto_tighten(_args()) assert _raw_kernel_backend_order() == ["geak"] @@ -82,7 +82,7 @@ def test_multi_node_still_fails_fast(monkeypatch): monkeypatch.delenv(_KEY, raising=False) with pytest.raises(SystemExit) as exc: - _apply_atom_auto_tighten(_args(nodes=2)) + cli._apply_atom_auto_tighten(_args(nodes=2)) assert exc.value.code == 2 assert _KEY not in os.environ @@ -147,31 +147,45 @@ def guards_on_atom(test: ast.expr) -> bool: ) -def test_resume_reaches_the_guard_too(): - """A resumed atom session gets the same IR-8 check the launch did. - - ``--resume-from`` is the documented crash-recovery path, and it re-reads the - CLI arguments in a fresh process. A resume that skips this function would let - a ``--nodes>=2`` atom session start the very configuration the launch refused. - """ - ast, tree = _cli_source_tree() - - resume_ifs = [ - node - for node in ast.walk(tree) - if isinstance(node, ast.If) - and any(isinstance(sub, ast.Attribute) and sub.attr == "resume_from" for sub in ast.walk(node.test)) - and (node.body or node.orelse) - ] - assert resume_ifs, "no `if args.resume_from:` branch found; this test needs updating" - - reached = [ - node - for node in resume_ifs - if any(_calls_auto_tighten(ast, stmt) for stmt in node.body) - and any(_calls_auto_tighten(ast, stmt) for stmt in node.orelse) - ] - assert reached, ( - "_apply_atom_auto_tighten is applied on only one side of `if args.resume_from:`; " - "a resumed atom session would skip the IR-8 multi-node guard" +@pytest.mark.asyncio +@pytest.mark.parametrize("resume", [False, True], ids=["fresh", "resume"]) +async def test_cli_rejects_multinode_atom_on_fresh_and_resume(tmp_path, monkeypatch, capsys, resume): + """The public CLI preserves the same IR-8 gate across startup and recovery.""" + import json + from types import SimpleNamespace + + from hyperloom.orchestrator.state.shared_state import SharedState + + monkeypatch.setattr(os, "environ", dict(os.environ)) + monkeypatch.setenv("USER_DATA_PATH", str(tmp_path)) + monkeypatch.setenv("HYPERLOOM_AGENTX", "0") + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.setattr(cli, "_load_dotenv_fallback", lambda: None) + monkeypatch.setattr(cli, "_preflight", lambda args: {}) + monkeypatch.setattr(cli, "_resolve_models_for_run", lambda *args, **kwargs: None) + monkeypatch.setattr(cli, "_enforce_expected_framework", lambda framework: None) + monkeypatch.setattr(cli, "_apply_operator_supplied_paths", lambda *args: None) + monkeypatch.setattr(cli, "_persist_install_event", lambda *args: None) + monkeypatch.setattr(cli, "clean_stale_aiter_locks", lambda: {"dir": "", "deleted": 0}) + monkeypatch.setattr(cli, "_acquire_session_lock_or_exit", lambda *args: SimpleNamespace(release=lambda: None)) + monkeypatch.setattr( + "hyperloom.inference_optimizer.session.resume_guard.ensure_resume_safe", lambda *args, **kwargs: None ) + + async def no_quantize(args): + pass + + monkeypatch.setattr(cli, "_run_quantization_prelude", no_quantize) + argv = ["optimize", "--framework", "atom", "--nodes", "2", "--tp", "1", "--ep", "1"] + if resume: + session = tmp_path / "saved" + session.mkdir() + SharedState(session_id="saved", framework="atom", benchmark_mode="synthetic").save(session) + (session / "manifest.json").write_text(json.dumps({"session_id": "saved"})) + argv += ["--resume-from", str(session)] + else: + argv += ["--model", "Qwen/Qwen3-0.6B"] + with pytest.raises(SystemExit) as exc: + await cli._run_optimize(cli._build_parser().parse_args(argv)) + assert exc.value.code == 2 + assert "atom" in capsys.readouterr().err.lower() diff --git a/src/hyperloom/inference_optimizer/tests/test_backend_gating.py b/src/hyperloom/inference_optimizer/tests/test_backend_gating.py index 0008ba0bcd..2d5c62b60b 100644 --- a/src/hyperloom/inference_optimizer/tests/test_backend_gating.py +++ b/src/hyperloom/inference_optimizer/tests/test_backend_gating.py @@ -73,13 +73,9 @@ def test_install_sh_gates_magpie_calls(): # InferenceX stays unconditional (after the fi). inferencex_idx = text.index("ensure_inferencex\n", fi_idx) assert inferencex_idx > fi_idx - # The atomic-scripts patch also scrubs the redundant --concurrent-requests eval flag from the InferenceX benchmark - # copies, so it must run AFTER ensure_inferencex (which exports $INFERENCEX_PATH) and is gated on non-bypass by - # its own guard rather than the first else branch. - patch_guard_idx = text.index('if [ "$HYPERLOOM_BENCHMARK_BACKEND_LC" != "bypass" ]; then', gate_idx) - patch_idx = text.index("ensure_magpie_compat_patches\n", gate_idx) - patch_fi_idx = text.index("\nfi\n", patch_guard_idx) - assert inferencex_idx < patch_guard_idx < patch_idx < patch_fi_idx + # Install preserves the immutable execution trees; generic runtime paths + # apply their compatibility patches only when they need them. + assert "ensure_magpie_compat_patches" not in text def test_kernel_install_validates_ray_cli_and_serving_slot(): diff --git a/src/hyperloom/inference_optimizer/tests/test_baseline_param_overrides.py b/src/hyperloom/inference_optimizer/tests/test_baseline_param_overrides.py index 4c2efe6175..980d58fe6a 100644 --- a/src/hyperloom/inference_optimizer/tests/test_baseline_param_overrides.py +++ b/src/hyperloom/inference_optimizer/tests/test_baseline_param_overrides.py @@ -786,6 +786,117 @@ def fake_run(cmd, *args, **kwargs): # pragma: no cover assert "result_dir" in result["error"] +@pytest.mark.asyncio +async def test_legacy_agentx_baseline_keeps_candidate_runtime(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.setenv("FRAMEWORK", "sglang") + checkout = tmp_path / "legacy-inferencex" + monkeypatch.setenv("INFERENCEX_PATH", str(checkout)) + base = tmp_path / "legacy.yaml" + _write_yaml(base, framework="sglang") + state = SharedState(benchmark_mode="agentx", agentx_epoch=1) + executor = BaselineExecutor(default_config_path=base, session_dir=tmp_path, shared_state=state) + captured = {} + + async def measure(**kwargs): + captured.update(kwargs) + return {"status": "succeeded", "output_throughput": 100.0, "submission_valid": True} + + monkeypatch.setattr(executor, "_run_single_benchmark", measure) + monkeypatch.setattr( + "hyperloom.orchestrator.actions.executors.baseline.prepare_agentx_runtime", lambda **kwargs: None + ) + runtime = {"path_prefix": "/candidate/bin", "pythonpath_prefix": "/candidate/python"} + ctx = _make_ctx({"output_dir": str(tmp_path / "run"), "runtime_override": runtime}) + ctx.extra["shared_state"] = state + + result = await executor(ctx) + + assert result["status"] == "succeeded" + bench = yaml.safe_load(captured["config_path"].read_text(encoding="utf-8"))["benchmark"] + assert bench["benchmark_script"] == "aiperf_client.sh" + assert bench["envs"]["PATH"].split(":")[0] == "/candidate/bin" + assert bench["envs"]["PYTHONPATH"].split(":")[0] == "/candidate/python" + assert captured["inferencex_path"] == str(checkout) + assert "agentx" not in bench + + +def test_native_agentx_rejects_runtime_override_before_materialization(tmp_path): + base = tmp_path / "agentx.yaml" + _write_yaml(base, framework="sglang") + state = SimpleNamespace( + benchmark_mode="agentx", + agentx_epoch=2, + model_path="/models/glm", + ) + executor = BaselineExecutor( + magpie_python="/opt/venv/bin/python", + default_config_path=base, + session_dir=tmp_path, + shared_state=state, + ) + ctx = _make_ctx( + { + "output_dir": str(tmp_path / "ws"), + "runtime_override": { + "path_prefix": "/candidate/bin", + "pythonpath_prefix": "/candidate/python", + }, + } + ) + ctx.extra["shared_state"] = state + + with patch("hyperloom.orchestrator.actions.executors.baseline.materialize_config_with_envs") as materialize: + result = _run(executor(ctx)) + + assert result["status"] == "failed" + assert result["error_class"] == "unsupported_upstream_launcher_hook" + assert result["output_dir"] == str(tmp_path / "ws") + materialize.assert_not_called() + + +@pytest.mark.parametrize( + "mutation_params", + ( + {"patches": [{"patch_file": "candidate.patch"}]}, + {"warm_kernel_plan": [{"column": "gemm"}]}, + ), +) +def test_native_agentx_rejects_mutation_replay_before_materialization( + tmp_path, + mutation_params, +): + base = tmp_path / "agentx.yaml" + _write_yaml(base, framework="sglang") + state = SimpleNamespace( + benchmark_mode="agentx", + agentx_epoch=2, + model_path="/models/glm", + ) + executor = BaselineExecutor( + magpie_python="/opt/venv/bin/python", + default_config_path=base, + session_dir=tmp_path, + shared_state=state, + ) + ctx = _make_ctx( + { + "output_dir": str(tmp_path / "ws"), + **mutation_params, + } + ) + ctx.extra["shared_state"] = state + + with patch("hyperloom.orchestrator.actions.executors.baseline.materialize_config_with_envs") as materialize: + result = _run(executor(ctx)) + + assert result["status"] == "skipped" + assert result["error_class"] == "unsupported_upstream_launcher_hook" + assert result["output_dir"] == str(tmp_path / "ws") + materialize.assert_not_called() + + # reference-script base layer (precedence + 0-degrade) def _fw_args(materialized: Path, env_name: str = "EXTRA_VLLM_ARGS") -> str: cfg = yaml.safe_load(materialized.read_text()) diff --git a/src/hyperloom/inference_optimizer/tests/test_baseline_warmup_double_run.py b/src/hyperloom/inference_optimizer/tests/test_baseline_warmup_double_run.py index 4b7a9db36e..01bbb6059f 100644 --- a/src/hyperloom/inference_optimizer/tests/test_baseline_warmup_double_run.py +++ b/src/hyperloom/inference_optimizer/tests/test_baseline_warmup_double_run.py @@ -1767,6 +1767,51 @@ def fake_run(cmd, *args, **kwargs): assert "[REDACTED]" in result["error"] +@pytest.mark.parametrize( + "protocol_errors", [[], ["materialized_config_snapshot_mismatch", "recipe_fingerprint_mismatch"]] +) +def test_baseline_invalid_measurement_reports_native_protocol_reasons(tmp_path, protocol_errors): + """Positive metrics with invalid native identity must report the rejected protocol.""" + base = tmp_path / "base.yaml" + _write_yaml(base) + + def fake_run(cmd, **kwargs): + _fake_workspace(Path(cmd[cmd.index("--output-dir") + 1]), tput=264.26) + return subprocess.CompletedProcess(cmd, 0, "ok", "") + + measurement = { + "valid_measurement": False, + "reported_success": True, + "throughput": 264.26, + "completed_requests": 1196, + "native_agentx_protocol_errors": protocol_errors, + } + executor = _executor(base, tmp_path) + ctx = _make_ctx( + { + "baseline_double_run": False, + "output_dir": str(tmp_path / "output"), + "timeout_sec": 10, + "gpu_type": "mi300x", + } + ) + with ( + patch("hyperloom.orchestrator.actions.executors.baseline.run_with_session_kill", side_effect=fake_run), + patch( + "hyperloom.orchestrator.actions.executors.baseline.extract_benchmark_measurement", return_value=measurement + ), + ): + result = _run(executor(ctx)) + + assert result["status"] == "failed" + assert result["error_class"] == "invalid_measurement" + assert result["reported_success"] is True + if protocol_errors: + assert result["error"] == "native AgentX report failed protocol validation: " + "; ".join(protocol_errors) + else: + assert result["error"] == "benchmark report did not contain positive throughput and completed requests" + + def test_baseline_clears_stale_server_log_before_run(tmp_path, monkeypatch): """A stale server.log death marker in a reused output_dir must NOT bias a fresh attempt's classification.""" base = tmp_path / "base.yaml" diff --git a/src/hyperloom/inference_optimizer/tests/test_benchmark_result.py b/src/hyperloom/inference_optimizer/tests/test_benchmark_result.py index 1bf08b4aec..2ee0843551 100644 --- a/src/hyperloom/inference_optimizer/tests/test_benchmark_result.py +++ b/src/hyperloom/inference_optimizer/tests/test_benchmark_result.py @@ -174,6 +174,826 @@ def _write_inferencex(path: Path, tput: float = 1761.6, completed: int = 640) -> ) +_NATIVE_MODEL = "deepseek-ai/DeepSeek-V4-Pro-0813" +_NATIVE_MODEL_PREFIX = "dsv4" +_NATIVE_IMAGE = "lmsysorg/sglang-rocm:test" +_NATIVE_RECIPE = "dsv4-fp4-mi355x-sglang-agentic-mtp" +_NATIVE_LAUNCHER = "single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh" +_NATIVE_DATASET = { + "source_type": "public_dataset", + "loader": "semianalysis_cc_traces_weka_062126", + "hf_dataset_name": "semianalysisai/cc-traces-weka-062126", + "hf_split": "train", + "num_dataset_entries": 393, +} + + +def _native_accounting() -> dict: + return { + "records_total": 392, + "records_profiled": 390, + "records_dropped_total": 2, + "records_warmup_dropped": 0, + "records_error_dropped": 2, + "error_categories": {"http_error": 2}, + } + + +def _native_raw(*, pcp: int = 1, ep: int = 1) -> dict: + return { + "scenario_type": "agentic-coding", + "recipe_fingerprint": "a" * 64, + "model": _NATIVE_MODEL, + "infmax_model_prefix": _NATIVE_MODEL_PREFIX, + "framework": "sglang", + "precision": "fp4", + "image": _NATIVE_IMAGE, + "conc": 16, + "tp": 8, + "pp": 1, + "pcp_size": pcp, + "ep": ep, + "num_requests_total": 392, + "num_requests_successful": 390, + "request_accounting": _native_accounting(), + "dataset": dict(_NATIVE_DATASET), + "request_metrics": { + "qps": {"mean": 0.42}, + "throughput": { + "input": {"tokens_per_second": 114_000.0}, + "output": {"tokens_per_second": 812.5}, + "total": {"tokens_per_second": 114_812.5}, + "duration_seconds": 3600.0, + }, + "latency": { + "ttft": {"mean": 0.410, "p50": 0.3, "p90": 0.9, "p95": 1.25}, + "tpot": {"mean": 0.019, "p50": 0.015, "p90": 0.027, "p95": 0.031}, + "e2el": {"mean": 3.3, "p95": 9.1}, + "e2e_norm_intvty": {"p50": 300.0, "p90": 184.75}, + }, + "tokens": {}, + "cache": {}, + }, + } + + +def _native_agentx_report(*, benchmark_valid: bool = True, publishable: bool = True) -> dict: + return { + "success": True, + "scenario": "agentx", + "benchmark_valid": benchmark_valid, + "publishable": publishable, + "framework": "sglang", + "model": _NATIVE_MODEL, + "throughput": { + "request_throughput": 0.42, + "output_throughput": 812.5, + "total_token_throughput": 114_812.5, + "completed_requests": 390, + "duration_seconds": 3600.0, + }, + "latency": { + "ttft": {"mean_ms": 410.0, "p99_ms": 0.0}, + "tpot": {"mean_ms": 19.0}, + "e2el": {"mean_ms": 3300.0, "p99_ms": 0.0}, + }, + "agentx_metrics": { + "mode": "canonical" if publishable else "fast", + "scenario_type": "agentic-coding", + "recipe_fingerprint_valid": True, + "requests": { + "total": 392, + "records_total": 392, + "profiled_total": 392, + "successful": 390, + "errors": 2, + "warmup_dropped": 0, + "error_rate": 2 / 392, + "threshold": 0.10, + }, + "throughput": { + "request_throughput": 0.42, + "input_tokens_per_second": 114_000.0, + "output_tokens_per_second": 812.5, + "total_tokens_per_second": 114_812.5, + "duration_seconds": 3600.0, + }, + "latency_seconds": { + "ttft": {"mean": 0.410, "p50": 0.3, "p90": 0.9, "p95": 1.25}, + "tpot": {"mean": 0.019, "p50": 0.015, "p90": 0.027, "p95": 0.031}, + "e2el": {"mean": 3.3, "p95": 9.1}, + "e2e_norm_intvty": {"p50": 300.0, "p90": 184.75}, + }, + "recipe": { + "tp": 8, + "pp": 1, + "ep": 1, + "conc": 16, + "image": _NATIVE_IMAGE, + "model": _NATIVE_MODEL, + "infmax_model_prefix": _NATIVE_MODEL_PREFIX, + "framework": "sglang", + "precision": "fp4", + "kv_offloading": "none", + "recipe_fingerprint": "a" * 64, + }, + "launch": { + "recipe": _NATIVE_RECIPE, + "docker_image": _NATIVE_IMAGE, + "benchmark_script": _NATIVE_LAUNCHER, + "recipe_fingerprint": "a" * 64, + }, + "dataset": dict(_NATIVE_DATASET), + "request_accounting": _native_accounting(), + }, + } + + +def _write_native_config_snapshot( + workspace: Path, + *, + fingerprint: str = "a" * 64, + tp: int = 8, + pp: int = 1, + pcp: int = 1, + ep: int = 1, + write_raw: bool = True, +) -> None: + (workspace / "config.yaml").write_text( + json.dumps( + { + "framework": "sglang", + "model": _NATIVE_MODEL, + "precision": "fp4", + "docker_image": _NATIVE_IMAGE, + "benchmark_script": _NATIVE_LAUNCHER, + "agentx": { + "recipe": _NATIVE_RECIPE, + "mode": "canonical", + "resolved": { + "image": _NATIVE_IMAGE, + "model": _NATIVE_MODEL, + "model-prefix": _NATIVE_MODEL_PREFIX, + "framework": "sglang", + "precision": "fp4", + "tp": tp, + "pp": pp, + "pcp-size": pcp, + "ep": ep, + "conc": 16, + "duration": 3600, + "recipe-fingerprint": fingerprint, + }, + }, + "workload_spec": { + "corpus": _NATIVE_DATASET["loader"], + "concurrency": 16, + "duration_s": 3600, + }, + } + ), + encoding="utf-8", + ) + artifact = workspace / "aiperf_artifacts" / "profile_export_aiperf.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text( + json.dumps( + { + "was_cancelled": False, + "input_config": { + "scenario": "inferencex-agentx-mvp", + "models": { + "strategy": "round_robin", + "items": [{"name": _NATIVE_MODEL}], + }, + "tokenizer": {"name": _NATIVE_MODEL}, + "phases": [ + { + "name": "profiling", + "kind": "profiling", + "timing_mode": "agentic_replay", + "concurrency": 16, + "duration": 3600, + } + ], + }, + "metadata": { + "scenario": "inferencex-agentx-mvp", + "submission_valid": True, + "submission_invalid_reasons": [], + "dataset": dict(_NATIVE_DATASET), + "metric_duration_coverage": [ + { + "phase_name": "profiling", + "expected_duration_seconds": 3600, + "required_ratio": 0.95, + "ttft_ratio": 0.99, + "inter_token_latency_ratio": 0.99, + } + ], + }, + "benchmark_duration": {"unit": "seconds", "avg": 3600.0}, + "request_count": {"unit": "requests", "avg": 390.0}, + } + ), + encoding="utf-8", + ) + if write_raw: + (workspace / "inferencex_result.json").write_text( + json.dumps(_native_raw(pcp=pcp, ep=ep)), + encoding="utf-8", + ) + + +def test_extracts_native_magpie_agentx_schema_with_resolved_topology(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is True + assert measurement["input_throughput"] == pytest.approx(114_000.0) + assert measurement["output_throughput"] == pytest.approx(812.5) + assert measurement["total_token_throughput"] == pytest.approx(114_812.5) + assert measurement["ttft_p99_ms"] is None + assert measurement["ttft_p95_ms"] == pytest.approx(1250.0) + assert measurement["ttft_p50_ms"] == pytest.approx(300.0) + assert measurement["ttft_p90_ms"] == pytest.approx(900.0) + assert measurement["tpot_p50_ms"] == pytest.approx(15.0) + assert measurement["tpot_p90_ms"] == pytest.approx(27.0) + assert measurement["e2el_p99_ms"] is None + assert measurement["e2el_p95_ms"] == pytest.approx(9100.0) + assert measurement["e2e_norm_intvty_p90"] == pytest.approx(184.75) + assert measurement["e2e_norm_intvty_p50"] == pytest.approx(300.0) + assert measurement["completed_requests"] == 390 + assert measurement["requested_requests"] is None + assert measurement["native_agentx_protocol_valid"] is True + assert measurement["submission_valid"] is True + assert measurement["request_error_rate"] == pytest.approx(100 * 2 / 392) + assert measurement["agentx_gpu_count"] == 8 + assert measurement["agentx_gpu_count_source"] == "inferencex_raw" + + +def _native_aiperf_path(workspace: Path) -> Path: + return workspace / "aiperf_artifacts" / "profile_export_aiperf.json" + + +def _write_native_request_accounting(workspace, *, successful, dropped, warmup, errors, measured_errors): + _write_native_config_snapshot(workspace) + total = successful + dropped + accounting = { + "records_total": total, + "records_profiled": successful, + "records_dropped_total": dropped, + "records_warmup_dropped": warmup, + "records_error_dropped": errors, + "error_categories": {"InvalidInferenceResultError": errors} if errors else {}, + } + raw = _native_raw() + raw.update(num_requests_total=total, num_requests_successful=successful, request_accounting=accounting) + (workspace / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + artifact = _native_aiperf_path(workspace) + aiperf = json.loads(artifact.read_text(encoding="utf-8")) + aiperf["request_count"]["avg"] = float(successful) + artifact.write_text(json.dumps(aiperf), encoding="utf-8") + report = _native_agentx_report() + report["throughput"]["completed_requests"] = successful + report["agentx_metrics"]["request_accounting"] = accounting + report["agentx_metrics"]["requests"] = { + "total": total, + "records_total": total, + "profiled_total": successful + measured_errors, + "successful": successful, + "errors": measured_errors, + "warmup_dropped": warmup, + "error_rate": measured_errors / (successful + measured_errors), + "threshold": 0.1, + } + return report + + +@pytest.mark.parametrize( + ("successful", "dropped", "warmup", "errors", "measured_errors"), + [ + (1279, 87, 87, 1, 0), # The real GPU run: one warmup error, no measured errors. + (100, 13, 10, 5, 3), # Two warmup errors and three measured errors. + (100, 12, 10, 2, 2), # Every error belongs to the measurement phase. + (100, 2, 0, 2, 2), + ], +) +def test_native_agentx_protocol_counts_only_measured_errors( + tmp_path, monkeypatch, successful, dropped, warmup, errors, measured_errors +): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _write_native_request_accounting( + tmp_path, + successful=successful, + dropped=dropped, + warmup=warmup, + errors=errors, + measured_errors=measured_errors, + ) + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is True + assert measurement["native_agentx_protocol_errors"] == [] + assert measurement["request_error_rate"] == pytest.approx(100 * measured_errors / (successful + measured_errors)) + assert measurement["agentx_request_accounting"]["records_error_dropped"] == errors + + +@pytest.mark.parametrize("reported_errors", [0, 5]) +def test_native_agentx_protocol_rejects_erased_or_warmup_inflated_errors(tmp_path, monkeypatch, reported_errors): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _write_native_request_accounting( + tmp_path, successful=100, dropped=13, warmup=10, errors=5, measured_errors=reported_errors + ) + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert { + "report_profiled_total_mismatch", + "report_error_count_mismatch", + "report_error_rate_mismatch", + } <= set(measurement["native_agentx_protocol_errors"]) + + +@pytest.mark.parametrize(("dropped", "warmup", "errors"), [(9, 10, 1), (10, 9, 11), (13, 10, 2)]) +def test_native_agentx_protocol_rejects_impossible_warmup_error_union(tmp_path, monkeypatch, dropped, warmup, errors): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _write_native_request_accounting( + tmp_path, successful=100, dropped=dropped, warmup=warmup, errors=errors, measured_errors=0 + ) + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert "request_accounting_drop_bounds_mismatch" in measurement["native_agentx_protocol_errors"] + + +def test_native_agentx_protocol_keeps_all_phase_error_category_validation(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _write_native_request_accounting( + tmp_path, successful=100, dropped=13, warmup=10, errors=5, measured_errors=3 + ) + report["agentx_metrics"]["request_accounting"]["error_categories"] = {"InvalidInferenceResultError": 3} + raw_path = tmp_path / "inferencex_result.json" + raw = json.loads(raw_path.read_text(encoding="utf-8")) + raw["request_accounting"] = report["agentx_metrics"]["request_accounting"] + raw_path.write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert "request_accounting_error_categories_mismatch" in measurement["native_agentx_protocol_errors"] + + +@pytest.mark.parametrize( + ("median", "tail", "output", "duration", "error_rate", "verdict"), + [ + (309.0, 184.75, 812.5, 3600.0, 100 * 2 / 392, "KEEP"), + (308.9, 184.75, 812.5, 3600.0, 100 * 2 / 392, "REVERT"), + (330.0, 170.0, 812.5, 3600.0, 100 * 2 / 392, "REVERT"), + (330.0, 184.75, 700.0, 3600.0, 100 * 2 / 392, "REVERT"), + (330.0, 184.75, 812.5, 3000.0, 100 * 2 / 392, "REVERT"), + (330.0, 184.75, 812.5, 3600.0, 1.0, "REVERT"), + ], +) +def test_native_measurement_uses_median_grading_and_all_guards( + tmp_path, monkeypatch, median, tail, output, duration, error_rate, verdict +): + from types import SimpleNamespace + + from hyperloom.orchestrator.state.shared_state import resolve_graded_comparison + + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + monkeypatch.delenv("HYPERLOOM_PERF_METRIC", raising=False) + _write_native_config_snapshot(tmp_path) + anchor = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + assert anchor["valid_measurement"] is True + candidate = { + **anchor, + "e2e_norm_intvty_p50": median, + "e2e_norm_intvty_p90": tail, + "output_throughput": output, + "duration_seconds": duration, + "request_error_rate": error_rate, + } + comparison = resolve_graded_comparison(SimpleNamespace(benchmark_mode="agentx", current_best=anchor), candidate) + + assert comparison.objective == "e2e_norm_intvty_p50" + assert comparison.verdict == verdict + + +def test_native_agentx_rejects_one_second_publishable_summary(tmp_path, monkeypatch): + """A compact Magpie success cannot substitute for the canonical window.""" + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + report = _native_agentx_report() + report["throughput"]["duration_seconds"] = 1.0 + report["agentx_metrics"]["throughput"]["duration_seconds"] = 1.0 + artifact = _native_aiperf_path(tmp_path) + payload = json.loads(artifact.read_text(encoding="utf-8")) + payload["benchmark_duration"]["avg"] = 1.0 + artifact.write_text(json.dumps(payload), encoding="utf-8") + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert "aiperf_benchmark_duration_incomplete" in measurement["native_agentx_protocol_errors"] + + +@pytest.mark.parametrize( + ("mutation", "expected_error"), + [ + ("cancelled", "aiperf_cancelled_or_unknown"), + ("coverage", "aiperf_duration_coverage_failed"), + ("accounting", "request_accounting_sum_mismatch"), + ("accounting_details", "request_accounting_drop_bounds_mismatch"), + ("report_accounting", "report_request_accounting_mismatch"), + ("throughput", "report_total_tokens_per_second_mismatch"), + ("latency", "report_latency_mismatch"), + ("model", "aiperf_model_mismatch"), + ("models_missing", "aiperf_models_invalid"), + ("tokenizer", "aiperf_tokenizer_mismatch"), + ("tokenizer_missing", "aiperf_tokenizer_invalid"), + ("dataset", "dataset_provenance_mismatch"), + ], +) +def test_native_agentx_protocol_tampering_fails_closed( + tmp_path, + monkeypatch, + mutation, + expected_error, +): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + report = _native_agentx_report() + if mutation in { + "cancelled", + "coverage", + "model", + "models_missing", + "tokenizer", + "tokenizer_missing", + }: + artifact = _native_aiperf_path(tmp_path) + payload = json.loads(artifact.read_text(encoding="utf-8")) + if mutation == "cancelled": + payload["was_cancelled"] = True + elif mutation == "coverage": + coverage = payload["metadata"]["metric_duration_coverage"][0] + coverage["ttft_ratio"] = 0.1 + coverage["inter_token_latency_ratio"] = 0.1 + elif mutation == "model": + payload["input_config"]["models"]["items"][0]["name"] = "other/model" + elif mutation == "models_missing": + payload["input_config"].pop("models") + elif mutation == "tokenizer": + payload["input_config"]["tokenizer"]["name"] = "other/model" + else: + payload["input_config"].pop("tokenizer") + artifact.write_text(json.dumps(payload), encoding="utf-8") + elif mutation == "latency": + report["agentx_metrics"]["latency_seconds"]["ttft"]["mean"] = 99.0 + elif mutation == "report_accounting": + report["agentx_metrics"]["request_accounting"]["records_error_dropped"] = 3 + else: + raw_path = tmp_path / "inferencex_result.json" + raw = json.loads(raw_path.read_text(encoding="utf-8")) + if mutation == "accounting": + raw["request_accounting"]["records_total"] = 391 + raw["num_requests_total"] = 391 + elif mutation == "accounting_details": + raw["request_accounting"]["records_error_dropped"] = 999 + raw["request_accounting"]["error_categories"] = {"boom": 999} + elif mutation == "throughput": + raw["request_metrics"]["throughput"]["total"] = {"tokens_per_second": 1.0} + else: + raw["dataset"]["loader"] = "untrusted_loader" + raw_path.write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert expected_error in measurement["native_agentx_protocol_errors"] + + +def test_native_agentx_requires_one_unambiguous_aiperf_export(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + duplicate = tmp_path / "aiperf_artifacts" / "run_1" / "profile_export_aiperf.json" + duplicate.parent.mkdir(parents=True) + duplicate.write_bytes(_native_aiperf_path(tmp_path).read_bytes()) + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert "aiperf_artifact_ambiguous" in measurement["native_agentx_protocol_errors"] + + +def test_native_agentx_without_explicit_pcp_or_resolved_topology_fails_closed(monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + measurement = extract_benchmark_measurement(_native_agentx_report()) + + assert measurement["native_agentx_schema_valid"] is True + assert "agentx_gpu_count" not in measurement + assert measurement["valid_measurement"] is False + + +def test_native_agentx_nonpublishable_report_is_rejected(monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report(publishable=False) + measurement = extract_benchmark_measurement(report) + + assert measurement["benchmark_valid"] is True + assert measurement["publishable"] is False + assert measurement["submission_valid"] is False + assert "native_agentx_fast_mode" in measurement["submission_invalid_reasons"] + assert measurement["valid_measurement"] is False + + +def test_native_agentx_requires_benchmark_valid_and_publishable(monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report(benchmark_valid=False, publishable=True) + measurement = extract_benchmark_measurement(report) + + assert measurement["submission_valid"] is False + assert "native_agentx_benchmark_invalid" in measurement["submission_invalid_reasons"] + assert measurement["valid_measurement"] is False + + +def test_native_agentx_missing_metrics_schema_fails_closed(monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report() + report.pop("agentx_metrics") + measurement = extract_benchmark_measurement(report) + + assert measurement["native_agentx_report"] is True + assert measurement["native_agentx_schema_valid"] is False + assert measurement["valid_measurement"] is False + + +def test_native_agentx_empty_metrics_schema_fails_closed(monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report() + report["agentx_metrics"] = {} + + measurement = extract_benchmark_measurement(report) + + assert measurement["native_agentx_schema_valid"] is False + assert measurement["native_agentx_schema_errors"] + assert measurement["valid_measurement"] is False + + +@pytest.mark.parametrize( + ("field", "value", "error"), + [ + ("scenario_type", "synthetic", "agentx_metrics.scenario_type_invalid"), + ("recipe_fingerprint_valid", False, "agentx_metrics.recipe_fingerprint_valid_invalid"), + ], +) +def test_native_agentx_identity_schema_fails_closed(monkeypatch, field, value, error): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report() + report["agentx_metrics"][field] = value + + measurement = extract_benchmark_measurement(report) + + assert measurement["native_agentx_schema_valid"] is False + assert error in measurement["native_agentx_schema_errors"] + assert measurement["valid_measurement"] is False + + +def test_native_agentx_recipe_and_launch_fingerprints_must_match(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + report = _native_agentx_report() + report["agentx_metrics"]["launch"]["recipe_fingerprint"] = "b" * 64 + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + assert measurement["native_agentx_schema_valid"] is False + assert "agentx_metrics.recipe_launch_fingerprint_mismatch" in measurement["native_agentx_schema_errors"] + assert measurement["valid_measurement"] is False + + +@pytest.mark.parametrize("launch", [None, {}, {"recipe_fingerprint": "not-a-sha256"}]) +def test_native_agentx_launch_fingerprint_is_required(monkeypatch, launch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report() + if launch is None: + report["agentx_metrics"].pop("launch") + else: + report["agentx_metrics"]["launch"] = launch + + measurement = extract_benchmark_measurement(report) + + assert measurement["native_agentx_schema_valid"] is False + assert measurement["valid_measurement"] is False + + +def test_native_agentx_config_snapshot_fingerprint_must_match(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path, fingerprint="b" * 64) + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert "native_agentx_config_recipe_fingerprint_mismatch" in measurement["nonfatal_warnings"] + assert "config_report_recipe_fingerprint_mismatch" in measurement["native_agentx_protocol_errors"] + assert measurement["valid_measurement"] is False + + +def test_native_agentx_config_snapshot_topology_must_match_report(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + report = _native_agentx_report() + report["agentx_metrics"]["recipe"]["pcp_size"] = 1 + _write_native_config_snapshot(tmp_path, tp=4, pp=2, pcp=1) + + measurement = extract_benchmark_measurement(report, workspace=tmp_path) + + # Both shapes total eight GPUs, so count-only checking would miss this + # corrupt recipe association. + assert measurement["native_agentx_topology_conflict"] is True + assert "native_agentx_config_topology_invalid" in measurement["nonfatal_warnings"] + assert measurement["valid_measurement"] is False + + +def test_native_raw_aggregate_restores_topology_corpus_and_cache(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path, pcp=2, write_raw=False) + raw = _native_raw(pcp=2) + raw["request_metrics"].update( + { + "tokens": { + "input": {"mean": 100_000, "p50": 90_000, "p75": 120_000, "p90": 160_000, "p95": 220_000}, + "output_actual": {"mean": 800, "p50": 330, "p75": 800, "p90": 1_800, "p95": 3_000}, + }, + "cache": {"theoretical_cache_hit_rate": 0.972}, + } + ) + raw["server_metrics"] = {"cache": {"overall_cache_hit_rate": 0.968}} + (tmp_path / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is True + assert measurement["native_agentx_protocol_valid"] is True + assert measurement["agentx_gpu_count"] == 16 + assert measurement["agentx_gpu_count_source"] == "inferencex_raw" + assert measurement["isl_distribution"]["avg"] == 100_000 + assert measurement["osl_distribution"]["p95"] == 3_000 + assert measurement["corpus_loader"] == "semianalysis_cc_traces_weka_062126" + assert measurement["theoretical_prefix_cache_hit"] == pytest.approx(0.972) + assert measurement["agentx_server_cache"]["overall_cache_hit_rate"] == pytest.approx(0.968) + + +def test_native_raw_partial_pinned_schema_enriches_with_trusted_topology(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path, tp=8, pp=1, pcp=2, ep=8) + raw = { + "scenario_type": "agentic-coding", + "recipe_fingerprint": "a" * 64, + # This is the schema emitted by the pinned InferenceX aggregator. PP + # and PCP exist only in Magpie's fingerprint-bound config snapshot. + "tp": 8, + "ep": 8, + "dataset": {"loader": "semianalysis_cc_traces_weka_062126"}, + "request_metrics": { + "tokens": {"input": {"mean": 100_000}, "output_actual": {"p95": 3_000}}, + "cache": {"theoretical_cache_hit_rate": 0.972}, + }, + } + raw_path = tmp_path / "inferencex_result.json" + raw_path.write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert measurement["native_agentx_protocol_valid"] is False + assert "pcp_missing" in measurement["native_agentx_protocol_errors"] + assert measurement["agentx_gpu_count"] == 16 + assert measurement["agentx_gpu_count_source"] == "magpie_config_snapshot" + assert measurement["agentx_gpu_topology"] == {"tp": 8, "pp": 1, "pcp": 2, "ep": 8} + assert measurement["isl_distribution"]["avg"] == 100_000 + assert measurement["osl_distribution"]["p95"] == 3_000 + assert measurement["corpus_loader"] == "semianalysis_cc_traces_weka_062126" + assert measurement["theoretical_prefix_cache_hit"] == pytest.approx(0.972) + assert measurement["raw_result_path"] == str(raw_path) + + +@pytest.mark.parametrize(("dimension", "contradicting_value"), [("tp", 4), ("ep", 4)]) +def test_native_raw_partial_topology_contradiction_is_rejected( + tmp_path, + monkeypatch, + dimension, + contradicting_value, +): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path, tp=8, pp=1, pcp=2, ep=8) + raw = { + "scenario_type": "agentic-coding", + "recipe_fingerprint": "a" * 64, + "tp": 8, + "ep": 8, + "dataset": {"loader": "must-not-be-consumed"}, + } + raw[dimension] = contradicting_value + (tmp_path / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + # A contradictory aggregate may not be used to validate the canonical + # protocol even though the compact Magpie report itself remains readable. + assert measurement["valid_measurement"] is False + assert measurement["agentx_gpu_count"] == 16 + assert measurement["agentx_gpu_count_source"] == "magpie_config_snapshot" + assert "corpus_loader" not in measurement + assert measurement["raw_result_path"] is None + assert any( + warning.startswith("native_agentx_raw_topology_mismatch:") for warning in measurement["nonfatal_warnings"] + ) + + +@pytest.mark.parametrize("fingerprint", [None, "b" * 64]) +def test_native_raw_aggregate_fingerprint_is_required_before_enrichment(tmp_path, monkeypatch, fingerprint): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path) + raw = { + "scenario_type": "agentic-coding", + "recipe_fingerprint": fingerprint, + "tp": 8, + "pp": 1, + "pcp_size": 1, + "dataset": {"loader": "untrusted-loader"}, + "request_metrics": { + "tokens": {"input": {"mean": 999_999}}, + "cache": {"theoretical_cache_hit_rate": 1.0}, + }, + } + (tmp_path / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is False + assert measurement["agentx_gpu_count_source"] == "magpie_config_snapshot" + assert "isl_distribution" not in measurement + assert "corpus_loader" not in measurement + assert "theoretical_prefix_cache_hit" not in measurement + assert measurement["raw_result_path"] is None + assert any( + warning.startswith("native_agentx_raw_recipe_fingerprint_") for warning in measurement["nonfatal_warnings"] + ) + + +def test_native_raw_fingerprint_rejection_does_not_block_later_matching_aggregate(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + _write_native_config_snapshot(tmp_path, pcp=2, write_raw=False) + rejected = { + "scenario_type": "agentic-coding", + "recipe_fingerprint": "b" * 64, + "tp": 8, + "pp": 1, + "pcp_size": 1, + "dataset": {"loader": "wrong"}, + } + accepted = _native_raw(pcp=2) + (tmp_path / "a_rejected.json").write_text(json.dumps(rejected), encoding="utf-8") + accepted_path = tmp_path / "b_accepted.json" + accepted_path.write_text(json.dumps(accepted), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert measurement["valid_measurement"] is True + assert measurement["agentx_gpu_count"] == 16 + assert measurement["corpus_loader"] == _NATIVE_DATASET["loader"] + assert measurement["raw_result_path"] == str(accepted_path) + + +def test_native_raw_without_complete_topology_does_not_guess_parallelism(tmp_path, monkeypatch): + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + raw = { + "scenario_type": "agentic-coding", + "recipe_fingerprint": "a" * 64, + "tp": 8, + "pp": 1, + # PCP is deliberately absent. A default of one would make aggregate + # throughput look like trustworthy per-GPU throughput for PCP>1. + "dataset": {"loader": "must-not-be-consumed"}, + } + (tmp_path / "inferencex_result.json").write_text(json.dumps(raw), encoding="utf-8") + + measurement = extract_benchmark_measurement(_native_agentx_report(), workspace=tmp_path) + + assert "agentx_gpu_count" not in measurement + assert "corpus_loader" not in measurement + assert measurement["raw_result_path"] is None + assert any( + warning.startswith("native_agentx_raw_topology_invalid:") for warning in measurement["nonfatal_warnings"] + ) + assert measurement["valid_measurement"] is False + + def test_rescue_from_env_path_adopted_after_subprocess_start( tmp_path, monkeypatch, diff --git a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py index ad207a66d3..7911fdc93e 100644 --- a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py +++ b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py @@ -5,6 +5,7 @@ from __future__ import annotations import argparse +import hashlib import json import re import time @@ -162,6 +163,66 @@ def _neutralize_seed_io(monkeypatch): monkeypatch.setattr(policy, "research_lane_ceiling", lambda: 1) +@pytest.mark.parametrize("ambient_agentx", [None, "1"]) +def test_seed_snapshots_agentx_source_yaml_and_runtime_pins( + tmp_path: Path, + monkeypatch, + ambient_agentx, +) -> None: + _neutralize_seed_io(monkeypatch) + inferencex = tmp_path / "InferenceX" + inferencex.mkdir() + source = tmp_path / "operator-agentx.yaml" + source_text = "benchmark:\n agentx: enable\n framework: sglang\n" + source.write_text(source_text, encoding="utf-8") + pins = { + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm.sh", + "INFERENCEX_PATH": str(inferencex), + "HYPERLOOM_IMAGE": "rocm/agentx:accepted", + "AGENTX_MODE": "canonical", + "AGENTX_RECIPE": "glm5-agentic", + "AGENTX_CONFIG_FILE": "configs/amd-master.yaml", + "AGENTX_FAILED_REQUEST_THRESHOLD": "0.1", + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": "a" * 64, + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": "b" * 64, + "HYPERLOOM_AGENTX_GPU_COUNT": "4", + "MAGPIE_REF": "c" * 40, + "INFERENCEX_REF": "d" * 40, + } + if ambient_agentx is None: + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + else: + monkeypatch.setenv("HYPERLOOM_AGENTX", ambient_agentx) + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + monkeypatch.setenv( + "HYPERLOOM_BENCHMARK_CONFIG_SHA256", + hashlib.sha256(source_text.encode("utf-8")).hexdigest(), + ) + for name, value in pins.items(): + monkeypatch.setenv(name, value) + + state = cb._seed_shared_state( + tmp_path, + _args(enable_conc_sweep=None), + session_id="agentx-session", + ) + + snapshot = tmp_path / "benchmark.source.yaml" + assert snapshot.read_text(encoding="utf-8") == source_text + assert state.benchmark_mode == "agentx" + assert state.agentx_epoch == cb.AGENTX_MEASUREMENT_EPOCH + assert state.conc_sweep_enabled is False + assert state.agentx_runtime_pins == pins + assert state.benchmark_source_config_path == str(snapshot) + assert state.active_inferencex_path == str(inferencex.resolve()) + assert state.warm_replay_enabled is True + persisted = json.loads((tmp_path / "state.json").read_text(encoding="utf-8")) + assert persisted["agentx_runtime_pins"] == pins + assert persisted["benchmark_source_config_path"] == str(snapshot) + assert persisted["warm_replay_enabled"] is True + + def test_seed_records_the_launch_verdict_for_the_partition_shape( tmp_path: Path, monkeypatch, @@ -667,3 +728,26 @@ def test_resolve_reference_recipe_branches(tmp_path: Path, monkeypatch) -> None: with pytest.raises(SystemExit) as exc_info: cb._resolve_reference_recipe(_args(reference_script="empty.sh")) assert exc_info.value.code == 2 + + +@pytest.mark.parametrize("enable_conc_sweep, expected", [(None, False), (False, False)]) +@pytest.mark.parametrize("client", ["", "aiperf", "mlperf"]) +def test_fresh_agentx_seed_preserves_requested_backend(tmp_path, monkeypatch, enable_conc_sweep, expected, client): + _neutralize_seed_io(monkeypatch) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", "irrelevant-ambient-native-pin.sh") + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", client) + state = cb._seed_shared_state( + tmp_path, _args(enable_conc_sweep=enable_conc_sweep, conc_sweep_concs=None), session_id="fresh-agentx" + ) + assert state.benchmark_mode == "agentx" + assert state.agentx_epoch == (1 if client == "mlperf" else 4) + assert state.agentx_backend == ("mlperf" if client == "mlperf" else "native") + restored = SharedState.load_or_init(tmp_path) + assert restored.agentx_epoch == state.agentx_epoch + assert restored.agentx_backend == state.agentx_backend + assert state.benchmark_source_config_path == "" + assert state.warm_replay_enabled is True + assert state.conc_sweep_enabled is expected + assert state.conc_sweep_concs == [1, 4, 8, 10, 14, 20, 28] diff --git a/src/hyperloom/inference_optimizer/tests/test_cli_conc_sweep_default.py b/src/hyperloom/inference_optimizer/tests/test_cli_conc_sweep_default.py index 8c179b4ff3..4a578b7878 100644 --- a/src/hyperloom/inference_optimizer/tests/test_cli_conc_sweep_default.py +++ b/src/hyperloom/inference_optimizer/tests/test_cli_conc_sweep_default.py @@ -149,7 +149,8 @@ def test_synthetic_sweeps_by_default(self, tmp_path, monkeypatch): def test_agentx_does_not_sweep_by_default(self, tmp_path, monkeypatch): assert self._seeded(tmp_path, monkeypatch, agentx=True).conc_sweep_enabled is False - def test_agentx_sweeps_when_asked(self, tmp_path, monkeypatch): + def test_mlperf_agentx_sweeps_when_asked(self, tmp_path, monkeypatch): + monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf") state = self._seeded(tmp_path, monkeypatch, agentx=True, extra=("--enable-conc-sweep",)) assert state.conc_sweep_enabled is True diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py index 90c05eb1d8..ec562ac830 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py @@ -1635,12 +1635,18 @@ async def integrate(payload, *, session_dir): self._assert_measurement(coord, expected) @pytest.fixture - def paired_handler(self, monkeypatch): + def paired_handler(self, coord, monkeypatch): from hyperloom.orchestrator.actions.executors import _multi_node_env, benchmark_backend def unexpected(*args, **kwargs): raise AssertionError("Paired measurements must not resolve, apply, or grade a patch") + # This exercises generic kernel P50 grading; native AgentX launchers + # cannot apply kernel mutations and are covered by the refusal tests. + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) + monkeypatch.setenv("HYPERLOOM_PERF_METRIC", "intvty_v1") + coord.shared_state.benchmark_mode = "synthetic" + coord.shared_state.save(coord.session_dir) monkeypatch.setenv("FRAMEWORK", "sglang") monkeypatch.setattr(krh_mod, "_resolve_integrate_payload", unexpected) monkeypatch.setattr( diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_runtime.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_runtime.py index 3047d4faa3..d70c82d412 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_runtime.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_runtime.py @@ -2700,3 +2700,40 @@ async def _ladder_already_settled(**_kwargs): assert ran == ["False"], f"post-opt roofline saw closing_phase={ran}" finally: await c.stop() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "status, ready", [("succeeded", True), ("failed", True), ("succeeded", False), ("succeeded", None)] +) +async def test_diagnostic_profile_dispatch_preserves_trace_without_promoting_performance(session_dir, status, ready): + c = Coordinator(session_dir, backends=_silent_backends()) + _mute_action_scoring(c) + trace = session_dir / "diagnostic.trace.json" + trace.write_text('{"traceEvents": []}') + result = { + "status": status, + "diagnostic_only": True, + "trace_input_ready": ready, + "main_trace_path": str(trace), + "valid_measurement": False, + "submission_valid": False, + } + c.sub.register_executor("profile", lambda ctx: _async_return(result)) + c.shared_state.baseline_tput = 100.0 + c.shared_state.current_best = {"action": "baseline", "tput": 100.0} + c.shared_state.current_best_measurement = {"output_throughput": 100.0, "completed_requests": 10} + try: + task = await c.tasks.create(kind="profile", params={"reason": "prelude_initial"}, idempotency_key="diagnostic") + c.shared_state.auto_roofline_pending_task_id = task.task_id + await pump_until_settled(c.dispatcher) + accepted = status == "succeeded" and ready is True + assert c.shared_state.last_profile["status"] == ("succeeded" if accepted else "failed") + assert c.shared_state.last_profile_trace == (str(trace) if accepted else "") + assert c.shared_state.auto_roofline_pending_task_id == "" + assert c.shared_state.baseline_tput == 100.0 + assert c.shared_state.current_best == {"action": "baseline", "tput": 100.0} + assert c.shared_state.current_best_measurement == {"output_throughput": 100.0, "completed_requests": 10} + assert not c.writeback.is_promotable_result("baseline", result) + finally: + await c.stop() diff --git a/src/hyperloom/inference_optimizer/tests/test_env_safety.py b/src/hyperloom/inference_optimizer/tests/test_env_safety.py index 1249306c25..65fabbbd46 100644 --- a/src/hyperloom/inference_optimizer/tests/test_env_safety.py +++ b/src/hyperloom/inference_optimizer/tests/test_env_safety.py @@ -230,6 +230,17 @@ def test_variant_env_key_allows_workload_pins_and_blocks_hijacks(): assert not common_env_safety.is_allowed_variant_env_key(hijack) for secret in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "LLM_GATEWAY_KEY"): assert not common_env_safety.is_allowed_variant_env_key(secret) + for agentx_identity in ( + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_RECIPE", + "AGENTX_SELECTOR", + "AGENTX_MODE", + "AGENTX_FAILED_REQUEST_THRESHOLD", + "AGENTX_DATASET", + "WEKA_LOADER_OVERRIDE", + ): + assert not common_env_safety.is_allowed_variant_env_key(agentx_identity) assert not common_env_safety.is_allowed_variant_env_key("bad key") assert not common_env_safety.is_allowed_variant_env_key("") diff --git a/src/hyperloom/inference_optimizer/tests/test_eval_result_dir_wiring.py b/src/hyperloom/inference_optimizer/tests/test_eval_result_dir_wiring.py index e343c4d283..019dfc35c4 100644 --- a/src/hyperloom/inference_optimizer/tests/test_eval_result_dir_wiring.py +++ b/src/hyperloom/inference_optimizer/tests/test_eval_result_dir_wiring.py @@ -456,7 +456,17 @@ def fake_run(cmd, *args, **kwargs): ctx.task.kind = "baseline" if state_in_context: ctx.extra["shared_state"] = state - with patch("hyperloom.orchestrator.actions.executors.baseline.run_with_session_kill", side_effect=fake_run): + # This test isolates the post-run accuracy mapping. Native AgentX recipe + # materialization is covered by its own resolver/protocol suites and needs + # a pinned InferenceX checkout, so keep this fixture at the already- + # materialized config boundary. + with ( + patch( + "hyperloom.orchestrator.actions.executors.baseline.materialize_config_with_envs", + return_value=base, + ), + patch("hyperloom.orchestrator.actions.executors.baseline.run_with_session_kill", side_effect=fake_run), + ): result = asyncio.run(executor(ctx)) assert result["status"] == "succeeded" diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_handoff_launch_server_script.py b/src/hyperloom/inference_optimizer/tests/test_geak_handoff_launch_server_script.py index 3aded00f0b..46132b8224 100644 --- a/src/hyperloom/inference_optimizer/tests/test_geak_handoff_launch_server_script.py +++ b/src/hyperloom/inference_optimizer/tests/test_geak_handoff_launch_server_script.py @@ -42,7 +42,9 @@ def _checkout(tmp_path: Path, *scripts: str, lib: bool = True) -> Path: if lib: (benchmarks / "benchmark_lib.sh").write_text("# stub\n", encoding="utf-8") for name in scripts: - (benchmarks / name).write_text("# stub\n", encoding="utf-8") + script = benchmarks / name + script.parent.mkdir(parents=True, exist_ok=True) + script.write_text("# stub\n", encoding="utf-8") return benchmarks.parent @@ -55,19 +57,16 @@ def _recipe( runner_type: str = "mi355x", envs: dict | None = None, ) -> str: + benchmark = { + "framework": framework, + "runner_type": runner_type, + "envs": envs if envs is not None else {"FRAMEWORK": framework}, + "benchmark_script": benchmark_script, + "inferencex_path": inferencex_path, + } recipe = tmp_path / "baseline_config.with_envs.yaml" recipe.write_text( - yaml.safe_dump( - { - "benchmark": { - "framework": framework, - "runner_type": runner_type, - "envs": envs if envs is not None else {"FRAMEWORK": framework}, - "benchmark_script": benchmark_script, - "inferencex_path": inferencex_path, - } - } - ), + yaml.safe_dump({"benchmark": benchmark}), encoding="utf-8", ) return str(recipe) diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_handoff_workload_contract.py b/src/hyperloom/inference_optimizer/tests/test_geak_handoff_workload_contract.py index a1c6e043c2..95eeeda85f 100644 --- a/src/hyperloom/inference_optimizer/tests/test_geak_handoff_workload_contract.py +++ b/src/hyperloom/inference_optimizer/tests/test_geak_handoff_workload_contract.py @@ -14,9 +14,10 @@ import yaml from hyperloom.orchestrator.loop.coordinator import Coordinator +from hyperloom.orchestrator.phases import machine_state from hyperloom.orchestrator.state.shared_state import SharedState -from ._geak_helpers import stop_geak_before_launch +from ._geak_helpers import forbid_geak_launch, stop_geak_before_launch @pytest.fixture(autouse=True) @@ -258,3 +259,39 @@ async def test_synthetic_handoff_keeps_existing_protocol_and_metric_policy( "num_warmups": 3, "seed": 41, } + + +@pytest.mark.parametrize("framework", ["sglang", "vllm"]) +@pytest.mark.asyncio +async def test_native_agentx_skips_geak_before_writing_a_handoff( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, framework: str +) -> None: + coord = _coord(tmp_path, framework=framework) + coord.shared_state.phase = machine_state.PHASE_KERNEL_AGENT + recipe_path = Path(coord.shared_state.baseline_config_path) + config = yaml.safe_load(recipe_path.read_text(encoding="utf-8")) + config["benchmark"]["agentx"] = "enable" + config["benchmark"]["benchmark_script"] = "single_node/agentic/glm.sh" + recipe_path.write_text(yaml.safe_dump(config), encoding="utf-8") + monkeypatch.setenv("FRAMEWORK", "vllm" if framework == "sglang" else "sglang") + monkeypatch.setenv("MODEL_PATH", "/models/wrong") + monkeypatch.setenv("GPU_TYPE", "mi300x") + monkeypatch.setenv("TP", "8") + monkeypatch.setenv("CONC", "99") + + forbid_geak_launch(monkeypatch) + await coord.phase_kernel._run_geak_kernel_phase(from_phase="KERNEL") + + assert not (tmp_path / "geak" / "handoff.json").exists() + assert coord.shared_state.geak_result == { + "status": "skipped", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "native AgentX kernel optimization is unavailable until " + "InferenceX exposes a fingerprinted optimizer-argv hook" + ), + } + assert coord.shared_state.pending_escalate_hint == "" + transition = machine_state.compute_next_phase(coord.shared_state) + assert transition is not None + assert transition[:2] == (machine_state.PHASE_SWEEP, "kernel_no_more_leverage") diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_native_source_revalidation.py b/src/hyperloom/inference_optimizer/tests/test_geak_native_source_revalidation.py new file mode 100644 index 0000000000..4c3fef4d58 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_geak_native_source_revalidation.py @@ -0,0 +1,299 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""GEAK source products require real review and canonical transactional measurement.""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from hyperloom.orchestrator.phases.geak_native_revalidation import ORIGIN +from .test_geak_revalidation_dispatch import coordinator as coordinator + + +def _candidate(c, tmp_path): + state = c.shared_state + state.benchmark_mode = "agentx" + state.agentx_epoch = 3 + state.agentx_backend = "native" + state.baseline_config_path = str(tmp_path / "native.yaml") + state.baseline_tput = 100.0 + state.current_best = {"tput": 110.0, "extra_server_args": "--incumbent", "extra_envs": {"SGLANG_USE_AITER": "0"}} + directory = tmp_path / "geak-eval" + directory.mkdir() + patch = directory / "final.patch" + patch.write_text("diff --git a/kernel.py b/kernel.py\n--- a/kernel.py\n+++ b/kernel.py\n@@ -1 +1 @@\n-old\n+new\n") + state.geak_result = { + "status": "ok", + "eval_dir": str(directory), + "final_patch": str(patch), + "final_throughput_tok_s": 99999.0, + "accepted_config": {"flags": "--candidate", "env_map": {"SGLANG_USE_AITER": "1"}}, + "accepted_kernels": [{"short_name": "kernel", "e2e_delta_pct": 10, "final_patch": str(patch)}], + } + return patch + + +async def _actual_verdict(c, verdict): + async def ready(): + while not c.state.pending_proposals: + await asyncio.sleep(0.01) + + await asyncio.wait_for(ready(), 2.0) + pending = next(iter(c.state.pending_proposals.values())) + assert pending.action_name == "integrate" + assert pending.payload["params"]["covered_executor"] == "integrate_patch" + await c.router._handle_single_verdict( + source="critic", pending=pending, verdict=verdict, reasoning="reviewed the staged patch" + ) + + +def test_source_route_retains_configuration_and_stages_exact_patches(coordinator, tmp_path): + patch = _candidate(coordinator, tmp_path) + params = coordinator.writeback.geak_rebench_params(reason="unit") + assert params["origin"] == ORIGIN + assert params["config_path"] == coordinator.shared_state.baseline_config_path + assert params["extra_server_args"] == "--candidate" + assert params["extra_envs"] == {"SGLANG_USE_AITER": "1"} + assert "--incumbent" in params["base_extra_args"] + assert Path(params["patches"][0]).read_bytes() == patch.read_bytes() + assert "final_throughput_tok_s" not in params + + +@pytest.mark.asyncio +@pytest.mark.parametrize("verdict", ["reject", "needs_review"]) +async def test_declined_review_never_runs_or_promotes_proxy(coordinator, tmp_path, monkeypatch, verdict): + _candidate(coordinator, tmp_path) + before = dict(coordinator.shared_state.current_best) + + async def forbidden(task): + raise AssertionError("unapproved source integration must not execute") + + monkeypatch.setattr(coordinator.sub, "execute_covered", forbidden) + run = asyncio.create_task(coordinator.phase_kernel._revalidate_geak_candidate(reason="unit")) + await _actual_verdict(coordinator, verdict) + await asyncio.wait_for(run, 2.0) + assert coordinator.shared_state.geak_pending == {} + assert coordinator.shared_state.current_best == before + assert coordinator.shared_state.geak_result["revalidation_status"] == "no_promote" + assert not coordinator.shared_state.optimization_stack + + +@pytest.mark.asyncio +async def test_approved_source_uses_covered_integrator_and_canonical_measurement(coordinator, tmp_path, monkeypatch): + _candidate(coordinator, tmp_path) + executed = [] + adopted = [] + canonical = {"output_throughput": 120.0, "agentx_epoch": 3} + + async def execute(task): + assert coordinator.shared_state.get_specialist_patch_verdict(task.params["specialist_task_id"]) == "approve" + executed.append(task) + return {"status": "kept", "bench_result": canonical} + + async def promote(kind, result, *, task): + assert kind == "integrate_patch" + assert result["bench_result"] == canonical + coordinator.shared_state.optimization_stack.append({"task_id": task.task_id, "action": kind}) + + monkeypatch.setattr(coordinator.sub, "execute_covered", execute) + monkeypatch.setattr(coordinator.writeback, "promote_to_shared_state", promote) + monkeypatch.setattr( + coordinator.phase_kernel, "_record_geak_adopted_kernels", lambda *args, **kwargs: adopted.append(kwargs) + ) + monkeypatch.setattr( + "hyperloom.orchestrator.state.shared_state.resolve_graded_comparison", + lambda *args, **kwargs: SimpleNamespace(comparable=True, candidate=120.0, reference=100.0), + ) + run = asyncio.create_task(coordinator.phase_kernel._revalidate_geak_candidate(reason="unit")) + await _actual_verdict(coordinator, "approve") + await asyncio.wait_for(run, 2.0) + assert len(executed) == 1 + assert executed[0].kind == "integrate_patch" + assert adopted[0]["measured_tput"] == 120.0 + assert adopted[0]["source_applied"] is True + assert adopted[0]["overlay_loaded"] is False + assert coordinator.shared_state.geak_result["canonical_revalidation"] == canonical + assert coordinator.shared_state.geak_pending == {} + + +@pytest.mark.asyncio +async def test_cancelled_covered_run_reuses_review_and_task_identity(coordinator, tmp_path, monkeypatch): + _candidate(coordinator, tmp_path) + tasks = [] + + async def execute(task): + tasks.append(task) + if len(tasks) == 1: + raise asyncio.CancelledError() + return {"status": "reverted", "reason": "canonical replay did not improve"} + + async def promote(*args, **kwargs): + pass + + monkeypatch.setattr(coordinator.sub, "execute_covered", execute) + monkeypatch.setattr(coordinator.writeback, "promote_to_shared_state", promote) + run = asyncio.create_task(coordinator.phase_kernel._revalidate_geak_candidate(reason="unit")) + await _actual_verdict(coordinator, "approve") + with pytest.raises(asyncio.CancelledError): + await run + saved_review = dict(coordinator.shared_state.geak_pending["native_review"]) + await coordinator.phase_kernel._revalidate_geak_candidate(reason="resume") + assert len(tasks) == 2 + assert tasks[0].task_id == tasks[1].task_id == saved_review["task_id"] + proposals = await coordinator.bus.tail(topic="proposal") + assert [proposal.msg_id for proposal in proposals] == [saved_review["proposal_msg_id"]] + assert not coordinator.state.pending_proposals + assert coordinator.shared_state.geak_pending == {} + assert not coordinator.shared_state.optimization_stack + + +@pytest.mark.asyncio +async def test_missing_declared_patch_fails_without_proxy_harness(coordinator, tmp_path, monkeypatch): + patch = _candidate(coordinator, tmp_path) + patch.unlink() + + async def forbidden(**kwargs): + raise AssertionError("native source failure must not use GEAK proxy harness") + + monkeypatch.setattr(coordinator.phase_kernel, "_revalidate_on_geak_harness", forbidden) + await coordinator.phase_kernel._revalidate_geak_candidate(reason="unit") + assert coordinator.shared_state.geak_pending == {} + assert coordinator.shared_state.geak_result["revalidation_status"] == "fallback_failed" + assert "source patch" in coordinator.shared_state.geak_result["revalidation_error"] + + +@pytest.mark.parametrize("missing", ["combined", "all"]) +def test_source_artifacts_are_required_even_with_accepted_config(coordinator, tmp_path, missing): + patch = _candidate(coordinator, tmp_path) + coordinator.shared_state.geak_result.pop("final_patch") + if missing == "all": + coordinator.shared_state.geak_result["accepted_kernels"][0].pop("final_patch") + params = coordinator.writeback.geak_rebench_params(reason="unit") + assert params["reason"] == "geak_native_source_invalid" + else: + params = coordinator.writeback.geak_rebench_params(reason="unit") + assert Path(params["patches"][0]).read_bytes() == patch.read_bytes() + + +@pytest.mark.asyncio +async def test_reviewed_patch_mutation_is_rejected_before_execution(coordinator, tmp_path, monkeypatch): + _candidate(coordinator, tmp_path) + from hyperloom.orchestrator.phases import geak_native_revalidation as native + + review = native._review + + async def mutate_after_review(c, params): + outcome = await review(c, params) + Path(params["patches"][0]).write_text("unreviewed patch contents") + return outcome + + async def forbidden(task): + raise AssertionError("changed source must not execute") + + monkeypatch.setattr(native, "_review", mutate_after_review) + monkeypatch.setattr(coordinator.sub, "execute_covered", forbidden) + run = asyncio.create_task(coordinator.phase_kernel._revalidate_geak_candidate(reason="unit")) + await _actual_verdict(coordinator, "approve") + await asyncio.wait_for(run, 2.0) + assert coordinator.shared_state.geak_pending == {} + assert coordinator.shared_state.geak_result["revalidation_status"] == "fallback_failed" + assert "changed or disappeared" in coordinator.shared_state.geak_result["revalidation_error"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("change", ["sitecustomize", "imported_module", "new_file", "deleted_module", "unchanged"]) +async def test_source_review_binds_entire_overlay_without_manifest(coordinator, tmp_path, monkeypatch, change): + _candidate(coordinator, tmp_path) + from hyperloom.orchestrator.phases import geak_native_revalidation as native + + overlay = tmp_path / "overlay" + overlay.mkdir() + (overlay / "sitecustomize.py").write_text("import kernel_impl\n") + module = overlay / "kernel_impl.py" + module.write_text("VALUE = 1\n") + coordinator.shared_state.geak_result["final_overlay"] = str(overlay) + executed = [] + review = native._review + + async def mutate_after_review(c, params): + outcome = await review(c, params) + assert set(params["native_geak_overlay_files"]) == {str(overlay / "sitecustomize.py"), str(module)} + if change == "sitecustomize": + (overlay / "sitecustomize.py").write_text("import unreviewed_impl\n") + elif change == "imported_module": + module.write_text("VALUE = 2\n") + elif change == "new_file": + (overlay / "unreviewed_impl.py").write_text("VALUE = 3\n") + elif change == "deleted_module": + module.unlink() + else: + cache = overlay / "__pycache__" + cache.mkdir() + (cache / "kernel_impl.cpython-311.pyc").write_bytes(b"interpreter cache") + return outcome + + async def execute(task): + executed.append(task) + return {"status": "reverted", "reason": "canonical candidate did not improve"} + + monkeypatch.setattr(native, "_review", mutate_after_review) + monkeypatch.setattr(coordinator.sub, "execute_covered", execute) + run = asyncio.create_task(coordinator.phase_kernel._revalidate_geak_candidate(reason="unit")) + await _actual_verdict(coordinator, "approve") + await asyncio.wait_for(run, 2.0) + assert coordinator.shared_state.geak_pending == {} + if change == "unchanged": + assert len(executed) == 1 + else: + assert executed == [] + assert coordinator.shared_state.geak_result["revalidation_status"] == "fallback_failed" + assert "reviewed GEAK overlay changed" in coordinator.shared_state.geak_result["revalidation_error"] + + +@pytest.mark.asyncio +async def test_expired_review_budget_clears_pending_without_promotion(coordinator, tmp_path, monkeypatch): + _candidate(coordinator, tmp_path) + monkeypatch.setattr(coordinator.shared_state, "remaining_minutes", lambda: 0.0) + await coordinator.phase_kernel._revalidate_geak_candidate(reason="unit") + assert coordinator.shared_state.geak_pending == {} + assert coordinator.shared_state.geak_result["native_source_revalidation"]["verdict"] == "timeout" + assert not coordinator.shared_state.optimization_stack + + +def test_native_review_recovery_retains_identity_only_for_same_candidate(coordinator, tmp_path): + _candidate(coordinator, tmp_path) + result = coordinator.shared_state.geak_result + saved = {"artifact_digest": "reviewed", "candidate": dict(result), "verdict": "approve", "task_id": "covered-task"} + coordinator.shared_state.geak_pending = { + "native_review": saved, + "status": "source_rebench_running", + "revalidation_task_id": "covered-task", + } + coordinator.phase_kernel._record_geak_candidate(result) + assert coordinator.shared_state.geak_pending["native_review"] == saved + assert coordinator.shared_state.geak_pending["revalidation_task_id"] == "covered-task" + changed = {**result, "accepted_config": {"flags": "--different"}} + coordinator.phase_kernel._record_geak_candidate(changed) + assert "native_review" not in coordinator.shared_state.geak_pending + + +def test_adjudicated_source_result_reopens_only_when_artifact_changes(coordinator, tmp_path): + from hyperloom.orchestrator.phases.geak_rebench import geak_candidate_is_adjudicated + + patch = _candidate(coordinator, tmp_path) + raw = dict(coordinator.shared_state.geak_result) + params = coordinator.writeback.geak_rebench_params(reason="unit") + settled = { + **raw, + "revalidation_status": "validated", + "native_source_revalidation": {"params": params, "status": "validated"}, + "canonical_revalidation": {"output_throughput": 120.0}, + } + assert geak_candidate_is_adjudicated(settled, raw, harness_can_replay=False) + patch.write_text(patch.read_text().replace("+new", "+changed")) + assert not geak_candidate_is_adjudicated(settled, raw, harness_can_replay=False) diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_revalidation_dispatch.py b/src/hyperloom/inference_optimizer/tests/test_geak_revalidation_dispatch.py index 4da1e77718..fbe4aab2d3 100644 --- a/src/hyperloom/inference_optimizer/tests/test_geak_revalidation_dispatch.py +++ b/src/hyperloom/inference_optimizer/tests/test_geak_revalidation_dispatch.py @@ -1338,3 +1338,75 @@ async def test_crash_recovery_retries_a_transiently_failed_revalidation(coordina assert revalidations == ["geak_e2e_win_recovered"] assert st.geak_pending["status"] == "awaiting_rebench" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("overlay_state", ["missing", "config_only"]) +async def test_native_agentx_refuses_geak_products_regardless_of_overlay_state( + coordinator, tmp_path, monkeypatch, overlay_state +) -> None: + c = coordinator + st = c.shared_state + _arm_kernel_to_sweep(st) + st.benchmark_mode = "agentx" + st.agentx_epoch = 2 + st.baseline_tput = 100.0 + st.baseline_config_path = "/run/canonical-agentx.yaml" + st.current_best = {"action": "explore", "tput": 120.0} + st.resume_pending_revalidation = True + overlay = tmp_path / "candidate" / "overlay" + if overlay_state == "config_only": + overlay.mkdir(parents=True) + (overlay / "sitecustomize.py").write_text("pass\n", encoding="utf-8") + (overlay / "_overlay_manifest.json").write_text('{"modules": [], "rebinds": []}', encoding="utf-8") + candidate = { + "status": "ok", + "accepted_config": {}, + "final_overlay": str(overlay.parent), + "final_throughput_tok_s": 130.0, + } + geak_dir = c.session_dir / "geak" + geak_dir.mkdir() + result_path = geak_dir / "result.json" + result_path.write_text(json.dumps(candidate), encoding="utf-8") + original_dispatch = c.writeback.geak_rebench_params + original_fallback = c.writeback.validate_geak_via_geak_harness + dispatches = [] + fallbacks = [] + + def dispatch(**kwargs): + dispatches.append(kwargs) + return original_dispatch(**kwargs) + + async def fallback(**kwargs): + fallbacks.append(kwargs) + return await original_fallback(**kwargs) + + monkeypatch.setattr(c.writeback, "geak_rebench_params", dispatch) + monkeypatch.setattr(c.writeback, "validate_geak_via_geak_harness", fallback) + monkeypatch.setattr(c.phase_kernel, "_record_geak_kernel_journey", lambda _result: None) + forbid_geak_launch(monkeypatch) + + await c.phase_kernel._run_geak_kernel_phase(from_phase="KERNEL") + assert dispatches == fallbacks == [] + assert st.geak_result["error_class"] == "unsupported_upstream_launcher_hook" + assert not await c.tasks.queued() + # Exercise the persisted verdict, not an in-memory object identity. + st.geak_result = json.loads(json.dumps(st.geak_result)) + for _ in range(2): + st.macro_cycle += 1 + await c.phase_kernel._run_geak_kernel_phase(from_phase="KERNEL") + assert dispatches == fallbacks == [] + assert st.resume_pending_revalidation is True + assert st.current_best["tput"] == 120.0 + assert not st.optimization_stack + + # The result JSON is unchanged; only the real overlay becomes loadable. + overlay.mkdir(parents=True, exist_ok=True) + (overlay / "sitecustomize.py").write_text("pass\n", encoding="utf-8") + (overlay / "_overlay_manifest.json").write_text('{"modules": ["kernel.py"]}', encoding="utf-8") + st.macro_cycle += 1 + await c.phase_kernel._run_geak_kernel_phase(from_phase="KERNEL") + assert not await c.tasks.queued() + assert dispatches == fallbacks == [] + assert json.loads(result_path.read_text(encoding="utf-8")) == candidate diff --git a/src/hyperloom/inference_optimizer/tests/test_graded_comparison.py b/src/hyperloom/inference_optimizer/tests/test_graded_comparison.py index fe2be40d10..b7f9486813 100644 --- a/src/hyperloom/inference_optimizer/tests/test_graded_comparison.py +++ b/src/hyperloom/inference_optimizer/tests/test_graded_comparison.py @@ -118,6 +118,23 @@ def test_agentx_keep_verdict_when_intvty_clears_threshold(monkeypatch): assert graded.verdict == VERDICT_KEEP +def test_agentx_output_guard_does_not_grade_total_throughput_with_native_topology(monkeypatch): + _agentx(monkeypatch) + anchor = {**_ANCHOR, "total_throughput": 800.0, "agentx_gpu_count": 8} + candidate = _full_measurement(total=600.0, output=200.0, intvty=24.8) + candidate["agentx_gpu_count"] = 8 + + graded = resolve_graded_comparison( + _State(current_best=anchor, baseline_tput=180.0), + candidate, + keep_threshold_pct=2.0, + ) + + assert graded.verdict == VERDICT_KEEP + assert graded.tput_reference == pytest.approx(800.0) + assert graded.tput_candidate == pytest.approx(600.0) + + def test_agentx_revert_when_both_axes_worse(monkeypatch): """Both interactivity AND tput regressed beyond the noise band -> REVERT.""" _agentx(monkeypatch) diff --git a/src/hyperloom/inference_optimizer/tests/test_grid_runner.py b/src/hyperloom/inference_optimizer/tests/test_grid_runner.py index 72d3e57596..8c47c00cfc 100644 --- a/src/hyperloom/inference_optimizer/tests/test_grid_runner.py +++ b/src/hyperloom/inference_optimizer/tests/test_grid_runner.py @@ -324,8 +324,26 @@ def test_preserves_measurement_fields(self): expected = asdict(result) expected["e2e_norm_intvty_p90"] = expected.pop("intvty_p90") expected["e2e_norm_intvty_p50"] = expected.pop("intvty_p50") + expected.pop("materialized_config") + expected.pop("native_measurement") assert encoded == expected + def test_preserves_native_identity_with_actual_candidate_config(self): + result = VariantResult( + name="native", + extra_server_args="", + extra_envs={}, + status="succeeded", + output_throughput=286.0, + materialized_config="/runs/candidate.yaml", + native_measurement={"agentx_workload_fingerprint": "a" * 64, "agentx_launch_contract": 1}, + ) + encoded = result.to_dict() + assert encoded["output_throughput"] == 286.0 + assert encoded["materialized_config"] == "/runs/candidate.yaml" + assert encoded["agentx_workload_fingerprint"] == "a" * 64 + assert "native_measurement" not in encoded + def test_preserves_unmeasured_axes(self): result = VariantResult(name="legacy", extra_server_args="", extra_envs={}, status="failed") encoded = result.to_dict() @@ -1724,9 +1742,12 @@ async def test_without_an_estimate_positive_budget_admits_the_round(self, tmp_pa assert [r.status for r in results] == ["succeeded"] @pytest.mark.asyncio - async def test_without_an_estimate_agentx_uses_the_same_benchmark_policy(self, tmp_path, monkeypatch): - """Legacy AgentX timeout inputs do not turn the hard cap into admission cost.""" - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + async def test_legacy_agentx_timeout_inputs_do_not_change_benchmark_policy(self, tmp_path, monkeypatch): + """Retired AgentX timeout hints do not turn the hard cap into admission cost.""" + # Native AgentX grids are measurement-only/fail-closed. This test is + # about the timeout resolver itself, so leave the native mode off and + # prove the legacy hints cannot affect an ordinary grid launch. + monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) monkeypatch.setenv("AGENTX_DURATION", "3600") monkeypatch.setenv("AGENTX_BASELINE_OVERHEAD_SEC", "7200") monkeypatch.delenv("AGENTX_BASELINE_TIMEOUT_SEC", raising=False) diff --git a/src/hyperloom/inference_optimizer/tests/test_grid_runner_runtime_override.py b/src/hyperloom/inference_optimizer/tests/test_grid_runner_runtime_override.py index f4528aeaae..d4082ea36e 100644 --- a/src/hyperloom/inference_optimizer/tests/test_grid_runner_runtime_override.py +++ b/src/hyperloom/inference_optimizer/tests/test_grid_runner_runtime_override.py @@ -7,6 +7,7 @@ import os import yaml +import pytest from pathlib import Path @@ -75,10 +76,12 @@ def test_empty_override_is_noop(): # Integration test: override lands in materialized YAML, NOT os.environ -def test_runtime_override_in_yaml_not_process_env(tmp_path): +@pytest.mark.parametrize("legacy_agentx", [False, True]) +def test_runtime_override_in_yaml_not_process_env(tmp_path, monkeypatch, legacy_agentx): """The attempt framework_bin must appear in materialized YAML benchmark.envs, and os.environ must be unchanged.""" + monkeypatch.setenv("HYPERLOOM_AGENTX", "1" if legacy_agentx else "0") base_yaml = _base_yaml(tmp_path) - variant = GridVariant(name="test-rt") + variant = GridVariant(name="test-rt", extra_server_args="--max-num-seqs 16", extra_envs={"CONC": "4"}) variant.runtime_override = { "path_prefix": "/attempt/bin", "framework_bin": "/attempt/bin/vllm", @@ -100,7 +103,12 @@ def test_runtime_override_in_yaml_not_process_env(tmp_path): with out_yaml.open(encoding="utf-8") as f: materialized = yaml.safe_load(f) + if legacy_agentx: + assert materialized["benchmark"]["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in materialized["benchmark"] + assert materialized["benchmark"]["workload_spec"]["concurrency"] == 4 envs = materialized["benchmark"]["envs"] + assert "--max-num-seqs 16" in envs["EXTRA_VLLM_ARGS"] assert envs.get("HYPERLOOM_FRAMEWORK_BIN") == "/attempt/bin/vllm" assert envs.get("HYPERLOOM_FRAMEWORK_PYTHON") == "/attempt/bin/python" assert "/attempt/bin" in envs.get("PATH", "") diff --git a/src/hyperloom/inference_optimizer/tests/test_inferencex_anchor_contract.py b/src/hyperloom/inference_optimizer/tests/test_inferencex_anchor_contract.py index ce6954bbc1..4b11d070cd 100644 --- a/src/hyperloom/inference_optimizer/tests/test_inferencex_anchor_contract.py +++ b/src/hyperloom/inference_optimizer/tests/test_inferencex_anchor_contract.py @@ -18,7 +18,10 @@ ) from hyperloom.orchestrator.actions.executors._inferencex_patcher import ( _ANCHOR_CONTRACT, + _ANCHOR_ALTERNATIVES, + _BENCH_SERVING_REL_PARTS, EVAL_PROBE_TARGET_PARTS, + EVAL_PROBE_TARGETS, count_anchor_hits, ) @@ -52,19 +55,13 @@ def _magpie_pattern_parts() -> list[str]: ] -def anchors_by_file() -> dict[str, list[tuple[str, str]]]: - """Group the patch anchors by the upstream file they are matched against.""" - grouped: dict[str, list[tuple[str, str]]] = {} - for name, rel_parts, _sentinel, anchor in _ANCHOR_CONTRACT: - grouped.setdefault("/".join(rel_parts), []).append((name, anchor)) - return grouped - - def anchors_fingerprint() -> str: """Fingerprint the anchor definitions themselves.""" parts = [f"{name}\x1f{'/'.join(rel_parts)}\x1f{anchor}" for name, rel_parts, _sentinel, anchor in _ANCHOR_CONTRACT] parts.append(f"probe_target\x1f{PROBE_TARGET_PATH}") parts.extend(_magpie_pattern_parts()) + parts.append(json.dumps(_ANCHOR_ALTERNATIVES, sort_keys=True)) + parts.append(json.dumps([_BENCH_SERVING_REL_PARTS, EVAL_PROBE_TARGETS])) return hashlib.sha256("\x1e".join(parts).encode("utf-8")).hexdigest() @@ -73,17 +70,22 @@ def github_slug(clone_url: str) -> str: return clone_url.rstrip("/").removesuffix(".git").split("github.com/", 1)[-1] -def fetch_pinned_file(rel_path: str, ref: str) -> str | None: +def fetch_pinned_file(rel_path: str, ref: str, checkout: Path | None = None) -> str | None: """Fetch one upstream file at ``ref``, or ``None`` when unreachable.""" + command = ( + ["git", "-C", str(checkout), "show", f"{ref}:{rel_path}"] + if checkout is not None + else [ + "gh", + "api", + f"repos/{github_slug(_INFERENCEX_REPO_DEFAULT)}/contents/{rel_path}?ref={ref}", + "-H", + "Accept: application/vnd.github.raw", + ] + ) try: proc = subprocess.run( - [ - "gh", - "api", - f"repos/{github_slug(_INFERENCEX_REPO_DEFAULT)}/contents/{rel_path}?ref={ref}", - "-H", - "Accept: application/vnd.github.raw", - ], + command, capture_output=True, timeout=_FETCH_TIMEOUT_SEC, ) @@ -94,52 +96,41 @@ def fetch_pinned_file(rel_path: str, ref: str) -> str | None: return proc.stdout.decode("utf-8", errors="replace") -def build_record(ref: str) -> dict: - """Verify every anchor against upstream at ``ref`` and return the record.""" +def upstream_candidates(paths: tuple[tuple[str, ...], ...]) -> list[str]: + """Resolve project-relative targets in either repository layout.""" + return ["/".join((*prefix, *parts)) for prefix in (("inferencex-e2e",), ()) for parts in paths] + + +def _fetch_target(paths: tuple[tuple[str, ...], ...], ref: str, checkout: Path | None) -> tuple[str, str]: + for path in upstream_candidates(paths): + text = fetch_pinned_file(path, ref, checkout) + if text is not None: + return path, text + raise RuntimeError(f"cannot read any supported path {upstream_candidates(paths)} at {ref}") + + +def build_record(ref: str, checkout: Path | None = None) -> dict: + """Verify every active synthetic patch against immutable upstream objects.""" files: dict[str, dict] = {} - texts: dict[str, str] = {} - for rel_path, anchors in anchors_by_file().items(): - text = fetch_pinned_file(rel_path, ref) - if text is None: - raise RuntimeError(f"cannot fetch {rel_path} at {ref}; is `gh auth status` clean?") - texts[rel_path] = text - hits = {name: count_anchor_hits(text, anchor) for name, anchor in anchors} - broken = {name: n for name, n in hits.items() if n != 1} - if broken: - raise RuntimeError( - f"{rel_path} at {ref}: expected each anchor to match exactly one site, got {broken}. " - "Re-anchor these patches in _inferencex_patcher.py before recording the contract." - ) - files[rel_path] = { - "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), - "anchors": hits, - } - probe_target = fetch_pinned_file(PROBE_TARGET_PATH, ref) - if probe_target is None: - raise RuntimeError( - f"cannot fetch {PROBE_TARGET_PATH} at {ref}. The probe and the request bounds are " - "appended to this file; if upstream moved it, re-home them in _inferencex_patcher.py " - "before recording the contract." - ) - magpie_text = texts.get(MAGPIE_LIB_PATH) or fetch_pinned_file(MAGPIE_LIB_PATH, ref) - if magpie_text is None: - raise RuntimeError(f"cannot fetch {MAGPIE_LIB_PATH} at {ref}; is `gh auth status` clean?") + for name, parts, _sentinel, anchor in _ANCHOR_CONTRACT: + paths = _BENCH_SERVING_REL_PARTS if name == "profile_extra_body" else (parts,) + rel_path, text = _fetch_target(paths, ref, checkout) + hits = count_anchor_hits(text, anchor) + if hits != 1: + raise RuntimeError(f"{rel_path} at {ref}: anchor {name} matched {hits} sites, expected exactly one") + spec = files.setdefault(rel_path, {"sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), "anchors": {}}) + spec["anchors"][name] = hits + probe_path, probe_text = _fetch_target(EVAL_PROBE_TARGETS, ref, checkout) + magpie_path, magpie_text = _fetch_target((("benchmarks", "benchmark_lib.sh"),), ref, checkout) if not magpie_patch_applies(magpie_text): - raise RuntimeError( - f"{MAGPIE_LIB_PATH} at {ref}: the run_lm_eval --concurrent-requests splice no longer " - "finds its site. install.sh die()s when this patch cannot apply, so recording the " - "contract now would ship a broken install. Re-anchor _magpie_patcher.py first." - ) + raise RuntimeError(f"{magpie_path} at {ref}: run_lm_eval --concurrent-requests splice no longer applies") return { "ref": ref, "anchors_fingerprint": anchors_fingerprint(), "refresh_with": REFRESH_CMD, "files": files, - "probe_target": { - "path": PROBE_TARGET_PATH, - "sha256": hashlib.sha256(probe_target.encode("utf-8")).hexdigest(), - }, - "magpie_patch": {"path": MAGPIE_LIB_PATH, "applies": True}, + "probe_target": {"path": probe_path, "sha256": hashlib.sha256(probe_text.encode("utf-8")).hexdigest()}, + "magpie_patch": {"path": magpie_path, "applies": True}, } @@ -195,13 +186,13 @@ def test_record_covers_the_magpie_patch(): assert record.get("magpie_patch", {}).get("applies") is True, ( f"the contract predates the magpie patch check, or the splice no longer applies. {REFRESH_CMD}" ) - assert record["magpie_patch"]["path"] == MAGPIE_LIB_PATH + assert record["magpie_patch"]["path"] in upstream_candidates((("benchmarks", "benchmark_lib.sh"),)) # --- networked -------------------------------------------------------------- -@pytest.mark.parametrize("rel_path", sorted(anchors_by_file())) +@pytest.mark.parametrize("rel_path", sorted(load_record()["files"])) def test_pinned_upstream_still_matches_every_anchor(rel_path): """The layer that actually re-verifies.""" record = load_record() @@ -209,7 +200,8 @@ def test_pinned_upstream_still_matches_every_anchor(rel_path): if text is None: pytest.skip(f"InferenceX@{record['ref'][:9]} unreachable (needs `gh` + repo access)") - hits = {name: count_anchor_hits(text, anchor) for name, anchor in anchors_by_file()[rel_path]} + definitions = {name: anchor for name, _parts, _sentinel, anchor in _ANCHOR_CONTRACT} + hits = {name: count_anchor_hits(text, definitions[name]) for name in record["files"][rel_path]["anchors"]} assert hits == record["files"][rel_path]["anchors"], ( f"upstream {rel_path} no longer matches the recorded anchors. Re-anchor the affected " f"patches in _inferencex_patcher.py, then refresh: {REFRESH_CMD}" @@ -223,7 +215,7 @@ def test_recorded_probe_target_is_the_path_the_patcher_appends_to(): """The probe has no anchor to rot, but it does need this file to exist: if upstream moves it the patch degrades to a warning and the eval runs unbounded again -- the exact failure the probe was written to stop.""" record = load_record() - assert record["probe_target"]["path"] == PROBE_TARGET_PATH, ( + assert record["probe_target"]["path"] in upstream_candidates(EVAL_PROBE_TARGETS), ( f"the probe target moved to {PROBE_TARGET_PATH}. Re-verify and refresh: {REFRESH_CMD}" ) @@ -231,12 +223,12 @@ def test_recorded_probe_target_is_the_path_the_patcher_appends_to(): def test_pinned_upstream_still_carries_the_probe_target(): """Networked counterpart: confirm the file is really there at the pin.""" record = load_record() - text = fetch_pinned_file(PROBE_TARGET_PATH, record["ref"]) + text = fetch_pinned_file(record["probe_target"]["path"], record["ref"]) if text is None: pytest.skip(f"InferenceX@{record['ref'][:9]} unreachable (needs `gh` + repo access)") assert hashlib.sha256(text.encode("utf-8")).hexdigest() == record["probe_target"]["sha256"], ( - f"{PROBE_TARGET_PATH} changed upstream. The probe and the bounds are appended to it, so " + f"{record['probe_target']['path']} changed upstream. The probe and the bounds are appended to it, so " f"re-read it before refreshing: {REFRESH_CMD}" ) @@ -244,7 +236,7 @@ def test_pinned_upstream_still_carries_the_probe_target(): def test_pinned_upstream_still_takes_the_magpie_splice(): """Re-verify the splice against upstream, not just against the record.""" record = load_record() - text = fetch_pinned_file(MAGPIE_LIB_PATH, record["ref"]) + text = fetch_pinned_file(record["magpie_patch"]["path"], record["ref"]) if text is None: pytest.skip(f"InferenceX@{record['ref'][:9]} unreachable (needs `gh` + repo access)") diff --git a/src/hyperloom/inference_optimizer/tests/test_inferencex_modern_contract.py b/src/hyperloom/inference_optimizer/tests/test_inferencex_modern_contract.py new file mode 100644 index 0000000000..11cd41f0d5 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_inferencex_modern_contract.py @@ -0,0 +1,166 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Synthetic profiling and eval patches work with the pinned infx project layout.""" + +from __future__ import annotations + +import subprocess + +import pytest + +from hyperloom.inference_optimizer.tests import test_inferencex_anchor_contract as contract +from hyperloom.orchestrator.actions.executors import _inferencex_patcher as patcher + + +_PROFILE_CLIENT = """def profile_request(): + return dict( + extra_body={ + "num_steps": 1, + "merge_profiles": True, + "profile_by_stage": True, + }, + ) +""" +_EVAL_LIBRARY = """stage_result() { + local stem=results extension=.json eval_conc=8 target suffix + target="./${stem}_conc${eval_conc}${extension}" + suffix=2 + while [ -e "$target" ]; do + target="./${stem}_conc${eval_conc}_${suffix}${extension}" + suffix=$((suffix + 1)) + done + printf 'result' > "$target" +} +clean_eval() { + local results_dir=cleanup + export EVAL_RESULT_DIR="$results_dir" +} +run_eval() { + local results_dir=accuracy + # Read by append_lm_eval_summary. + export EVAL_RESULT_DIR="$results_dir" +} +run_benchmark() { + local max_concurrency=8 num_prompts + num_prompts="$max_concurrency" + printf '%s' "$num_prompts" +} +""" + + +def _modern_tree(tmp_path, nested): + project = tmp_path / "inferencex-e2e" if nested else tmp_path + paths = { + "benchmarks/benchmark_lib.sh": _EVAL_LIBRARY, + "infx/bench_serving/benchmark_serving.py": _PROFILE_CLIENT, + "infx/evals/patches/lm_eval_sitecustomize.py": "# upstream eval hooks\n", + } + for name, content in paths.items(): + path = project / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + return project + + +@pytest.mark.parametrize("nested", [False, True]) +def test_modern_profile_settings_reach_request_body_and_keep_legacy_defaults(tmp_path, monkeypatch, nested): + project = _modern_tree(tmp_path, nested) + client = project / "infx/bench_serving/benchmark_serving.py" + assert patcher.benchmark_serving_path_in(tmp_path) == client + assert patcher.ensure_benchmark_serving_patched(tmp_path) + once = client.read_text() + assert patcher.ensure_benchmark_serving_patched(project) + assert client.read_text() == once + namespace = {} + exec(compile(once, str(client), "exec"), namespace) + monkeypatch.delenv("PROFILE_EXTRA_BODY", raising=False) + assert namespace["profile_request"]()["extra_body"] == { + "num_steps": 1, + "merge_profiles": True, + "profile_by_stage": True, + } + monkeypatch.setenv("PROFILE_EXTRA_BODY", '{"num_steps": 20, "start_step": 5}') + assert namespace["profile_request"]()["extra_body"] == {"num_steps": 20, "start_step": 5} + + +@pytest.mark.parametrize("nested", [False, True]) +def test_modern_eval_artifacts_and_collision_suffixes_stay_in_result_dir(tmp_path, nested): + project = _modern_tree(tmp_path, nested) + assert patcher.ensure_benchmark_lib_eval_dest_patched(tmp_path) + assert patcher.ensure_benchmark_lib_eval_start_patched(tmp_path) + assert patcher.ensure_benchmark_lib_patched(tmp_path) + library = project / "benchmarks/benchmark_lib.sh" + output = tmp_path / "run outputs" + output.mkdir() + result = subprocess.run( + [ + "bash", + "-c", + 'source "$1"; export RESULT_DIR="$2"; stage_result; stage_result; clean_eval; run_eval; NUM_PROMPTS=23 run_benchmark', + "bash", + str(library), + str(output), + ], + cwd=tmp_path, + capture_output=True, + text=True, + check=True, + ) + assert sorted(path.name for path in output.iterdir()) == ["results_conc8.json", "results_conc8_2.json"] + assert not (tmp_path / "results_conc8.json").exists() + assert result.stderr.splitlines() == ["HYPERLOOM_EVAL_START"] + assert result.stdout == "23" + assert patcher.ensure_benchmark_lib_eval_dest_patched(project) + assert patcher.ensure_benchmark_lib_eval_start_patched(project) + + +@pytest.mark.parametrize("nested", [False, True]) +def test_modern_probe_targets_active_module_and_all_anchors_verify(tmp_path, nested): + project = _modern_tree(tmp_path, nested) + target = project / "infx/evals/patches/lm_eval_sitecustomize.py" + statuses = patcher.verify_patch_anchors(tmp_path) + assert len(statuses) == 4 + assert all(status.hits == 1 and status.ok for status in statuses) + assert patcher.failed_patch_anchors_in(tmp_path) == [] + assert patcher.eval_probe_targets_exist(tmp_path) + assert patcher.ensure_eval_probe_patched(tmp_path) + assert patcher.ensure_eval_unbound_outputs_patched(tmp_path) + code = target.read_text() + assert "HYPERLOOM_EVAL_PROBE" in code + assert "HYPERLOOM_EVAL_UNBOUND_OUTPUTS" in code + compile(code, str(target), "exec") + assert patcher.ensure_eval_probe_patched(project) + assert target.read_text() == code + + +def test_contract_local_refresh_reads_committed_bytes_not_modified_checkout(tmp_path): + subprocess.run(["git", "init", "-q", str(tmp_path)], check=True) + target = tmp_path / "file.txt" + target.write_text("committed\n") + subprocess.run(["git", "-C", str(tmp_path), "add", "file.txt"], check=True) + subprocess.run( + [ + "git", + "-C", + str(tmp_path), + "-c", + "user.name=Test", + "-c", + "user.email=test@example.test", + "commit", + "-qm", + "fixture", + ], + check=True, + ) + revision = subprocess.check_output(["git", "-C", str(tmp_path), "rev-parse", "HEAD"], text=True).strip() + target.write_text("changed\n") + assert contract.fetch_pinned_file("file.txt", revision, tmp_path) == "committed\n" + assert contract.fetch_pinned_file("missing", revision, tmp_path) is None + + +def test_contract_refuses_missing_current_anchor(monkeypatch): + monkeypatch.setattr(contract, "fetch_pinned_file", lambda *_args: "incompatible upstream content\n") + with pytest.raises(RuntimeError, match="anchor num_prompts matched 0"): + contract.build_record("a" * 40) diff --git a/src/hyperloom/inference_optimizer/tests/test_inferencex_patcher.py b/src/hyperloom/inference_optimizer/tests/test_inferencex_patcher.py index 67f12d9060..be8a0ba47c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_inferencex_patcher.py +++ b/src/hyperloom/inference_optimizer/tests/test_inferencex_patcher.py @@ -436,6 +436,27 @@ def test_ensure_benchmark_serving_patched_patches_both_roots_when_they_differ( assert "PROFILE_EXTRA_BODY" in text_magpie, "$MAGPIE_PATH/InferenceX file must ALSO be patched (the #210 fix)" +@pytest.mark.parametrize("explicit_exists", [True, False]) +def test_explicit_profile_checkout_preserves_native_and_bundled_sources(tmp_path, monkeypatch, explicit_exists): + native = tmp_path / "native" + magpie = tmp_path / "Magpie" + isolated = tmp_path / "profile" / ".agentx-profile-inferencex" + untouched = [ + _make_inferencex_tree_with_serving(native), + _make_inferencex_tree_with_serving(magpie / "InferenceX"), + ] + snapshots = {path: path.read_bytes() for path in untouched} + target = _make_inferencex_tree_with_serving(isolated) if explicit_exists else None + monkeypatch.setenv("INFERENCEX_PATH", str(native)) + monkeypatch.setenv("MAGPIE_PATH", str(magpie)) + + assert ensure_benchmark_serving_patched(isolated) is explicit_exists + + assert {path: path.read_bytes() for path in untouched} == snapshots + if target is not None: + assert "PROFILE_EXTRA_BODY" in target.read_text(encoding="utf-8") + + def test_ensure_benchmark_serving_patched_returns_true_when_only_magpie_path_present( tmp_path, monkeypatch, @@ -996,7 +1017,8 @@ def test_failed_patch_anchors_in_ignores_rot_outside_the_named_root(tmp_path, mo monkeypatch.delenv("INFERENCEX_PATH", raising=False) monkeypatch.setenv("MAGPIE_PATH", str(bundled)) - assert [s.name for s in failed_patch_anchors(pinned)] == ["eval_dest"] + assert [s.name for s in failed_patch_anchors()] == ["eval_dest"] + assert failed_patch_anchors(pinned) == [] assert failed_patch_anchors_in(pinned) == [] assert [s.name for s in failed_patch_anchors_in(bundled / "InferenceX")] == ["eval_dest"] diff --git a/src/hyperloom/inference_optimizer/tests/test_inferencex_preflight_clone.py b/src/hyperloom/inference_optimizer/tests/test_inferencex_preflight_clone.py index 3008723016..e55c46e627 100644 --- a/src/hyperloom/inference_optimizer/tests/test_inferencex_preflight_clone.py +++ b/src/hyperloom/inference_optimizer/tests/test_inferencex_preflight_clone.py @@ -158,6 +158,7 @@ def test_auto_detected_inferencex_candidates_must_be_writable(): _PIN = "3d5581562f643f9bdeb8410cd924e2c70906c966" _OTHER = "a4bb43afa7fd74c1356583ed29e51421be010f0f" +_AIPERF_PIN = "754356e9a39acc6cc6afb242d123bb57c3fb6f75" def _checkout(tmp_path, name="co"): @@ -205,6 +206,61 @@ def test_explicit_empty_ref_skips_the_comparison(tmp_path, monkeypatch): assert cli_preflight._inferencex_checkout_ok(_checkout(tmp_path), ref="") is True +def test_native_agentx_checkout_fails_closed_without_exact_git_identity(tmp_path, monkeypatch): + """Legacy source drops stay supported, but native recipes require an exact pin.""" + checkout = _checkout(tmp_path) + (checkout / "utils" / "aiperf").mkdir(parents=True) + (checkout / "utils" / "aiperf" / "pyproject.toml").write_text("[project]\n") + _at(monkeypatch, "") + + assert ( + cli_preflight._inferencex_checkout_ok( + checkout, + ref=_PIN, + require_agentx_submodule=True, + ) + is False + ) + assert ( + cli_preflight._inferencex_checkout_ok( + checkout, + ref="main", + require_agentx_submodule=True, + ) + is False + ) + + +def test_native_agentx_checkout_verifies_the_pinned_aiperf_gitlink(tmp_path, monkeypatch): + checkout = _checkout(tmp_path) + aiperf = checkout / "utils" / "aiperf" + aiperf.mkdir(parents=True) + (aiperf / "pyproject.toml").write_text("[project]\n") + + def head(path): + return _AIPERF_PIN if Path(path) == aiperf else _PIN + + monkeypatch.setattr(cli_preflight, "_inferencex_head_sha", head) + monkeypatch.setattr( + cli_preflight.subprocess, + "run", + lambda *args, **kwargs: subprocess.CompletedProcess( + args[0], + 0, + stdout=f"160000 commit {_AIPERF_PIN}\tutils/aiperf\n", + ), + ) + + assert ( + cli_preflight._inferencex_checkout_ok( + checkout, + ref=_PIN, + require_agentx_submodule=True, + ) + is True + ) + + def test_incomplete_checkout_still_rejected_before_any_ref_work(tmp_path, monkeypatch): def _boom(_p): raise AssertionError("ref check must not run on an incomplete checkout") diff --git a/src/hyperloom/inference_optimizer/tests/test_install_inferencex.py b/src/hyperloom/inference_optimizer/tests/test_install_inferencex.py new file mode 100644 index 0000000000..e049ed329c --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_install_inferencex.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Installer ownership of existing and newly cloned InferenceX checkouts.""" + +from __future__ import annotations + +import os +import re +import subprocess +from pathlib import Path + +import pytest + + +INSTALL_SCRIPT = Path(__file__).resolve().parents[1] / "assets" / "install.sh" + + +def _run_inferencex_install(tmp_path: Path, *, explicit_path: Path | None = None) -> subprocess.CompletedProcess[str]: + text = INSTALL_SCRIPT.read_text(encoding="utf-8") + match = re.search(r"^ensure_inferencex\(\) \{.*?^\}", text, re.S | re.M) + assert match is not None + normalize = re.search(r"^normalize_inferencex_project\(\) \{.*?^\}", text, re.S | re.M) + assert normalize is not None + script = tmp_path / "installer-harness.sh" + script.write_text( + """set -euo pipefail +log() { :; } +warn() { echo "$*" >&2; } +CHECK_ONLY=0 +DRY_RUN=0 +INFERENCEX_REPO=https://example.invalid/InferenceX.git +INFERENCEX_REF=deadbeef +git_fetch_pinned() { mkdir -p "$2/benchmarks"; } +ensure_inferencex_aiperf_submodule() { + printf '%s\\n' "$INFERENCEX_PATH" >> "$SUBMODULE_UPDATE_LOG" +} +""" + + normalize.group(0) + + "\n" + + match.group(0) + + '\nensure_inferencex\nprintf "%s\\n" "$INFERENCEX_PATH"\n', + encoding="utf-8", + ) + return subprocess.run( + ["bash", str(script)], + env={ + **os.environ, + "INFERENCEX_PATH": str(explicit_path or ""), + "INFERENCEX_DEFAULT_DIR": str(tmp_path / "cached"), + "SUBMODULE_UPDATE_LOG": str(tmp_path / "submodule-updates"), + }, + check=True, + capture_output=True, + text=True, + ) + + +@pytest.mark.parametrize("explicit", [True, False]) +@pytest.mark.parametrize("has_aiperf", [True, False]) +def test_installer_preserves_existing_checkout_without_submodule_updates(tmp_path, explicit, has_aiperf): + checkout = tmp_path / ("user-checkout" if explicit else "cached") + (checkout / "benchmarks").mkdir(parents=True) + launcher = checkout / "benchmarks" / "custom.sh" + launcher.write_text("# operator-owned launcher\n", encoding="utf-8") + if has_aiperf: + submodule = checkout / "utils" / "aiperf" + submodule.mkdir(parents=True) + (submodule / "pyproject.toml").write_text("# operator-owned revision\n", encoding="utf-8") + before = {path.relative_to(checkout): path.read_bytes() for path in checkout.rglob("*") if path.is_file()} + + result = _run_inferencex_install(tmp_path, explicit_path=checkout if explicit else None) + + assert result.stdout.strip() == str(checkout) + assert not (tmp_path / "submodule-updates").exists() + assert {path.relative_to(checkout): path.read_bytes() for path in checkout.rglob("*") if path.is_file()} == before + + +def test_installer_initializes_submodule_for_its_new_clone(tmp_path): + result = _run_inferencex_install(tmp_path) + + assert result.stdout.strip() == str(tmp_path / "cached") + assert (tmp_path / "submodule-updates").read_text().splitlines() == [str(tmp_path / "cached")] + + +@pytest.mark.parametrize("explicit", [False, True]) +def test_installer_normalizes_packaged_project_without_mutating_existing_checkout(tmp_path, explicit): + repository = tmp_path / ("user-checkout" if explicit else "cached") + (repository / ".git").mkdir(parents=True) + project = repository / "inferencex-e2e" + (project / "benchmarks").mkdir(parents=True) + library = project / "benchmarks/benchmark_lib.sh" + library.write_text("# upstream client\n") + result = _run_inferencex_install(tmp_path, explicit_path=repository if explicit else None) + assert result.stdout.strip() == str(project) + assert library.read_text() == "# upstream client\n" + assert not (tmp_path / "submodule-updates").exists() diff --git a/src/hyperloom/inference_optimizer/tests/test_install_magpie_idempotent.py b/src/hyperloom/inference_optimizer/tests/test_install_magpie_idempotent.py index 2f79d01859..ba47cb3da3 100644 --- a/src/hyperloom/inference_optimizer/tests/test_install_magpie_idempotent.py +++ b/src/hyperloom/inference_optimizer/tests/test_install_magpie_idempotent.py @@ -10,12 +10,14 @@ import subprocess from pathlib import Path +import pytest + REPO_ROOT = Path(__file__).resolve().parents[4] IO_INSTALL = REPO_ROOT / "src" / "hyperloom" / "inference_optimizer" / "assets" / "install.sh" PREFLIGHT = REPO_ROOT / "src" / "hyperloom" / "inference_optimizer" / "cli" / "preflight.py" -MAGPIE_V020_COMMIT = "e6833b8183c6c41adf6038252337550876ca0433" +MAGPIE_AGENTX_COMMIT = "d80eb4d3dad7fabe01ce81d049e2983adf2c86dd" PIP_MARKER = "pip-install-called" @@ -27,13 +29,18 @@ def _extract_ensure_magpie() -> str: return m.group(0) -def _fake_python(tmp_path: Path, *, import_ok: bool, installed_root: Path) -> Path: +def _fake_python(tmp_path: Path, *, capability_ok: bool, installed_root: Path, repair_required: bool = False) -> Path: """A stub ``$PYTHON``.""" marker = tmp_path / PIP_MARKER - import_check = "exit 0" if import_ok else f'[ -f "{marker}" ] && exit 0 || exit 1' + repaired = tmp_path / "pip-force-reinstall-called" + health_marker = repaired if repair_required else marker + capability_check = "exit 0" if capability_ok else f'[ -f "{health_marker}" ] && exit 0 || exit 1' body = f"""#!/usr/bin/env bash if [ "$1" = "-m" ] && [ "$2" = "pip" ]; then - touch "{marker}" + printf '%s\n' "$*" >> "{marker}" + case " $* " in + *" --force-reinstall "*) touch "{repaired}" ;; + esac exit 0 fi if [ "$1" = "-" ]; then @@ -41,7 +48,7 @@ def _fake_python(tmp_path: Path, *, import_ok: bool, installed_root: Path) -> Pa exit 0 fi if [ "$1" = "-c" ]; then - {import_check} + {capability_check} fi exit 0 """ @@ -54,14 +61,17 @@ def _fake_python(tmp_path: Path, *, import_ok: bool, installed_root: Path) -> Pa def _run_ensure_magpie( tmp_path: Path, *, - import_ok: bool, + capability_ok: bool, explicit_magpie_path: bool = False, + repair_required: bool = False, ) -> tuple[str, bool]: """Run the extracted ensure_magpie body; return (stdout, pip_called).""" installed_root = tmp_path / "site-packages" installed_root.mkdir(parents=True) magpie_dir = tmp_path / "operator" / "Magpie" - fake_py = _fake_python(tmp_path, import_ok=import_ok, installed_root=installed_root) + fake_py = _fake_python( + tmp_path, capability_ok=capability_ok, installed_root=installed_root, repair_required=repair_required + ) magpie_path_line = ( f'MAGPIE_PATH="{magpie_dir}"\nMAGPIE_PATH_EXPLICIT=1' if explicit_magpie_path @@ -79,7 +89,7 @@ def _run_ensure_magpie( MAGPIE_PACKAGE_SPEC="magpie-eval @ git+https://example.invalid/Magpie.git@deadbeef" {magpie_path_line} PYTHON="{fake_py}" -PIP_EXTRA=() +PIP_EXTRA=("--disable-pip-version-check") {_extract_ensure_magpie()} @@ -100,21 +110,32 @@ def _run_ensure_magpie( return proc.stdout, pip_called -def test_skip_reinstall_when_already_installed_from_magpie_dir(tmp_path: Path) -> None: - out, pip_called = _run_ensure_magpie(tmp_path, import_ok=True) +def test_skip_reinstall_when_native_agentx_capability_is_available(tmp_path: Path) -> None: + out, pip_called = _run_ensure_magpie(tmp_path, capability_ok=True) assert not pip_called, f"reinstall should have been skipped:\n{out}" - assert "Magpie already importable; skipping pip install" in out + assert "Magpie package healthy; skipping pip install" in out assert "MAGPIE_PATH resolved from installed package" in out -def test_install_when_import_fails(tmp_path: Path) -> None: - out, pip_called = _run_ensure_magpie(tmp_path, import_ok=False) - assert pip_called, f"reinstall should have run on import failure:\n{out}" +def test_install_when_native_agentx_capability_is_missing(tmp_path: Path) -> None: + out, pip_called = _run_ensure_magpie(tmp_path, capability_ok=False) + assert pip_called, f"reinstall should have run on capability miss:\n{out}" assert "Magpie installed OK from magpie-eval @ git+https://example.invalid/Magpie.git@deadbeef" in out +def test_repairs_an_installed_commit_whose_execution_tree_was_patched(tmp_path: Path) -> None: + out, pip_called = _run_ensure_magpie(tmp_path, capability_ok=False, repair_required=True) + + assert pip_called + calls = (tmp_path / PIP_MARKER).read_text().splitlines() + assert len(calls) == 2 + assert "--force-reinstall" not in calls[0] + assert "--force-reinstall --no-deps" in calls[1] + assert "Magpie installed OK" in out + + def test_preserves_explicit_magpie_path(tmp_path: Path) -> None: - out, pip_called = _run_ensure_magpie(tmp_path, import_ok=True, explicit_magpie_path=True) + out, pip_called = _run_ensure_magpie(tmp_path, capability_ok=True, explicit_magpie_path=True) assert not pip_called assert "MAGPIE_PATH override preserved" in out @@ -122,14 +143,77 @@ def test_preserves_explicit_magpie_path(tmp_path: Path) -> None: # Static guard: the idempotent skip must stay wired in. def test_io_install_magpie_reinstall_is_idempotent_guarded() -> None: body = _extract_ensure_magpie() - assert "Magpie already importable; skipping pip install" in body + assert "magpie_health_code(native_agentx=native_agentx_session())" in body + assert "Magpie package healthy; skipping pip install" in body assert "MAGPIE_PACKAGE_SPEC" in body assert "pip install" in body, "reinstall path must still exist for the miss case" -def test_default_magpie_pin_is_v020_and_consistent() -> None: +def test_default_magpie_pin_has_native_agentx_and_is_consistent() -> None: install_text = IO_INSTALL.read_text(encoding="utf-8") preflight_text = PREFLIGHT.read_text(encoding="utf-8") - assert f'MAGPIE_REF="${{MAGPIE_REF:-{MAGPIE_V020_COMMIT}}}"' in install_text - assert f'os.environ.get("MAGPIE_REF", "{MAGPIE_V020_COMMIT}")' in preflight_text + assert f'MAGPIE_REF="${{MAGPIE_REF:-{MAGPIE_AGENTX_COMMIT}}}"' in install_text + assert f'_MAGPIE_REF_DEFAULT = "{MAGPIE_AGENTX_COMMIT}"' in preflight_text + assert 'os.environ.get("MAGPIE_REF") or _MAGPIE_REF_DEFAULT' in preflight_text + + +@pytest.mark.parametrize( + "mode, requires_native", + [ + ("synthetic", False), + ("native-switch", True), + ("native-yaml", True), + ("legacy-resume", False), + ("native-resume", True), + ("mlperf", False), + ], +) +def test_installer_checks_the_session_contract_for_an_importable_legacy_package(tmp_path, mode, requires_native): + import json + import os + import sys + + package = tmp_path / "packages" / "Magpie" + package.mkdir(parents=True) + (package / "__init__.py").write_text("# Importable Magpie without AgentX support.\n") + python = tmp_path / "python" + python.write_text( + f"#!{sys.executable}\n" + "import os, sys\n" + "if sys.argv[1:3] == ['-m', 'pip']:\n" + " print('pip-install-called'); sys.exit(71)\n" + f"os.execv({sys.executable!r}, [{sys.executable!r}] + sys.argv[1:])\n" + ) + python.chmod(0o755) + env = dict( + os.environ, + PYTHONPATH=os.pathsep.join([str(package.parent), str(REPO_ROOT / "src")]), + HYPERLOOM_AGENTX="1" if mode in {"native-switch", "legacy-resume", "mlperf"} else "0", + HYPERLOOM_AGENTIC_BACKEND="mlperf" if mode == "mlperf" else "aiperf", + HYPERLOOM_BENCHMARK_CONFIG="", + INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR="", + ) + if mode == "native-yaml": + config = tmp_path / "native.yaml" + config.write_text("benchmark:\n agentx: enable\n") + env["HYPERLOOM_BENCHMARK_CONFIG"] = str(config) + if mode.endswith("resume"): + (tmp_path / "state.json").write_text( + json.dumps({"benchmark_mode": "agentx", "agentx_epoch": 1 if mode == "legacy-resume" else 4}) + ) + env["INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR"] = str(tmp_path) + script = tmp_path / "install.sh" + script.write_text( + 'set -euo pipefail\nlog() { echo "$*"; }\nwarn() { echo "$*"; }\n' + f'PYTHON="{python}"\nMAGPIE_REF=custom-legacy-ref\nMAGPIE_PACKAGE_SPEC=local-package\n' + "CHECK_ONLY=0\nDRY_RUN=0\nMAGPIE_PATH_EXPLICIT=0\nPIP_EXTRA=()\n" + + _extract_ensure_magpie() + + "\nensure_magpie\n" + ) + proc = subprocess.run(["bash", str(script)], env=env, text=True, capture_output=True, check=False) + assert proc.returncode == (71 if requires_native else 0), proc.stderr + assert (PIP_MARKER in proc.stdout) is requires_native + if not requires_native: + assert "skipping pip install" in proc.stdout + assert str(package.parent) in proc.stdout diff --git a/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py b/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py index cfb40321cd..86e8b6435c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py +++ b/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py @@ -215,7 +215,10 @@ async def _measure(ctx): # integrate_handler @pytest.mark.asyncio -async def test_integrate_handler_materializes_persisted_agentx_mode(session_dir, tmp_path, monkeypatch): +@pytest.mark.parametrize("native", [False, True], ids=["legacy", "native"]) +async def test_integrate_handler_preserves_legacy_mutations_and_refuses_native( + session_dir, tmp_path, monkeypatch, native +): from hyperloom.orchestrator.actions.executors.baseline import BaselineExecutor monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False) @@ -226,6 +229,7 @@ async def test_integrate_handler_materializes_persisted_agentx_mode(session_dir, state = SharedState.load_or_init(session_dir) state.framework = "sglang" state.benchmark_mode = "agentx" + state.agentx_epoch = 2 if native else 1 state.baseline_tput = 100.0 state.baseline_accuracy = 0.80 state.baseline_perf = { @@ -282,6 +286,18 @@ async def _measure(executor, *, config_path, output_dir, **_kwargs): session_dir=session_dir, ) + if native: + assert seen == {} + assert target.read_text(encoding="utf-8") == "def kernel():\n return 'original'\n" + assert result["status"] == "skipped" + assert result["decision"] == "NEEDS_REVIEW" + assert result["error_class"] == "unsupported_upstream_launcher_hook" + assert result["patches_applied"] == result["patches_reverted"] == [] + assert ( + yaml.safe_load(base_yaml.read_text(encoding="utf-8"))["benchmark"]["benchmark_script"] == "sglang_mi300x.sh" + ) + return + assert seen["benchmark"]["benchmark_script"] == "aiperf_client.sh" assert seen["shared_state"] is not None assert seen["shared_state"].benchmark_mode == "agentx" diff --git a/src/hyperloom/inference_optimizer/tests/test_managed_agentx_profile.py b/src/hyperloom/inference_optimizer/tests/test_managed_agentx_profile.py new file mode 100644 index 0000000000..738dc38a45 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_managed_agentx_profile.py @@ -0,0 +1,411 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Managed diagnostic captures preserve candidates and cannot become KEEP evidence.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import time +from copy import deepcopy +from pathlib import Path +from types import SimpleNamespace + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx import identity, managed, native +from hyperloom.inference_optimizer.tests.test_agentx_optimization_identity import _candidate_artifacts +from hyperloom.orchestrator.actions.executors import baseline, profile +from hyperloom.orchestrator.actions.executors._native_candidate import apply_native_candidate +from hyperloom.orchestrator.actions.executors._managed_profile_result import diagnostic_profile_result +from hyperloom.orchestrator.actions.executors._native_profile import prepare_managed_profile +from hyperloom.orchestrator.actions.executors._workload_envs import materialize_config_with_envs +from hyperloom.orchestrator.actions.executors.benchmark_result import extract_benchmark_measurement + + +def _accepted(tmp_path): + config, report = _candidate_artifacts(tmp_path / "accepted-artifacts", env={"SGLANG_USE_AITER": "1"}) + evidence = report["agentx_metrics"]["server_launch"] + checkout = tmp_path / "inferencex" + (checkout / "benchmarks").mkdir(parents=True) + for name in ("srt_agentic.sh", "benchmark_lib.sh"): + (checkout / "benchmarks" / name).write_text("# fixture\n") + spec = {"version": 1, "argv": evidence["base_argv"], "source_files": {}} + config["agentx"]["resolved"]["server-launch-spec"] = spec + config.update(benchmark_script="srt_agentic.sh", inferencex_path=str(checkout), run_mode="local") + config["profiler"] = {"torch_profiler": {"enabled": False}} + config.setdefault("envs", {})["RUN_EVAL"] = "false" + apply_native_candidate(config) + evidence["overrides_sha256"] = identity.canonical_sha256(config["agentx"]["launch_overrides"]) + evidence.update(owner="magpie", server_spec_sha256=identity.canonical_sha256(spec), recipe_source_files={}) + evidence["evidence_sha256"] = identity.canonical_sha256( + {k: v for k, v in evidence.items() if k != "evidence_sha256"} + ) + config["workload_spec"]["server_launch"] = evidence + config["workload_spec"]["resolved_topology"] = {"gpu_count": 2, "tp": 2, "pp": 1, "pcp_size": 1, "ep": 1, "conc": 8} + config["workload_spec"]["execution"]["workload_fingerprint"] = identity.native_workload_fingerprint( + config, "b" * 64 + ) + accepted = tmp_path / "accepted.yaml" + accepted.write_text(yaml.safe_dump({"benchmark": config})) + state = SimpleNamespace( + benchmark_mode="agentx", + agentx_backend="native", + agentx_epoch=4, + baseline_config_path=str(accepted), + baseline_double_run=False, + current_best_measurement={"materialized_config": str(accepted), "agentx_server_launch": evidence}, + ) + return config, evidence, state + + +def _write_capture(workspace, evidence, *, profiles=2, num_steps=20): + workspace.mkdir(parents=True, exist_ok=True) + root = workspace / "torch_trace" / ("a" * 32) + rows = [] + for index in range(1, profiles + 1): + directory = root / f"profile_{index:03d}" + directory.mkdir(parents=True) + # Names intentionally have identical TP components: the manifest's global ranks are authoritative. + files = [directory / f"worker-{rank}-TP-0-EP-0.trace.json" for rank in range(2)] + for rank, path in enumerate(files): + path.write_text( + json.dumps( + { + "distributedInfo": {"rank": rank}, + "traceEvents": [{"cat": "kernel", "ph": "X", "name": "kernel", "dur": 1}], + } + ) + ) + rows.append( + { + "status": "complete", + "capture_id": root.name, + "framework": evidence["framework"], + "num_steps": num_steps, + "profile_index": index, + "expected_ranks": 2, + "trace_dir": str(directory), + "trace_files": list(map(str, files)), + "rank_trace_files": {str(rank): [str(path)] for rank, path in enumerate(files)}, + } + ) + capture = { + "status": "complete", + "capture_id": root.name, + "framework": evidence["framework"], + "num_steps": num_steps, + "expected_ranks": 2, + "requested_profiles": 99, + "max_profiles": 18, + "planned_profiles": 18, + "effective_profiles": profiles, + "completed_profiles": profiles, + "stop_reason": "insufficient_measurement_time", + "profiles": rows, + "trace_files": [path for row in rows for path in row["trace_files"]], + } + (root / "capture.json").write_text(json.dumps(capture)) + evidence = deepcopy(evidence) + evidence["torch_profiler"] = {"capture_id": root.name, "trace_dir": str(root), "num_steps": num_steps} + evidence["evidence_sha256"] = identity.canonical_sha256( + {key: value for key, value in evidence.items() if key != "evidence_sha256"} + ) + report = { + "success": True, + "errors": [], + "benchmark_valid": False, + "publishable": False, + "agentx_metrics": { + "diagnostic_only": True, + "server_launch": evidence, + "profile_capture": capture, + "profile_analyses": [{"profile_index": index} for index in range(1, profiles + 1)], + }, + } + (workspace / "agentx_server_launch.json").write_text(json.dumps(evidence)) + (workspace / "benchmark_report.json").write_text(json.dumps(report)) + return report + + +def _write_step_trace(path, num_steps): + events = [] + for index in range(num_steps): + events.extend( + [ + {"ph": "X", "cat": "user_annotation", "name": "step[DECODE bs=8]", "ts": index * 10, "dur": 10}, + {"ph": "X", "cat": "kernel", "name": "fixture_kernel", "ts": index * 10 + 1, "dur": 5}, + ] + ) + path.write_text(json.dumps({"traceEvents": events})) + + +def test_profile_derivation_keeps_accepted_candidate_and_structured_profiler_options(tmp_path, monkeypatch): + monkeypatch.setattr(identity, "_validate_managed_receipt", lambda *args: []) + config, _, state = _accepted(tmp_path) + accepted = Path(state.baseline_config_path) + original = accepted.read_bytes() + params = { + "base_extra_args": "--must-not-reapply", + "base_extra_envs": {"SGLANG_USE_AITER": "0"}, + "num_steps": 12, + "num_profiles": 99, + "start_seconds": 30, + "interval_seconds": 200, + "capture_timeout_seconds": 400, + "flush_timeout_seconds": 1900, + "detailed_annotations": True, + } + prepare_managed_profile(params, state, tmp_path / "profile") + actual = yaml.safe_load(Path(params["config_path"]).read_text())["benchmark"] + assert actual["agentx"] == config["agentx"] + assert actual["envs"] == config["envs"] + assert actual["profiler"]["torch_profiler"] == { + "enabled": True, + "num_steps": 12, + "num_profiles": 99, + "start_seconds": 30, + "interval_seconds": 200, + "capture_timeout_seconds": 400, + "flush_timeout_seconds": 1900, + "detailed_annotations": True, + } + assert actual["workload_spec"]["profile_parent"]["config_sha256"] == hashlib.sha256(original).hexdigest() + assert params["native_launch_overrides"] == config["agentx"]["launch_overrides"] + assert accepted.read_bytes() == original + assert ( + managed.identity_benchmark(actual, config["workload_spec"]["execution"]["workload_fingerprint"])["profiler"] + == config["profiler"] + ) + with pytest.raises(ValueError, match="only.*ProfileExecutor"): + materialize_config_with_envs(Path(params["config_path"]), tmp_path / "baseline") + + +@pytest.mark.parametrize( + "change", + [ + None, + "returncode", + "runtime", + "partial", + "rank", + "missing", + "manifest", + "receipt", + "stale", + "capture_id", + "capture_dir", + "gpu_count", + "framework", + "num_steps", + "round_num_steps", + ], +) +def test_diagnostic_report_requires_complete_runtime_and_bound_round_artifacts(tmp_path, monkeypatch, change): + config, evidence, _ = _accepted(tmp_path) + config["profiler"]["torch_profiler"]["enabled"] = True + config_path = tmp_path / "profile.yaml" + config_path.write_text(yaml.safe_dump({"benchmark": config})) + workspace = tmp_path / "benchmark_profile" + report = _write_capture(workspace, evidence) + monkeypatch.setattr(identity, "_validate_managed_receipt", lambda *args: []) + capture = report["agentx_metrics"]["profile_capture"] + if change == "runtime": + report["success"] = False + if change == "partial": + capture["status"] = "failed" + if change == "rank": + capture["profiles"][-1]["rank_trace_files"].pop("1") + if change == "missing": + Path(capture["profiles"][-1]["trace_files"][0]).unlink() + if change == "manifest": + capture["completed_profiles"] = 88 + if change == "gpu_count": + config["workload_spec"]["resolved_topology"]["gpu_count"] = 4 + config_path.write_text(yaml.safe_dump({"benchmark": config})) + if change == "framework": + capture["framework"] = "vllm" + if change == "num_steps": + capture["num_steps"] += 1 + if change == "round_num_steps": + capture["profiles"][-1]["num_steps"] += 1 + if change in {"capture_id", "capture_dir"}: + launch = report["agentx_metrics"]["server_launch"] + key = "capture_id" if change == "capture_id" else "trace_dir" + launch["torch_profiler"][key] = "b" * 32 if change == "capture_id" else str(workspace / "other") + launch["evidence_sha256"] = identity.canonical_sha256( + {key: value for key, value in launch.items() if key != "evidence_sha256"} + ) + (workspace / "agentx_server_launch.json").write_text(json.dumps(launch)) + (workspace / "torch_trace" / capture["capture_id"] / "capture.json").write_text(json.dumps(capture)) + if change == "receipt": + (workspace / "agentx_server_launch.json").write_text("{}") + if change == "stale": + os.utime(workspace / "benchmark_report.json", (1, 1)) + result = diagnostic_profile_result( + report, + workspace=workspace, + config_path=config_path, + returncode=1 if change == "returncode" else 0, + subprocess_started_unix=time.time(), + ) + assert result["status"] == ("failed" if change else "succeeded"), result + assert result["trace_input_ready"] is (not bool(change)) + assert result["valid_measurement"] is False + assert result["submission_valid"] is False + if change in {"capture_id", "capture_dir", "gpu_count", "framework", "num_steps", "round_num_steps"}: + expected = { + "capture_id": "capture ID", + "capture_dir": "capture directory", + "gpu_count": "rank count", + "framework": "capture framework", + "num_steps": "step count", + "round_num_steps": "round num_steps", + } + assert expected[change] in result["error"] + if not change: + assert result["selected_profile_index"] == 2 + assert result["trace_files"] == capture["profiles"][1]["trace_files"] + assert result["main_trace_path"] == capture["profiles"][1]["rank_trace_files"]["0"][0] + assert result["profile_capture"]["effective_profiles"] == 2 + assert not extract_benchmark_measurement(report, workspace=workspace, materialized_config_path=config_path)[ + "valid_measurement" + ] + + +def test_single_profile_manifest_keeps_the_same_kernel_input_contract(tmp_path, monkeypatch): + config, evidence, _ = _accepted(tmp_path) + config["profiler"]["torch_profiler"]["enabled"] = True + config_path = tmp_path / "profile.yaml" + config_path.write_text(yaml.safe_dump({"benchmark": config})) + workspace = tmp_path / "benchmark_profile" + report = _write_capture(workspace, evidence, profiles=1) + capture = dict(report["agentx_metrics"]["profile_capture"]["profiles"][0]) + capture.pop("profile_index") + capture.pop("trace_dir") + report["agentx_metrics"]["profile_capture"] = capture + (workspace / "torch_trace" / capture["capture_id"] / "capture.json").write_text(json.dumps(capture)) + monkeypatch.setattr(identity, "_validate_managed_receipt", lambda *args: []) + result = diagnostic_profile_result( + report, workspace=workspace, config_path=config_path, returncode=0, subprocess_started_unix=time.time() + ) + assert result["trace_input_ready"] is True + assert result["selected_profile_index"] == 1 + assert result["main_trace_path"] == capture["rank_trace_files"]["0"][0] + assert result["trace_files"] == capture["trace_files"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("returncode", [0, 1]) +async def test_managed_profile_executor_runs_native_materialization_and_diagnostic_gate( + tmp_path, monkeypatch, returncode +): + config, evidence, state = _accepted(tmp_path) + fingerprint = config["workload_spec"]["execution"]["workload_fingerprint"] + for key in list(os.environ): + if key.startswith(("AGENTX_", "HYPERLOOM_AGENTX_EXPECTED_")) or key in { + "HYPERLOOM_AGENTX_GPU_COUNT", + "HYPERLOOM_AGENTIC_BACKEND", + }: + monkeypatch.delenv(key, raising=False) + monkeypatch.setenv("INFERENCE_OPTIMIZER_LEAK_ROOTS", str(tmp_path / "no-leaks")) + monkeypatch.setenv("INFERENCEX_PATH", config["inferencex_path"]) + monkeypatch.setenv("FRAMEWORK", "sglang") + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setattr(identity, "_validate_managed_receipt", lambda *args: []) + resolved = [] + + def resolve(bench, **kwargs): + managed.identity_benchmark(bench, fingerprint) + resolved.append(deepcopy(bench)) + return bench["workload_spec"]["resolved_topology"] + + monkeypatch.setattr(native, "resolve_native_recipe", resolve) + captured = {} + + def run(cmd, **kwargs): + path = Path(cmd[cmd.index("--benchmark-config") + 1]) + bench = yaml.safe_load(path.read_text())["benchmark"] + captured.update(bench) + launched = deepcopy(evidence) + launched["overrides_sha256"] = identity.canonical_sha256(bench["agentx"]["launch_overrides"]) + launched["evidence_sha256"] = identity.canonical_sha256( + {k: v for k, v in launched.items() if k != "evidence_sha256"} + ) + workspace = Path(cmd[cmd.index("--output-dir") + 1]) / "benchmark_sglang_profile" + report = _write_capture(workspace, launched, num_steps=bench["profiler"]["torch_profiler"]["num_steps"]) + for capture in report["agentx_metrics"]["profile_capture"]["profiles"]: + directory = Path(capture["trace_dir"]) + # A misplaced neighbouring config must not override the verified materialized workload. + (directory.parent / "config.yaml").write_text("framework: vllm\n num_steps: 999\n CONC: 99\n") + for trace in capture["trace_files"]: + _write_step_trace(Path(trace), 20) + return subprocess.CompletedProcess(cmd, returncode, "", "runtime failed" if returncode else "") + + monkeypatch.setattr(baseline, "run_with_session_kill", run) + executor = profile.ProfileExecutor(magpie_python="python3", session_dir=tmp_path) + executor.shared_state = state + ctx = SimpleNamespace( + task=SimpleNamespace( + kind="profile", + task_id="native-profile", + params={ + "output_dir": str(tmp_path / "run"), + "num_steps": 20, + "num_profiles": 99, + }, + ), + extra={}, + ) + result = await executor(ctx) + assert result["status"] == ("failed" if returncode else "succeeded"), result + assert result["trace_input_ready"] is (returncode == 0) + assert captured["benchmark_script"] == "srt_agentic.sh" + assert captured["profiler"]["torch_profiler"]["num_profiles"] == 99 + assert "PROFILE_EXTRA_BODY" not in captured["envs"] + assert "EXTRA_SGLANG_ARGS" not in captured["envs"] + assert not (tmp_path / "run" / ".agentx-profile-inferencex").exists() + assert len(resolved) == 2 + if returncode == 0: + assert result["valid_measurement"] is False + assert "trace_validate" in result + assert all("trace_validate" in item for item in result["profile_rounds"]) + for capture in result["profile_rounds"]: + assert Path(capture["trace_dir"]).name == f"profile_{capture['profile_index']:03d}" + validation = capture["trace_validate"] + assert validation["probe_status"] == "ok" + assert validation["workload_params"] == { + "source": result["materialized_config"], + "framework": "sglang", + "num_steps": 20, + "conc": 8, + "osl": None, + "r": 1.0, + } + assert validation["rank_level"][0]["split_forecast"]["num_steps_param"] == 20 + assert any(check["status"] == "failed" for check in validation["checks"]) + assert json.loads(Path(capture["trace_validate_path"]).read_text()) == validation + + +def test_legacy_profile_certification_keeps_adjacent_config_context(tmp_path): + directory = tmp_path / "torch_trace" + directory.mkdir() + _write_step_trace(directory / "rank0.trace.json", 20) + (tmp_path / "config.yaml").write_text( + "framework: sglang\nprofiler:\n torch_profiler:\n num_steps: 12\n" + "envs:\n CONC: 3\n OSL: 256\n RANDOM_RANGE_RATIO: 0.5\n" + ) + certificate = profile._certify_trace_dir(directory, "sglang") + assert certificate["workload_params"] == { + "source": str(tmp_path / "config.yaml"), + "framework": "sglang", + "num_steps": 12, + "conc": 3, + "osl": 256.0, + "r": 0.5, + } + assert certificate["rank_level"][0]["split_forecast"]["num_steps_param"] == 12 diff --git a/src/hyperloom/inference_optimizer/tests/test_map_aiperf.py b/src/hyperloom/inference_optimizer/tests/test_map_aiperf.py index 47f21bb190..9b8749b4ad 100644 --- a/src/hyperloom/inference_optimizer/tests/test_map_aiperf.py +++ b/src/hyperloom/inference_optimizer/tests/test_map_aiperf.py @@ -266,6 +266,22 @@ def test_map_corpus_shape_carries_the_loader_and_the_percentiles(): assert shape["request_error_rate"] == pytest.approx(0.007) +def test_map_corpus_shape_accepts_native_magpie_accounting_keys(): + """Native Magpie normalization uses explicit request/duration field names.""" + from hyperloom.inference_optimizer.agentx.mapping import map_corpus_shape + + shape = map_corpus_shape( + { + "completed_requests": 376, + "duration_seconds": 1200.5, + "isl_distribution": {"p50": 94821}, + } + ) + + assert shape["completed_requests"] == 376 + assert shape["duration_s"] == pytest.approx(1200.5) + + def test_an_export_with_no_sequence_metrics_yields_empty_distributions(): """A synthetic result carries none of this; the record must not invent it.""" from hyperloom.inference_optimizer.agentx.mapping import map_corpus_shape diff --git a/src/hyperloom/inference_optimizer/tests/test_native_candidate.py b/src/hyperloom/inference_optimizer/tests/test_native_candidate.py new file mode 100644 index 0000000000..ac8c7105a2 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_candidate.py @@ -0,0 +1,442 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native candidate launch materialization and deployed-source provenance.""" + +from __future__ import annotations + +import copy +import hashlib +import json +import os +import sys +from pathlib import Path + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx.identity import native_workload_fingerprint +from hyperloom.orchestrator.actions.executors._grid_runner import GridVariant, _build_variant_yaml +from hyperloom.orchestrator.actions.executors._native_candidate import ( + apply_native_candidate, + record_launch_evidence, + update_native_candidate_file, +) +from hyperloom.orchestrator.actions.executors._native_source import ( + applied_source_evidence, + source_file_hashes, + verify_native_source_imports, + warm_source_evidence, +) + + +def _benchmark(): + topology = {"gpu_count": 4, "tp": 4, "pp": 1, "pcp_size": 1, "ep": 4, "conc": 8, "recipe_fingerprint": "a" * 64} + return { + "framework": "sglang", + "model": "amd/GLM-5.2-MXFP4", + "benchmark_script": "single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh", + "inferencex_path": "/pinned/inferencex", + "agentx": {"enabled": True, "mode": "canonical", "launch_overrides": {"version": 1}}, + "envs": {"CONC": "8", "TP": "4", "RUN_EVAL": "false", "MODEL_PATH": "/models/glm", "PATH": os.environ["PATH"]}, + "workload_spec": { + "resolved_topology": topology, + "server_launch": { + "base_argv": [ + sys.executable, + "-m", + "sglang.launch_server", + "--model-path", + "/models/glm", + "--disable-cuda-graph", + ] + }, + }, + } + + +def test_candidate_changes_only_server_contract_and_keeps_workload_identity(): + bench = _benchmark() + before = copy.deepcopy(bench) + fingerprint = native_workload_fingerprint(bench, "b" * 64) + apply_native_candidate( + bench, + extra_server_args="--mem-fraction-static 0.75", + extra_envs={"SGLANG_ENABLE_JIT_DEEPGEMM": "1", "CONC": "8"}, + remove_args=["--disable-cuda-graph"], + unset_envs=["SGLANG_ENABLE_FLASHINFER"], + ) + override = bench["agentx"]["launch_overrides"] + assert override["append_args"] == ["--mem-fraction-static", "0.75"] + assert override["remove_args"] == ["--disable-cuda-graph"] + assert override["env"] == {"SGLANG_ENABLE_JIT_DEEPGEMM": "1"} + assert override["unset_env"] == ["SGLANG_ENABLE_FLASHINFER"] + assert bench["envs"] == before["envs"] + assert native_workload_fingerprint(bench, "b" * 64) == fingerprint + + +def test_native_argv_preserves_json_and_does_not_silently_drop_flags(): + bench = _benchmark() + apply_native_candidate( + bench, + extra_server_args='--compilation-config \'{"mode": "max-autotune"}\' --no-enable-prefix-caching', + ) + assert bench["agentx"]["launch_overrides"]["append_args"] == [ + "--compilation-config", + '{"mode": "max-autotune"}', + "--no-enable-prefix-caching", + ] + + +@pytest.mark.parametrize( + "env", [{"CONC": "16"}, {"AGENTX_DATASET": "other"}, {"NUM_PROMPTS": "1"}, {"RUN_EVAL": "true"}] +) +def test_native_candidate_rejects_workload_retargeting(env): + with pytest.raises(ValueError, match="canonical workload"): + apply_native_candidate(_benchmark(), extra_envs=env) + + +def test_old_native_config_cannot_gain_optimizer_contract(): + bench = _benchmark() + bench["agentx"].pop("launch_overrides") + with pytest.raises(ValueError, match="accepted launch_overrides"): + apply_native_candidate(bench, extra_server_args="--disable-cuda-graph") + + +def test_layered_removals_distinguish_base_flags_from_previous_candidate(): + bench = _benchmark() + apply_native_candidate(bench, extra_server_args="--mem-fraction-static 0.7") + apply_native_candidate(bench, remove_args=["--mem-fraction-static", "--disable-cuda-graph"]) + override = bench["agentx"]["launch_overrides"] + assert override["append_args"] == [] + assert override["remove_args"] == ["--disable-cuda-graph"] + + +def test_inherited_candidate_removal_requires_observed_base(): + bench = _benchmark() + bench["workload_spec"].pop("server_launch") + apply_native_candidate(bench, extra_server_args="--mem-fraction-static 0.7") + with pytest.raises(ValueError, match="verified base_argv"): + apply_native_candidate(bench, remove_args=["--mem-fraction-static"]) + + +def test_runtime_is_server_only_and_selects_exact_interpreter(tmp_path): + bench = _benchmark() + fixed = copy.deepcopy(bench["envs"]) + apply_native_candidate( + bench, + runtime_override={"runtime_python_exe": sys.executable, "pythonpath_prefix": str(tmp_path)}, + ) + override = bench["agentx"]["launch_overrides"] + assert override["executable"] == sys.executable + assert override["env"]["PYTHONPATH"].split(":")[0] == str(tmp_path) + assert bench["envs"] == fixed + + +@pytest.mark.parametrize( + "runtime", [{"unknown_runtime_knob": True}, {"pythonpath_prefix": "/does-not-exist"}, {"runtime_env": {"TP": "8"}}] +) +def test_runtime_override_cannot_be_silently_dropped(runtime): + with pytest.raises(ValueError): + apply_native_candidate(_benchmark(), runtime_override=runtime) + + +def _package(root: Path, value: str) -> Path: + package = root / "sglang" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + module = package / "candidate.py" + module.write_text(f"VALUE = {value!r}\n", encoding="utf-8") + return module + + +def test_source_probe_uses_candidate_import_root_and_rejects_other_install(tmp_path): + target = _package(tmp_path / "candidate", "patched") + _package(tmp_path / "original", "original") + bench = _benchmark() + apply_native_candidate( + bench, + runtime_override={"runtime_python_exe": sys.executable, "pythonpath_prefix": str(target.parent.parent)}, + source_files=source_file_hashes([target]), + ) + verify_native_source_imports(bench) + assert bench["workload_spec"]["source_imports"]["sglang"]["origin"] == str(target.parent / "__init__.py") + bench["agentx"]["launch_overrides"]["env"]["PYTHONPATH"] = str(tmp_path / "original") + with pytest.raises(ValueError, match="not imported"): + verify_native_source_imports(bench) + + +def test_source_probe_respects_launcher_runtime_before_candidate_overrides(tmp_path): + target = _package(tmp_path / "candidate", "patched") + _package(tmp_path / "launcher-install", "original") + bench = _benchmark() + bench["envs"]["PYTHONPATH"] = str(target.parent.parent) + bench["workload_spec"]["server_launch"]["runtime_environment"] = {"PYTHONPATH": str(tmp_path / "launcher-install")} + apply_native_candidate(bench, source_files=source_file_hashes([target])) + with pytest.raises(ValueError, match="not imported"): + verify_native_source_imports(bench) + apply_native_candidate(bench, runtime_override={"pythonpath_prefix": str(target.parent.parent)}) + verify_native_source_imports(bench) + + +@pytest.mark.parametrize("fails_during_startup", [False, True]) +def test_overlay_source_probe_requires_successful_startup_import(tmp_path, fails_during_startup): + overlay = tmp_path / "overlay" + overlay.mkdir() + source = 'raise RuntimeError("overlay startup failed")\n' if fails_during_startup else "LOADED = True\n" + (overlay / "sitecustomize.py").write_text(source, encoding="utf-8") + bench = _benchmark() + apply_native_candidate(bench, overlay_pythonpath=str(overlay)) + + if fails_during_startup: + with pytest.raises(ValueError, match="overlay did not load successfully"): + verify_native_source_imports(bench) + assert "source_imports" not in bench["workload_spec"] + else: + verify_native_source_imports(bench) + assert bench["workload_spec"]["source_imports"]["sitecustomize"]["loaded"] is True + + +def test_source_transaction_attests_writes_and_deletions(tmp_path): + root = tmp_path / "source" + root.mkdir() + kept = root / "keep.py" + kept.write_text("VALUE = 2\n", encoding="utf-8") + patch = tmp_path / "applied.patch" + patch.write_text( + "diff --git a/keep.py b/keep.py\n--- a/keep.py\n+++ b/keep.py\n@@ -1 +1 @@\n-VALUE = 1\n+VALUE = 2\n" + "diff --git a/old.py b/old.py\ndeleted file mode 100644\n--- a/old.py\n+++ /dev/null\n@@ -1 +0,0 @@\n-old\n", + encoding="utf-8", + ) + evidence = applied_source_evidence(root, [patch]) + assert evidence["source_files"] == {str(kept): hashlib.sha256(kept.read_bytes()).hexdigest()} + assert evidence["absent_source_files"] == [str(root / "old.py")] + (root / "old.py").write_text("not deleted\n", encoding="utf-8") + with pytest.raises(ValueError, match="deletion was not applied"): + applied_source_evidence(root, [patch]) + + +def test_grid_uses_native_override_without_retargeting_benchmark(tmp_path, monkeypatch): + bench = _benchmark() + base = tmp_path / "baseline.yaml" + base.write_text(yaml.safe_dump({"benchmark": bench}), encoding="utf-8") + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", + lambda benchmark, **kwargs: benchmark["workload_spec"]["resolved_topology"], + ) + variant = GridVariant("candidate", extra_server_args="--mem-fraction-static 0.72", extra_envs={"SGLANG_TEST": "1"}) + variant.runtime_override = {"runtime_python_exe": sys.executable} + output = _build_variant_yaml(base, "", variant, output_subdir=tmp_path / "candidate") + actual = yaml.safe_load(output.read_text(encoding="utf-8"))["benchmark"] + assert actual["envs"] == bench["envs"] + assert actual["agentx"]["launch_overrides"]["append_args"] == ["--mem-fraction-static", "0.72"] + assert actual["agentx"]["launch_overrides"]["env"] == {"SGLANG_TEST": "1"} + assert actual["agentx"]["launch_overrides"]["executable"] == sys.executable + + +def test_native_grid_restores_snapshot_without_reapplying_display_stack(tmp_path, monkeypatch): + bench = _benchmark() + apply_native_candidate(bench, extra_server_args='--json \'{"mode": "accepted"}\'') + snapshot = copy.deepcopy(bench["agentx"]["launch_overrides"]) + base = tmp_path / "baseline.yaml" + base.write_text(yaml.safe_dump({"benchmark": bench})) + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", + lambda benchmark, **kw: benchmark["workload_spec"]["resolved_topology"], + ) + output = _build_variant_yaml( + base, + '--json \'{"mode": "accepted"}\'', + GridVariant("next", extra_server_args="--mem-fraction-static 0.75"), + output_subdir=tmp_path / "next", + base_args_mode="replace", + base_extra_envs={"SGLANG_DUPLICATE": "1"}, + base_native_launch_overrides=snapshot, + ) + actual = yaml.safe_load(output.read_text())["benchmark"]["agentx"]["launch_overrides"] + assert actual["append_args"] == ["--json", '{"mode": "accepted"}', "--mem-fraction-static", "0.75"] + assert actual["replace_args"] is False + assert "SGLANG_DUPLICATE" not in actual["env"] + + +def test_native_warm_replay_attests_kernel_and_final_recipe_layer(tmp_path): + source = _package(tmp_path / "source", "recreated by recipe") + manifest = tmp_path / "kernel-manifest.json" + manifest.write_text(json.dumps({"source_backups": [{"path": str(source), "disposition": "deleted"}]})) + patches = tmp_path / "warm_patches" + patches.mkdir() + (patches / "000_recreate.diff").write_text( + "--- /dev/null\n+++ b/sglang/candidate.py\n@@ -0,0 +1 @@\n+VALUE = 'recreated by recipe'\n" + ) + params = { + "warm_kernel_apply_results": [{"manifest_path": str(manifest)}], + "patches": [{"patch_ref": "recreate.patch"}], + "_warm_patch_statuses": [ + {"status": "applied", "patch_ref": "recreate.patch", "target_repo": str(source.parent.parent)} + ], + } + assert warm_source_evidence(params, tmp_path) == { + "source_files": source_file_hashes([source]), + "absent_source_files": [], + } + + +def test_verified_launch_evidence_survives_config_roundtrip(tmp_path, monkeypatch): + monkeypatch.setattr("hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", lambda *a, **kw: None) + path = tmp_path / "native.yaml" + path.write_text(yaml.safe_dump({"benchmark": _benchmark()}), encoding="utf-8") + evidence = {"base_argv": [sys.executable, "-m", "sglang.launch_server"], "evidence_sha256": "a" * 64} + record_launch_evidence( + path, {"valid_measurement": False, "agentx_launch_contract": 1, "agentx_server_launch": evidence} + ) + assert yaml.safe_load(path.read_text())["benchmark"]["workload_spec"]["server_launch"] != evidence + record_launch_evidence( + path, + { + "valid_measurement": True, + "agentx_launch_contract": 1, + "agentx_server_launch": evidence, + "agentx_candidate_fingerprint": "a" * 64, + }, + ) + update_native_candidate_file(path, extra_server_args="--disable-cuda-graph") + assert yaml.safe_load(path.read_text())["benchmark"]["workload_spec"]["server_launch"] == evidence + + +@pytest.mark.asyncio +@pytest.mark.parametrize("correct_import", [True, False]) +async def test_epoch_three_baseline_applies_and_attests_warm_source_transaction(tmp_path, monkeypatch, correct_import): + import subprocess + + from hyperloom.orchestrator.actions.executors.baseline import BaselineExecutor + from hyperloom.orchestrator.loop.sub_agent_runner import RunnerContext + from hyperloom.orchestrator.state.shared_state import SharedState + from hyperloom.orchestrator.state.task_registry import Task + + target = _package(tmp_path / "candidate", "before") + root = target.parent.parent + other = _package(tmp_path / "other", "unmodified") + import_root = root if correct_import else other.parent.parent + subprocess.run(["git", "init", "-q", str(root)], check=True) + subprocess.run(["git", "-C", str(root), "add", "."], check=True) + subprocess.run( + ["git", "-C", str(root), "-c", "user.name=Test", "-c", "user.email=test@example.com", "commit", "-qm", "base"], + check=True, + ) + patch = tmp_path / "change.patch" + patch.write_text( + "--- a/sglang/candidate.py\n+++ b/sglang/candidate.py\n@@ -1 +1 @@\n-VALUE = 'before'\n+VALUE = 'after'\n", + encoding="utf-8", + ) + bench = _benchmark() + bench["envs"]["PYTHONPATH"] = str(root) + base = tmp_path / "baseline.yaml" + base.write_text(yaml.safe_dump({"benchmark": bench}), encoding="utf-8") + monkeypatch.setenv("AGENTX_MODEL_ID", bench["model"]) + monkeypatch.setenv("AGENTX_SERVER_SCRIPT", bench["benchmark_script"]) + monkeypatch.setenv("INFERENCEX_PATH", str(tmp_path / "ix")) + monkeypatch.setenv("TP", "4") + monkeypatch.setenv("CONC", "8") + state = SharedState( + benchmark_mode="agentx", + agentx_epoch=3, + agentx_backend="native", + model_path="/models/glm", + ) + executor = BaselineExecutor(default_config_path=base, session_dir=tmp_path, shared_state=state) + captured = {} + + def resolve(benchmark, **kwargs): + benchmark.setdefault("workload_spec", {})["resolved_topology"] = copy.deepcopy( + bench["workload_spec"]["resolved_topology"] + ) + return benchmark["workload_spec"]["resolved_topology"] + + async def measure(**kwargs): + actual = yaml.safe_load(kwargs["config_path"].read_text())["benchmark"] + captured.update(actual) + assert target.read_text() == "VALUE = 'after'\n" + return {"status": "succeeded", "output_throughput": 100.0, "submission_valid": True} + + async def cleanup(**kwargs): + return None + + monkeypatch.setattr("hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", resolve) + monkeypatch.setattr( + "hyperloom.orchestrator.actions.executors.baseline.prepare_agentx_runtime", lambda **kwargs: None + ) + monkeypatch.setattr(executor, "_run_single_benchmark", measure) + monkeypatch.setattr(executor, "_pre_start_cleanup", cleanup) + task = Task( + task_id="warm", + kind="replay_warm_recipe", + state="running", + idempotency_key="warm", + params={ + "output_dir": str(tmp_path / "run"), + "extra_server_args": "--mem-fraction-static 0.73", + "runtime_override": {"runtime_python_exe": sys.executable, "pythonpath_prefix": str(import_root)}, + "patches": [{"framework_root": str(root), "patch_file": "change.patch", "patch_ref": str(patch)}], + "required_patch_timeline": True, + }, + ) + ctx = RunnerContext(task=task, lease=None, extra={"shared_state": state}) + result = await executor(ctx) + if not correct_import: + assert result["status"] == "failed" + assert result["error_class"] == "native_source_attestation_failed" + assert result["rollback"]["ok"] is True + assert target.read_text() == "VALUE = 'before'\n" + assert not captured + return + assert result["status"] == "succeeded" + override = captured["agentx"]["launch_overrides"] + assert override["source_files"][str(target)] == hashlib.sha256(target.read_bytes()).hexdigest() + assert override["append_args"] == ["--mem-fraction-static", "0.73"] + assert captured["workload_spec"]["source_imports"]["sglang"]["origin"] == str(root / "sglang/__init__.py") + + +def test_grid_rejects_environment_previously_dropped_by_generic_filter(tmp_path): + bench = _benchmark() + base = tmp_path / "baseline.yaml" + base.write_text(yaml.safe_dump({"benchmark": bench}), encoding="utf-8") + variant = GridVariant("unsafe", extra_envs={"PYTHONPATH": "/not-an-attested-runtime"}) + with pytest.raises(ValueError, match="unsupported environment"): + _build_variant_yaml(base, "", variant, output_subdir=tmp_path / "candidate") + + +def test_native_profile_replays_observed_flags_and_runtime(tmp_path): + from types import SimpleNamespace + + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + from hyperloom.orchestrator.actions.executors._native_profile import project_native_profile + + bench = _benchmark() + evidence = { + "effective_argv": [ + sys.executable, + "-m", + "sglang.launch_server", + "--model-path", + "/models/glm", + "--port", + "8000", + "--tp", + "4", + "--mem-fraction-static", + "0.73", + ], + "runtime_environment": {"PATH": os.environ["PATH"], "PYTHONPATH": str(tmp_path), "SGLANG_TEST": "1"}, + } + evidence["evidence_sha256"] = canonical_sha256(evidence) + bench["workload_spec"]["server_launch"] = evidence + source = tmp_path / "accepted.yaml" + source.write_text(yaml.safe_dump({"benchmark": bench}), encoding="utf-8") + state = SimpleNamespace(current_best_measurement={}, baseline_config_path=str(source)) + params = {"extra_envs": {"PROFILE": "1"}} + project_native_profile(params, state) + assert params["extra_server_args"] == "--tp 4 --mem-fraction-static 0.73" + assert params["extra_envs"]["PROFILE"] == "1" + assert params["extra_envs"]["SGLANG_TEST"] == "1" + assert params["runtime_override"]["runtime_python_exe"] == sys.executable + assert params["runtime_override"]["pythonpath_prefixes"] == [str(tmp_path)] diff --git a/src/hyperloom/inference_optimizer/tests/test_native_candidate_boundary.py b/src/hyperloom/inference_optimizer/tests/test_native_candidate_boundary.py new file mode 100644 index 0000000000..7d8dc2bba8 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_candidate_boundary.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Candidate updates must survive the same final launch authority as a baseline.""" + +from __future__ import annotations + +import sys + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx import native +from hyperloom.inference_optimizer.agentx.identity import canonical_sha256, native_workload_fingerprint +from hyperloom.inference_optimizer.agentx.native import resolve_native_recipe +from hyperloom.inference_optimizer.agentx.runtime import maybe_prepare_agentx +from hyperloom.inference_optimizer.tests.test_agentx_runtime import _pinned_native_cfg +from hyperloom.orchestrator.actions.executors._native_candidate import update_native_candidate_file + + +@pytest.mark.parametrize("candidate_kind", ["runtime", "source"]) +def test_post_materialization_candidate_refreshes_execution_and_passes_final_boundary( + tmp_path, monkeypatch, candidate_kind +): + config_path, pins = _pinned_native_cfg(tmp_path, monkeypatch) + data = yaml.safe_load(config_path.read_text()) + bench = data["benchmark"] + bench["agentx"] = {"enabled": True, "launch_overrides": {"version": 1}} + bench["inferencex_path"] = str(tmp_path) + bench["envs"]["ROCR_VISIBLE_DEVICES"] = "0,1,2,3" + bench["gpu_selection"] = {"auto": False} + + # The external recipe/source resolver is represented by a deterministic + # identity over its returned config; both real Hyperloom boundary checks run. + def execution_identity(**kwargs): + current = kwargs["resolved_benchmark"] + return { + "inferencex_commit": "d" * 40, + "magpie_commit": "c" * 40, + "static_execution_fingerprint": "b" * 64, + "execution_fingerprint": canonical_sha256(current["agentx"]), + "workload_fingerprint": native_workload_fingerprint(current, "b" * 64), + } + + monkeypatch.setattr("hyperloom.inference_optimizer.agentx.native.native_execution_identity", execution_identity) + original_preview = native.preview_native_recipe + + def with_recipe_fingerprint(current, **kwargs): + result = original_preview(current, **kwargs) + result["entry"]["recipe-fingerprint"] = "a" * 64 + return result + + monkeypatch.setattr(native, "preview_native_recipe", with_recipe_fingerprint) + pins.pop("HYPERLOOM_AGENTX_EXPECTED_MATERIALIZED_EXECUTION_FINGERPRINT") + for name, value in pins.items(): + monkeypatch.setenv(name, value) + monkeypatch.setenv("HYPERLOOM_IMAGE", "example/sglang:agentx") + monkeypatch.setenv("HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT", "") + resolve_native_recipe(bench, inferencex_path=tmp_path, expected_gpu_count=4) + initial_execution = dict(bench["workload_spec"]["execution"]) + pins["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] = initial_execution["workload_fingerprint"] + monkeypatch.setenv( + "HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT", pins["HYPERLOOM_AGENTX_EXPECTED_WORKLOAD_FINGERPRINT"] + ) + config_path.write_text(yaml.safe_dump(data)) + + candidate = {"runtime_override": {"runtime_python_exe": sys.executable}} + if candidate_kind == "source": + source_root = tmp_path / "source" + package = source_root / "sglang" + package.mkdir(parents=True) + init = package / "__init__.py" + init.write_text("") + module = package / "kernel.py" + module.write_text("OPTIMIZED = True\n") + from hyperloom.orchestrator.actions.executors._native_source import source_file_hashes + + candidate["runtime_override"]["pythonpath_prefix"] = str(source_root) + candidate["source_files"] = source_file_hashes([module]) + update_native_candidate_file(config_path, **candidate) + actual = yaml.safe_load(config_path.read_text())["benchmark"] + assert actual["workload_spec"]["execution"]["execution_fingerprint"] != initial_execution["execution_fingerprint"] + assert actual["workload_spec"]["execution"]["workload_fingerprint"] == initial_execution["workload_fingerprint"] + assert maybe_prepare_agentx(env=dict(pins), inferencex_path=str(tmp_path), config_path=config_path) is True + + # Updating the payload without refreshing its accepted receipt still fails. + actual["agentx"]["launch_overrides"]["env"]["SGLANG_USE_AITER"] = "1" + config_path.write_text(yaml.safe_dump({"benchmark": actual})) + with pytest.raises(ValueError, match="identity changed after materialization"): + maybe_prepare_agentx(env=dict(pins), inferencex_path=str(tmp_path), config_path=config_path) diff --git a/src/hyperloom/inference_optimizer/tests/test_native_explore.py b/src/hyperloom/inference_optimizer/tests/test_native_explore.py new file mode 100644 index 0000000000..3c77e6ca74 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_explore.py @@ -0,0 +1,115 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native explore must retain launch identity while stacking canonical wins.""" + +from __future__ import annotations + +import shlex + +import pytest +import yaml + +from hyperloom.inference_optimizer.tests.test_native_candidate import _benchmark +from hyperloom.orchestrator.actions.executors import explore +from hyperloom.orchestrator.actions.executors._grid_runner import VariantResult, _build_variant_yaml +from hyperloom.orchestrator.loop.sub_agent_runner import RunnerContext +from hyperloom.orchestrator.state.shared_state import SharedState +from hyperloom.orchestrator.state.task_registry import Task + + +@pytest.mark.asyncio +async def test_public_native_explore_keeps_comparable_candidates_without_reapplying_snapshot(tmp_path, monkeypatch): + config = tmp_path / "native.yaml" + config.write_text(yaml.safe_dump({"benchmark": _benchmark()})) + state = SharedState(framework="sglang", benchmark_mode="agentx", agentx_epoch=3, agentx_backend="native") + state.baseline_tput = 200.0 + state.baseline_perf = { + "output_throughput": 200.0, + "total_token_throughput": 20000.0, + "e2e_norm_intvty_p90": 300.0, + "e2e_norm_intvty_p50": 300.0, + "duration_seconds": 3600.0, + "request_error_rate": 0.0, + "agentx_launch_contract": 1, + "agentx_workload_fingerprint": "a" * 64, + } + monkeypatch.setattr(explore, "materialize_config_with_envs", lambda *a, **kw: config) + monkeypatch.setattr(explore, "resolve_lifecycle_params", lambda *a: {"eligible": False}) + monkeypatch.setattr(explore, "maybe_serving_lease", lambda **kw: None) + monkeypatch.setattr(explore, "apply_compatibility_filter", lambda rows, **kw: (rows, [])) + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", + lambda benchmark, **kw: benchmark["workload_spec"]["resolved_topology"], + ) + launched = [] + + async def canonical_grid(**kwargs): + variant = kwargs["grid"][0] + path = _build_variant_yaml( + kwargs["base_yaml_path"], + kwargs["base_extra_args"], + variant, + output_subdir=kwargs["output_root"], + base_extra_envs=kwargs["base_extra_envs"], + base_remove_args=kwargs["base_remove_args"], + base_unset_envs=kwargs["base_unset_envs"], + base_args_mode=kwargs["base_args_mode"], + base_native_launch_overrides=kwargs["base_native_launch_overrides"], + ) + snapshot = yaml.safe_load(path.read_text())["benchmark"]["agentx"]["launch_overrides"] + launched.append(snapshot) + return [ + VariantResult( + name=variant.name, + extra_server_args=variant.extra_server_args, + extra_envs=variant.extra_envs, + status="succeeded", + output_throughput=200.0 + len(launched) * 10, + total_token_throughput=22000.0, + intvty_p90=300.0 + len(launched) * 30, + intvty_p50=300.0 + len(launched) * 30, + duration_seconds=3600.0, + request_error_rate=0.0, + materialized_config=str(path), + native_measurement={ + "agentx_launch_contract": 1, + "agentx_workload_fingerprint": "a" * 64, + "agentx_candidate_fingerprint": str(len(launched)) * 64, + }, + ) + ] + + monkeypatch.setattr(explore, "run_grid", canonical_grid) + task = Task( + task_id="native-explore", + kind="explore", + state="running", + idempotency_key="native-explore", + params={ + "config_path": str(config), + "base_tput": 200.0, + "grid": [ + {"name": "json", "extra_args": '--compilation-config \'{"mode": "max-autotune"}\''}, + {"name": "next", "extra_args": "--mem-fraction-static 0.75"}, + ], + }, + ) + result = await explore.ExploreExecutor(session_dir=tmp_path)( + RunnerContext(task=task, lease=None, extra={"shared_state": state}) + ) + assert [winner["name"] for winner in result["winners"]] == ["json", "next"] + assert launched[1]["append_args"] == [ + "--compilation-config", + '{"mode": "max-autotune"}', + "--mem-fraction-static", + "0.75", + ] + assert launched[1]["replace_args"] is False + winner = result["best_variant"] + assert winner["agentx_workload_fingerprint"] == "a" * 64 + assert winner["e2e_norm_intvty_p50"] == 360.0 + assert winner["duration_seconds"] == 3600.0 + assert winner["materialized_config"].endswith("config.yaml") + assert shlex.split(winner["extra_server_args"]) == launched[1]["append_args"] + assert winner["native_launch_overrides"] == launched[1] diff --git a/src/hyperloom/inference_optimizer/tests/test_native_integrate_launch.py b/src/hyperloom/inference_optimizer/tests/test_native_integrate_launch.py new file mode 100644 index 0000000000..3abd036098 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_integrate_launch.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native source handback must benchmark the complete accepted candidate stack.""" + +from __future__ import annotations + +import pytest +import yaml + +from hyperloom.inference_optimizer.tests.test_native_candidate import _benchmark +from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 +from hyperloom.orchestrator.actions.executors import _ray_serving, integrate_patch +from hyperloom.orchestrator.actions.executors._grid_runner import VariantResult, _build_variant_yaml + + +@pytest.mark.asyncio +async def test_native_integration_applies_controls_overlay_and_retains_measured_identity(tmp_path, monkeypatch): + config = tmp_path / "baseline.yaml" + config.write_text(yaml.safe_dump({"benchmark": _benchmark()})) + overlay = tmp_path / "overlay" + overlay.mkdir() + (overlay / "sitecustomize.py").write_text("# accepted kernel overlay\n") + monkeypatch.setattr(integrate_patch, "materialize_config_with_envs", lambda *a, **kw: config) + monkeypatch.setattr(_ray_serving, "maybe_serving_lease", lambda **kw: None) + monkeypatch.setattr( + "hyperloom.inference_optimizer.agentx.native.resolve_native_recipe", + lambda benchmark, **kw: benchmark["workload_spec"]["resolved_topology"], + ) + actual = {} + + async def measured_grid(**kwargs): + variant = kwargs["grid"][0] + candidate_path = _build_variant_yaml( + kwargs["base_yaml_path"], + kwargs["base_extra_args"], + variant, + output_subdir=tmp_path / "candidate", + base_extra_envs=kwargs["base_extra_envs"], + base_args_mode=kwargs["base_args_mode"], + base_remove_args=kwargs["base_remove_args"], + base_unset_envs=kwargs["base_unset_envs"], + ) + actual.update(yaml.safe_load(candidate_path.read_text())["benchmark"]) + overrides = actual["agentx"]["launch_overrides"] + base_argv = actual["workload_spec"]["server_launch"]["base_argv"] + evidence = { + "version": 1, + "framework": "sglang", + "base_argv": base_argv, + "effective_argv": base_argv[:-1] + overrides["append_args"], + "effective_env": {**overrides["env"], **{key: None for key in overrides["unset_env"]}}, + "base_env": {key: None for key in set(overrides["env"]) | set(overrides["unset_env"])}, + "resolved_executable": base_argv[0], + "runtime_environment": dict(overrides["env"]), + "source_files": overrides["source_files"], + "absent_source_files": overrides["absent_source_files"], + "overrides_sha256": canonical_sha256(overrides), + } + evidence["evidence_sha256"] = canonical_sha256(evidence) + return [ + VariantResult( + name=variant.name, + extra_server_args=variant.extra_server_args, + extra_envs=variant.extra_envs, + status="succeeded", + output_throughput=280.0, + materialized_config=str(candidate_path), + native_measurement={ + "agentx_launch_contract": 1, + "agentx_workload_fingerprint": "a" * 64, + "agentx_server_launch": evidence, + }, + ) + ] + + monkeypatch.setattr(integrate_patch, "run_grid", measured_grid) + executor = integrate_patch.IntegratePatchExecutor(session_dir=tmp_path) + bench, _ = await executor._bench_patch( + params={ + "config_path": str(config), + "base_extra_args": "--mem-fraction-static 0.7", + "base_extra_envs": {"SGLANG_TEST_OLD": "1"}, + "remove_args": ["--disable-cuda-graph"], + "unset_envs": ["SGLANG_TEST_OLD"], + "args_mode": "replace", + "overlay_pythonpath": str(overlay), + }, + output_root=tmp_path / "out", + extra_server_args_applied="--mem-fraction-static 0.8", + extra_envs_applied={"SGLANG_TEST_NEW": "1"}, + specialist_task_id="source-handback", + ) + override = actual["agentx"]["launch_overrides"] + assert override["append_args"] == ["--mem-fraction-static", "0.8"] + assert override["remove_args"] == ["--disable-cuda-graph"] + assert override["replace_args"] is True + assert override["unset_env"] == ["SGLANG_TEST_OLD"] + assert override["env"]["PYTHONPATH"].split(":")[0] == str(overlay) + assert override["env"]["SGLANG_TEST_NEW"] == "1" + assert str(overlay / "sitecustomize.py") in override["source_files"] + assert bench["materialized_config"] == str(tmp_path / "candidate/config.yaml") + assert bench["agentx_workload_fingerprint"] == "a" * 64 + assert bench["effective_config"]["args_mode"] == "replace" + assert bench["effective_config"]["final_overlay"] == str(overlay) diff --git a/src/hyperloom/inference_optimizer/tests/test_native_launch_boundary.py b/src/hyperloom/inference_optimizer/tests/test_native_launch_boundary.py new file mode 100644 index 0000000000..82a58be0b7 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_launch_boundary.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Validate the final launch inputs after watchdog preparation.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest +import yaml + +from hyperloom.inference_optimizer.agentx.identity import native_workload_fingerprint, verified_workload_fingerprint +from hyperloom.orchestrator.actions.executors import baseline +from hyperloom.orchestrator.loop.sub_agent_runner import RunnerContext +from hyperloom.orchestrator.state.task_registry import Task + + +@pytest.mark.asyncio +@pytest.mark.parametrize("native", [False, True]) +@pytest.mark.parametrize("run_eval_disabled", [False, True]) +async def test_baseline_checks_final_watchdog_and_process_inputs_before_spawn( + tmp_path, monkeypatch, native, run_eval_disabled +): + monkeypatch.setenv("INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC", "7800") + benchmark = {"framework": "sglang", "timeout_seconds": 7200, "envs": {"CONC": 8}} + if native: + benchmark["agentx"] = {"enabled": True, "launch_overrides": {"version": 1}} + benchmark["workload_spec"] = { + "execution": { + "static_execution_fingerprint": "a" * 64, + "workload_fingerprint": native_workload_fingerprint(benchmark, "a" * 64), + } + } + config = tmp_path / "benchmark.yaml" + config.write_text(yaml.safe_dump({"benchmark": benchmark})) + original = config.read_bytes() + output = tmp_path / "output" + checkout = str(tmp_path / "InferenceX") + checked = [] + + def prepare(**kwargs): + env = kwargs["env"] + assert env["PYTHONUNBUFFERED"] == "1" + assert env["MAGPIE_INFERENCEX_PATH"] == checkout + assert env["RESULT_DIR"] == str(output) + assert env["SERVER_LOG"] == str(output / "server.log") + assert ("EVAL_RESULT_DIR" in env) is not native + materialized = yaml.safe_load(kwargs["config_path"].read_text())["benchmark"] + if native: + assert config.read_bytes() == original + assert verified_workload_fingerprint(materialized) is not None + else: + assert materialized["timeout_seconds"] == 7800.0 + assert materialized["envs"]["PYTHONUNBUFFERED"] == "1" + checked.append(dict(env)) + + def spawn(cmd, **kwargs): + assert checked == [kwargs["env"]] + assert kwargs["timeout"] == 7800.0 + assert Path(cmd[cmd.index("--benchmark-config") + 1]) == config + raise RuntimeError("verified process boundary reached") + + monkeypatch.setattr("hyperloom.inference_optimizer.agentx.runtime.maybe_prepare_agentx", prepare) + monkeypatch.setattr(baseline, "run_with_session_kill", spawn) + executor = baseline.BaselineExecutor(magpie_python=sys.executable, session_dir=tmp_path) + monkeypatch.setattr(executor, "_preflight_server_argv", lambda **kwargs: None) + task = Task(task_id="boundary", kind="baseline", state="running", idempotency_key="boundary", params={}) + context = RunnerContext(task=task, lease=None, extra={"mn_round_restarted": True}) + + with pytest.raises(RuntimeError, match="verified process boundary reached"): + await executor._run_single_benchmark( + config_path=config, + output_dir=output, + timeout_sec=7200, + override_result_dir=None, + resolved_model="example/model", + materialized_config_path=config, + inferencex_path=checkout, + effective_extra_server_args="", + params={}, + ctx=context, + run_eval_disabled=run_eval_disabled, + server_already_ready=True, + ) diff --git a/src/hyperloom/inference_optimizer/tests/test_native_throughput_display.py b/src/hyperloom/inference_optimizer/tests/test_native_throughput_display.py new file mode 100644 index 0000000000..8a7b09c3ba --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_native_throughput_display.py @@ -0,0 +1,173 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Native totals keep their scoring value while displays use measured GPU counts.""" + +from __future__ import annotations + +from copy import deepcopy + +import pytest + +from hyperloom.inference_optimizer.breakdown.collectors.v6 import collect_v6_outcome +from hyperloom.inference_optimizer.breakdown.exporter import write_minimal_final_report +from hyperloom.inference_optimizer.breakdown.recorder.baseline_event import baseline_event_id, make_baseline_recorder +from hyperloom.inference_optimizer.breakdown.recorder.event_sink import make_sink +from hyperloom.inference_optimizer.breakdown.reporters._renderers import baseline, final +from hyperloom.inference_optimizer.breakdown.reporters.cross_section import build_global_facts +from hyperloom.inference_optimizer.cli.bootstrap import _print_final_summary +from hyperloom.inference_optimizer.performance_display import ( + format_measurement_metric, + format_session_metric, + throughput_fields, +) +from hyperloom.inference_optimizer.session.sbd_v6 import read_timeline_events +from hyperloom.inference_optimizer.session.session_binding import session_scope +from hyperloom.orchestrator.actions.executors import report +from hyperloom.orchestrator.state.shared_state import SharedState + +AGGREGATE = 286.78509 + + +def _native_state(gpu_count=4): + perf = {} if gpu_count is None else {"agentx_gpu_count": gpu_count} + return SharedState( + session_id="native-display", + framework="sglang", + benchmark_mode="agentx", + agentx_epoch=2, + tp=4, + ep=4, + baseline_tput=AGGREGATE, + baseline_perf=perf, + current_best={"tput": AGGREGATE, "action": "baseline", **perf}, + ) + + +@pytest.mark.parametrize("gpu_count,expected", [(4, "71.69627 tok/s/GPU"), (None, "286.78509 tok/s (aggregate)")]) +def test_native_display_uses_physical_count_without_changing_measurements(gpu_count, expected, monkeypatch): + state = _native_state(gpu_count) + # TP and EP overlap on the same four GPUs; ambient topology must never override a recorded measurement. + monkeypatch.setenv("TP", "16") + monkeypatch.setenv("HYPERLOOM_AGENTX_GPU_COUNT", "16") + result = {"output_throughput": AGGREGATE, "native_agentx_report": True, **state.baseline_perf} + before = deepcopy(result) + + assert format_session_metric(state, state.baseline_tput, precision=5) == expected + assert format_measurement_metric("sglang", result, precision=5) == expected + assert result == before + assert state.baseline_tput == AGGREGATE + assert state.current_best["tput"] == AGGREGATE + + +@pytest.mark.parametrize("count", [0, -4, 1.5, True, "4", float("inf"), float("nan")]) +def test_native_invalid_topology_remains_explicitly_aggregate(count): + fields = throughput_fields(AGGREGATE, "sglang", native_agentx=True, gpu_count=count) + assert fields["throughput_tok_s_per_gpu"] is None + assert fields["throughput_tok_s"] == AGGREGATE + assert fields["throughput_unit"] == "tok/s (aggregate)" + + +@pytest.mark.parametrize("mode,epoch", [("agentx", 1), ("synthetic", 0)]) +def test_legacy_and_synthetic_display_keep_the_existing_throughput(mode, epoch): + state = _native_state() + state.benchmark_mode, state.agentx_epoch = mode, epoch + assert format_session_metric(state, AGGREGATE, precision=2) == "286.79 tok/s/GPU" + assert format_measurement_metric("sglang", {"output_throughput": AGGREGATE, "agentx_gpu_count": 4}) == ( + "286.8 tok/s/GPU" + ) + + +def test_scriptable_primary_metric_stays_latency(): + state = SharedState(session_id="image-display", framework="xdit", baseline_tput=2.0) + assert format_session_metric(state, 2.0) == "500.0 ms" + + +@pytest.mark.parametrize("gpu_count,expected", [(4, "71.7 tok/s/GPU"), (None, "286.8 tok/s (aggregate)")]) +def test_native_cli_mission_and_final_reports_share_correct_units(tmp_path, capsys, gpu_count, expected): + state = _native_state(gpu_count) + state.save(tmp_path) + _print_final_summary(state, "time_exhausted", tmp_path) + assert expected in capsys.readouterr().out + assert expected in state.to_mission_summary() + assert expected in state._format_current_best_for_mission() + + summary = report._build_summary_dict(state, {}, []) + assert summary["baseline_tput"] == AGGREGATE + assert summary["current_best"]["tput"] == AGGREGATE + assert report._format_md(summary).count(expected) == 2 + emergency = write_minimal_final_report(tmp_path).read_text() + expected_precise = "71.70 tok/s/GPU" if gpu_count else "286.79 tok/s (aggregate)" + assert emergency.count(expected_precise) == 2 + + +@pytest.mark.parametrize("gpu_count", [4, None]) +def test_native_breakdown_records_and_renders_matching_units(tmp_path, gpu_count): + state = _native_state(gpu_count) + result = { + "status": "succeeded", + "output_throughput": AGGREGATE, + "native_agentx_report": True, + **state.baseline_perf, + } + with session_scope(tmp_path): + recorder = make_baseline_recorder( + make_sink(baseline_event_id("prelude", 0), producer="orchestrator"), + task_id="baseline", + task_kind="baseline", + framework="sglang", + establishes_quality_ref=True, + ) + recorder.finish(result) + timeline = read_timeline_events(tmp_path) + timeline.append( + { + "type": "stack", + "ext": { + "validations": {"settled": {"perf": state.baseline_perf}, "at_head": True}, + "validated_total_gain_pct": 0.0, + }, + } + ) + outcome = collect_v6_outcome( + session={"stop_reason": "time_exhausted"}, + close={"final_recipe": {"throughput": AGGREGATE}}, + state=state.to_dict(), + timeline=timeline, + ) + expected = "71.70 tok/s/GPU" if gpu_count else "286.79 tok/s (aggregate)" + for key in ("baseline", "final"): + measurement = outcome[key] + assert measurement["throughput_tok_s"] == AGGREGATE + if gpu_count: + assert measurement["throughput_tok_s_per_gpu"] == pytest.approx(AGGREGATE / 4) + assert measurement["perf"]["agentx_gpu_count"] == 4 + else: + assert measurement["throughput_tok_s_per_gpu"] is None + breakdown = { + "metadata": {"task_config": {"framework_name": "sglang"}}, + "timeline": timeline, + "outcome": outcome, + } + rendered = [baseline.render(breakdown), final.render(breakdown)] + assert all(any(expected in fact for fact in section.key_facts) for section in rendered) + assert expected in rendered[0].markdown_block # The attempts table states each row's units too. + assert build_global_facts(breakdown, rendered).headline.count(expected) == 2 + assert result["output_throughput"] == AGGREGATE + + +def test_breakdown_headline_does_not_equate_different_units_or_interactivity_gain(): + baseline_fields = throughput_fields(AGGREGATE, "sglang", native_agentx=True, gpu_count=4) + final_fields = throughput_fields(AGGREGATE, "sglang", native_agentx=True) + breakdown = { + "metadata": {"task_config": {"framework_name": "sglang"}}, + "outcome": { + "baseline": baseline_fields, + "final": {**final_fields, "gain_pct": 3.0, "graded_on": "e2e_norm_intvty_p50"}, + }, + } + headline = build_global_facts(breakdown, []).headline + assert "71.70 tok/s/GPU" in headline + assert "286.79 tok/s (aggregate)" in headline + assert "= +3.00%" not in headline + assert "validated gain: +3.00% (e2e_norm_intvty_p50)" in headline diff --git a/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py b/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py index 5be6135b78..f6a9b3dbbc 100644 --- a/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py +++ b/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py @@ -9,9 +9,10 @@ import json import os import subprocess +import threading from pathlib import Path from types import SimpleNamespace -from unittest.mock import patch +from unittest.mock import Mock, patch import pytest @@ -143,6 +144,73 @@ def _isolate_leak_root(tmp_path_factory, monkeypatch): # ProfileExecutor +def _agentx_profile_state( + tmp_path: Path, + *, + tp: int = 2, + pp: int = 1, + pcp_size: int = 1, + include_topology: bool = True, +) -> SimpleNamespace: + """Build the accepted-config state required by AgentX profile tests.""" + import yaml + + workload_spec: dict[str, object] = {} + if include_topology: + workload_spec["resolved_topology"] = { + "tp": tp, + "pp": pp, + "pcp_size": pcp_size, + "gpu_count": tp * pp * pcp_size, + } + config_path = tmp_path / "accepted-agentx-baseline.yaml" + config_path.write_text( + yaml.safe_dump( + { + "benchmark": { + "agentx": {"enabled": True}, + "workload_spec": workload_spec, + } + } + ), + encoding="utf-8", + ) + return SimpleNamespace( + benchmark_mode="agentx", + baseline_config_path=str(config_path), + ) + + +def _init_profile_inferencex_checkout(path: Path) -> str: + """Create the minimal tracked checkout ProfileExecutor validates.""" + benchmark_lib = path / "benchmarks" / "benchmark_lib.sh" + benchmark_lib.parent.mkdir(parents=True) + benchmark_lib.write_text("#!/bin/sh\n", encoding="utf-8") + subprocess.run(["git", "init", "-q", str(path)], check=True) + subprocess.run(["git", "-C", str(path), "add", "."], check=True) + subprocess.run( + [ + "git", + "-C", + str(path), + "-c", + "user.name=Hyperloom Test", + "-c", + "user.email=hyperloom@example.invalid", + "commit", + "-qm", + "profile fixture", + ], + check=True, + ) + return subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + def test_profile_yaml_has_torch_profiler_enabled(): """The whole point of the profile config is profiler ON.""" import yaml @@ -887,9 +955,14 @@ def test_materialize_profile_agentx_clamp_warns_below_steady_floor( _clear_workload_env(monkeypatch) monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "example/agentx-model") src = _profile_yaml(tmp_path, "vllm", {"CONC": 32, "ISL": 256, "OSL": 1024}) with caplog.at_level("WARNING"): - out = _materialize_config_with_envs(src, tmp_path) + out = _materialize_config_with_envs( + src, + tmp_path, + allow_agentx_profile_compat=True, + ) rendered = yaml.safe_load(out.read_text()) extra = rendered["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] assert "--profiler-config.max_iterations 8" in extra, extra @@ -906,10 +979,15 @@ def test_materialize_profile_agentx_clamp_warns_on_explicit_override( _clear_workload_env(monkeypatch) monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "example/agentx-model") monkeypatch.setenv("HYPERLOOM_PROFILE_MAX_STEPS_CAP", "64") src = _profile_yaml(tmp_path, "vllm", {"CONC": 32, "ISL": 256, "OSL": 1024}) with caplog.at_level("WARNING"): - out = _materialize_config_with_envs(src, tmp_path) + out = _materialize_config_with_envs( + src, + tmp_path, + allow_agentx_profile_compat=True, + ) rendered = yaml.safe_load(out.read_text()) extra = rendered["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] assert "--profiler-config.max_iterations 8" in extra, extra @@ -926,10 +1004,15 @@ def test_materialize_profile_max_iters_override_warns_it_undoes_the_agentx_bound _clear_workload_env(monkeypatch) monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "example/agentx-model") monkeypatch.setenv("HYPERLOOM_PROFILE_MAX_ITERS", "128") src = _profile_yaml(tmp_path, "vllm", {"CONC": 32, "ISL": 256, "OSL": 1024}) with caplog.at_level("WARNING"): - out = _materialize_config_with_envs(src, tmp_path) + out = _materialize_config_with_envs( + src, + tmp_path, + allow_agentx_profile_compat=True, + ) rendered = yaml.safe_load(out.read_text()) extra = rendered["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] # The override is still honored verbatim; this is a visibility fix only. @@ -1266,7 +1349,7 @@ def test_instrumentation_preflight_skips_without_an_envs_block(tmp_path): assert "benchmark.envs" in row["skip_reason"] -def test_trace_certificate_stays_out_of_the_resolver_namespace(tmp_path): +def test_trace_certificate_stays_out_of_the_resolver_namespace(tmp_path, monkeypatch): """The certificate must not become a trace candidate for the directory it describes. ``_trace_candidates`` rglobs the trace dir for anything ending in ``_TRACE_EXTS``, and a bare ``.json`` is in @@ -2182,7 +2265,407 @@ async def _fake_baseline(_self, _ctx): @pytest.mark.asyncio -async def test_agentx_profile_executor_passes_rank_zero_not_merged(tmp_path, monkeypatch): +@pytest.mark.parametrize( + "state_kwargs", + [ + pytest.param({"pp": 2}, id="pipeline-parallel"), + pytest.param({"pcp_size": 2}, id="prefill-context-parallel"), + pytest.param({"include_topology": False}, id="missing-topology"), + ], +) +async def test_agentx_profile_rejects_incompatible_topology_before_side_effects( + tmp_path, + monkeypatch, + state_kwargs, +): + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + parent_called = False + + async def _fake_baseline(_self, _ctx): + nonlocal parent_called + parent_called = True + return {"status": "succeeded"} + + monkeypatch.setattr(BaselineExecutor, "__call__", _fake_baseline) + output_dir = tmp_path / "must-not-be-created" + pe = ProfileExecutor(session_dir=tmp_path / "ignored-root") + pe.shared_state = _agentx_profile_state(tmp_path, **state_kwargs) + ctx = SimpleNamespace( + task=SimpleNamespace( + params={"output_dir": str(output_dir)}, + task_id="t-agentx-incompatible-profile", + ), + extra=None, + ) + + result = await pe(ctx) + + assert result["status"] == "failed" + assert result["error_class"] == "agentx_profile_topology_incompatible" + assert result["trace_input_ready"] is False + assert parent_called is False + assert not output_dir.exists() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("native", [False, True, "optimization"], ids=["legacy", "native", "native-optimization"]) +async def test_profile_executor_preserves_session_identity_with_generic_template(tmp_path, monkeypatch, native): + import yaml + + from hyperloom.inference_optimizer.agentx.runtime import _profile_compatibility_checkout + from hyperloom.orchestrator.actions.executors import _workload_envs + from hyperloom.orchestrator.actions.executors import profile as profile_mod + + source = tmp_path / "inferencex" + head = _init_profile_inferencex_checkout(source) + model = tmp_path / "local-model" + model.mkdir() + for name, value in { + "USER_DATA_PATH": str(tmp_path), + "FRAMEWORK": "sglang", + "MODEL_PATH": str(model), + "GPU_TYPE": "mi355x", + "TP": "2", + "CONC": "1", + "INFERENCEX_PATH": str(source), + "INFERENCEX_REF": head, + "HYPERLOOM_AGENTX": "1", + "HYPERLOOM_ENABLE_PATCH": "1", + "HYPERLOOM_SGLANG_SHAPE_MODE": "patch", + "AGENTX_MODEL_ID": "amd/GLM-5.2-MXFP4", + "AGENTX_SERVER_SCRIPT": "single_node/agentic/glm.sh" if native else "legacy_server.sh", + }.items(): + monkeypatch.setenv(name, value) + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + if native: + for name, value in { + "MAGPIE_REF": "a" * 40, + "HYPERLOOM_AGENTX_EXPECTED_RECIPE_FINGERPRINT": "recipe", + "HYPERLOOM_AGENTX_EXPECTED_EXECUTION_FINGERPRINT": "execution", + "HYPERLOOM_AGENTX_GPU_COUNT": "2", + }.items(): + monkeypatch.setenv(name, value) + config = tmp_path / "profile.yaml" + config.write_text( + yaml.safe_dump( + { + "benchmark": { + "framework": "sglang", + "model": str(model), + "envs": {"TP": 2, "CONC": 1}, + "profiler": {"torch_profiler": {"enabled": True}}, + } + } + ), + encoding="utf-8", + ) + original_config = config.read_bytes() + framework_patch = Mock(return_value=True) + monkeypatch.setattr(_workload_envs, "ensure_sglang_patched_for_tracelens", framework_patch) + patch_roots = [] + + def patch_checkout(path): + patch_roots.append(Path(path)) + (Path(path) / "benchmarks" / "benchmark_lib.sh").write_text("${NUM_PROMPTS:-$max_concurrency}\n") + serving = profile_mod.benchmark_serving_path_in(Path(path)) + serving.parent.mkdir(parents=True, exist_ok=True) + serving.write_text("PROFILE_EXTRA_BODY\n") + return True + + monkeypatch.setattr(profile_mod, "ensure_benchmark_lib_patched", patch_checkout) + monkeypatch.setattr(profile_mod, "ensure_benchmark_lib_eval_dest_patched", patch_checkout) + monkeypatch.setattr(profile_mod, "ensure_benchmark_serving_patched", patch_checkout) + executor = ProfileExecutor(session_dir=tmp_path) + executor.shared_state = _agentx_profile_state(tmp_path) if native else SimpleNamespace(benchmark_mode="agentx") + if native == "optimization": + import sys + + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + + executor.shared_state.agentx_epoch = 3 + executor.shared_state.agentx_backend = "native" + accepted = Path(executor.shared_state.baseline_config_path) + accepted_config = yaml.safe_load(accepted.read_text()) + evidence = { + "effective_argv": [sys.executable, "-m", "sglang.launch_server", "--mem-fraction-static", "0.73"], + "runtime_environment": {"PATH": os.environ["PATH"], "SGLANG_NATIVE_TEST": "1"}, + } + evidence["evidence_sha256"] = canonical_sha256(evidence) + accepted_config["benchmark"].update(framework="sglang") + accepted_config["benchmark"]["workload_spec"]["server_launch"] = evidence + accepted.write_text(yaml.safe_dump(accepted_config)) + captured = {} + + async def measure(**kwargs): + bench = yaml.safe_load(kwargs["config_path"].read_text(encoding="utf-8"))["benchmark"] + captured.update(bench) + if native: + _profile_compatibility_checkout( + bench, + config_path=kwargs["config_path"], + explicit_inferencex_path=kwargs["inferencex_path"], + env=os.environ, + ) + workspace = kwargs["output_dir"] / "benchmark_sglang_agentx" + traces = workspace / "torch_trace" + traces.mkdir(parents=True) + (traces / "177-TP-0-DECODE.trace.json.gz").write_bytes(b"rank-zero") + Path(bench["envs"]["AGENTX_CAPTURE_STATUS_PATH"]).write_text( + json.dumps({"capture_id": bench["envs"]["AGENTX_CAPTURE_ID"], "status": "succeeded"}), + encoding="utf-8", + ) + return {"status": "succeeded", "framework": "sglang", "workspace": str(workspace), "submission_valid": True} + + monkeypatch.setattr(executor, "_run_single_benchmark", measure) + output = tmp_path / "output" + ctx = SimpleNamespace( + task=SimpleNamespace( + kind="profile", + task_id="profile-identity", + params={ + "config_path": str(config), + "output_dir": str(output), + **({"native_launch_overrides": {"version": 1}} if native == "optimization" else {}), + }, + ), + extra={}, + ) + + result = await executor(ctx) + + assert result["status"] == "succeeded", result + assert config.read_bytes() == original_config + assert captured["benchmark_script"] == "aiperf_client.sh" + assert "agentx" not in captured + if native == "optimization": + assert "--mem-fraction-static 0.73" in captured["envs"]["EXTRA_SGLANG_ARGS"] + assert captured["envs"]["SGLANG_NATIVE_TEST"] == "1" + assert captured["envs"]["HYPERLOOM_FRAMEWORK_PYTHON"] == sys.executable + if native: + assert captured["workload_spec"]["harness"] == "hyperloom-profiler-compat" + assert captured["envs"]["AGENTX_SERVER_SCRIPT"] == "" + assert captured["envs"]["HYPERLOOM_TRACELENS_PATCH_STATUS"] == "not_attempted" + framework_patch.assert_not_called() + assert set(patch_roots) == {output / ".agentx-profile-inferencex"} + assert not subprocess.run( + ["git", "-C", str(source), "status", "--porcelain"], capture_output=True, text=True, check=True + ).stdout.strip() + else: + assert "harness" not in captured["workload_spec"] + assert captured["envs"]["AGENTX_SERVER_SCRIPT"] == "legacy_server.sh" + framework_patch.assert_called_once() + assert set(patch_roots) == {source} + + +@pytest.mark.asyncio +async def test_agentx_profile_checkout_keeps_event_loop_responsive(tmp_path, monkeypatch): + from hyperloom.orchestrator.actions.executors import baseline as baseline_mod + + source = tmp_path / "pinned-inferencex" + head = _init_profile_inferencex_checkout(source) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("INFERENCEX_PATH", str(source)) + monkeypatch.setenv("INFERENCEX_REF", head) + monkeypatch.setenv("FRAMEWORK", "sglang") + config = tmp_path / "profile.yaml" + config.write_text("benchmark:\n envs: {}\n", encoding="utf-8") + monkeypatch.setattr(baseline_mod, "materialize_config_with_envs", lambda *args, **kwargs: config) + executor = ProfileExecutor(session_dir=tmp_path) + executor.shared_state = _agentx_profile_state(tmp_path) + ctx = SimpleNamespace( + task=SimpleNamespace( + kind="profile", + task_id="profile-slow-checkout", + params={"config_path": str(config), "output_dir": str(tmp_path / "output")}, + ), + extra={}, + ) + loop = asyncio.get_running_loop() + clone_started = asyncio.Event() + release_clone = threading.Event() + git_run = subprocess.run + loop_progressed_during_clone = False + + def slow_clone(argv, **kwargs): + nonlocal loop_progressed_during_clone + if argv[:2] == ["git", "clone"]: + loop.call_soon_threadsafe(clone_started.set) + loop_progressed_during_clone = release_clone.wait(timeout=5) + return subprocess.CompletedProcess(argv, 1, "", "simulated clone failure") + return git_run(argv, **kwargs) + + monkeypatch.setattr("hyperloom.orchestrator.actions.executors.profile.subprocess.run", slow_clone) + profile_task = asyncio.create_task(executor(ctx)) + try: + await asyncio.wait_for(clone_started.wait(), timeout=10) + release_clone.set() + result = await asyncio.wait_for(profile_task, timeout=10) + finally: + release_clone.set() + if not profile_task.done(): + profile_task.cancel() + await asyncio.gather(profile_task, return_exceptions=True) + + assert loop_progressed_during_clone, "the event loop was blocked until git clone returned" + assert result["status"] == "failed" + assert result["error_class"] == "agentx_profile_checkout_unavailable" + assert "simulated clone failure" in result["error"] + + +def test_legacy_agentx_profile_checkout_needs_no_native_pin(tmp_path, monkeypatch): + source = tmp_path / "legacy-inferencex" + source.mkdir() + monkeypatch.setenv("INFERENCEX_REF", "native-pin-is-not-relevant") + config = tmp_path / "profile.yaml" + original = "benchmark:\n benchmark_script: aiperf_client.sh\n" + config.write_text(original, encoding="utf-8") + output = tmp_path / "profile-output" + + selected, error = ProfileExecutor(session_dir=tmp_path)._agentx_runtime_checkout( + config_path=config, + output_dir=output, + inferencex_path=str(source), + agentx_session=False, + ) + + assert (selected, error) == (str(source), None) + assert config.read_text(encoding="utf-8") == original + assert not output.exists() + + +def test_agentx_profile_checkout_rejects_symlink_to_pinned_source(tmp_path): + source = tmp_path / "pinned-inferencex" + _init_profile_inferencex_checkout(source) + output_dir = tmp_path / "profile-output" + output_dir.mkdir() + isolated = output_dir / ".agentx-profile-inferencex" + isolated.symlink_to(source, target_is_directory=True) + config_path = tmp_path / "profile.yaml" + original_config = "benchmark:\n inferencex_path: pinned\n" + config_path.write_text(original_config, encoding="utf-8") + + resolved, error = ProfileExecutor()._agentx_runtime_checkout( + config_path=config_path, + output_dir=output_dir, + inferencex_path=str(source), + agentx_session=True, + ) + + assert resolved == str(source) + assert error is not None + assert error["error_class"] == "agentx_profile_checkout_unavailable" + assert "symbolic link" in error["error"] + assert config_path.read_text(encoding="utf-8") == original_config + assert (source / "benchmarks" / "benchmark_lib.sh").read_text(encoding="utf-8") == "#!/bin/sh\n" + + +def test_agentx_profile_checkout_rejects_nested_foreign_git_worktree(tmp_path): + source = tmp_path / "pinned-inferencex" + _init_profile_inferencex_checkout(source) + output_dir = tmp_path / "profile-output" + subprocess.run( + ["git", "clone", "-q", "--local", str(source), str(output_dir)], + check=True, + ) + isolated = output_dir / ".agentx-profile-inferencex" + isolated.mkdir() + config_path = tmp_path / "profile.yaml" + original_config = "benchmark:\n inferencex_path: pinned\n" + config_path.write_text(original_config, encoding="utf-8") + + resolved, error = ProfileExecutor()._agentx_runtime_checkout( + config_path=config_path, + output_dir=output_dir, + inferencex_path=str(source), + agentx_session=True, + ) + + assert resolved == str(source) + assert error is not None + assert error["error_class"] == "agentx_profile_checkout_unavailable" + assert "not its own git worktree" in error["error"] + assert config_path.read_text(encoding="utf-8") == original_config + + +@pytest.mark.parametrize( + "failure", + [ + pytest.param(OSError("git unavailable"), id="os-error"), + pytest.param( + subprocess.TimeoutExpired(["git", "rev-parse"], 30), + id="timeout", + ), + ], +) +def test_agentx_profile_checkout_surfaces_git_process_failure( + tmp_path, + monkeypatch, + failure, +): + source = tmp_path / "pinned-inferencex" + _init_profile_inferencex_checkout(source) + output_dir = tmp_path / "profile-output" + output_dir.mkdir() + config_path = tmp_path / "profile.yaml" + original_config = "benchmark:\n inferencex_path: pinned\n" + config_path.write_text(original_config, encoding="utf-8") + + def fail_git(*_args, **_kwargs): + raise failure + + monkeypatch.setattr( + "hyperloom.orchestrator.actions.executors.profile.subprocess.run", + fail_git, + ) + resolved, error = ProfileExecutor()._agentx_runtime_checkout( + config_path=config_path, + output_dir=output_dir, + inferencex_path=str(source), + agentx_session=True, + ) + + assert resolved == str(source) + assert error is not None + assert error["error_class"] == "agentx_profile_checkout_unavailable" + assert "cannot prepare isolated AgentX profile checkout" in error["error"] + assert config_path.read_text(encoding="utf-8") == original_config + + +def test_agentx_profile_checkout_rejects_dirty_reused_clone(tmp_path): + source = tmp_path / "pinned-inferencex" + _init_profile_inferencex_checkout(source) + output_dir = tmp_path / "profile-output" + output_dir.mkdir() + isolated = output_dir / ".agentx-profile-inferencex" + subprocess.run( + ["git", "clone", "-q", "--local", "--no-hardlinks", str(source), str(isolated)], + check=True, + ) + isolated_benchmark_lib = isolated / "benchmarks" / "benchmark_lib.sh" + isolated_benchmark_lib.write_text("#!/bin/sh\n# stale profiler patch\n", encoding="utf-8") + config_path = tmp_path / "profile.yaml" + original_config = "benchmark:\n inferencex_path: pinned\n" + config_path.write_text(original_config, encoding="utf-8") + + resolved, error = ProfileExecutor()._agentx_runtime_checkout( + config_path=config_path, + output_dir=output_dir, + inferencex_path=str(source), + agentx_session=True, + ) + + assert resolved == str(source) + assert error is not None + assert error["error_class"] == "agentx_profile_checkout_unavailable" + assert "not clean" in error["error"] + assert "stale profiler patches" in error["error"] + assert config_path.read_text(encoding="utf-8") == original_config + assert isolated_benchmark_lib.read_text(encoding="utf-8").endswith("stale profiler patch\n") + + +@pytest.mark.asyncio +@pytest.mark.parametrize("native", [False, True], ids=["legacy", "native"]) +async def test_agentx_profile_executor_passes_rank_zero_not_merged(tmp_path, monkeypatch, native): monkeypatch.setenv("HYPERLOOM_AGENTX", "1") monkeypatch.setenv("TP", "2") db = SqliteConnection(tmp_path / "x.db") @@ -2221,6 +2704,7 @@ async def _fake_baseline(_self, _ctx): } pe = ProfileExecutor(session_dir=tmp_path / "ignored_root") + pe.shared_state = _agentx_profile_state(tmp_path) if native else SimpleNamespace(benchmark_mode="agentx") task = await tr.create( kind="profile", params={"output_dir": str(output_dir), "config_path": str(_PROFILE_SGLANG_CONFIG)}, @@ -2279,6 +2763,7 @@ async def _fake_baseline(_self, _ctx): } pe = ProfileExecutor(session_dir=tmp_path / "ignored_root") + pe.shared_state = _agentx_profile_state(tmp_path) task = await tr.create( kind="profile", params={"output_dir": str(output_dir), "config_path": str(_PROFILE_SGLANG_CONFIG)}, @@ -2319,6 +2804,7 @@ async def _fake_baseline(_self, _ctx): } pe = ProfileExecutor(session_dir=tmp_path / "ignored_root") + pe.shared_state = _agentx_profile_state(tmp_path) task = await tr.create( kind="profile", params={"output_dir": str(output_dir), "config_path": str(_PROFILE_SGLANG_CONFIG)}, @@ -2361,6 +2847,7 @@ async def _fake_baseline(_self, _ctx): } pe = ProfileExecutor(session_dir=tmp_path / "ignored_root") + pe.shared_state = _agentx_profile_state(tmp_path) task = await tr.create( kind="profile", params={"output_dir": str(output_dir), "config_path": str(_PROFILE_SGLANG_CONFIG)}, @@ -2593,7 +3080,7 @@ def test_upstream_sglang_capture_directory_passes_health_check(tmp_path): # kernel_request_handlers — direct unit @pytest.mark.asyncio -async def test_trace_analyze_handler_dry_run_returns_structured_result(session_dir): +async def test_trace_analyze_handler_dry_run_returns_structured_result(session_dir, monkeypatch): """The handler surfaces the tool's structured JSON verbatim (status + run_id + session_id).""" fake_trace = session_dir / "fake_trace_dir" fake_trace.mkdir() @@ -2609,6 +3096,10 @@ async def test_trace_analyze_handler_dry_run_returns_structured_result(session_d # agent route, which needs a real root). "analysis_route": "bypass", } + # Production uses an installed /opt/venv package. The local Darwin test + # invokes the standalone script with system ``python3``, so expose src in + # the same way an installation does. + monkeypatch.setenv("PYTHONPATH", str(Path(__file__).resolve().parents[3])) res = await ta.trace_analyze_handler(payload, session_dir=session_dir) # Structured result surfaced verbatim by the bypass backend. assert res["status"] in ("ok", "succeeded", "failed") diff --git a/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py b/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py index dc4013962d..efd31cf41f 100644 --- a/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py +++ b/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py @@ -2725,6 +2725,110 @@ def test_source_layer_handles_omit_a_completeness_the_result_never_recorded(): assert "source_snapshot_complete" not in handles +@pytest.mark.parametrize( + "result_overlay", + [ + {"final_overlay": "/accepted/overlay"}, + {"overlay_pythonpath": "/accepted/overlay"}, + {"effective_config": {"final_overlay": "/accepted/overlay"}}, + ], +) +def test_source_patch_keep_preserves_overlay_for_resume(session_dir, tmp_path, result_overlay): + coord = _coord(session_dir) + coord.shared_state.baseline_tput = 1000.0 + result = {**_keep_result(tmp_path, import_root="python"), **result_overlay} + assert coord.writeback.lift_to_current_best( + "integrate_patch", + 1200.0, + {"name": "patch-with-overlay", "scope": "source_patch", **wb._source_layer_handles(result)}, + ) + assert coord.shared_state.current_best["final_overlay"] == "/accepted/overlay" + assert coord.shared_state.optimization_stack[-1]["final_overlay"] == "/accepted/overlay" + + +def test_best_measurement_retains_native_execution_evidence(session_dir): + coord = _coord(session_dir) + coord.shared_state.current_best = {"tput": 1200.0} + evidence = { + "materialized_config": "/accepted/native.yaml", + "agentx_server_launch": {"effective_argv": ["python", "-m", "sglang.launch_server"]}, + "agentx_launch_contract": 1, + "agentx_workload_fingerprint": "a" * 64, + "agentx_candidate_fingerprint": "b" * 64, + "native_agentx_protocol_valid": True, + "valid_measurement": True, + } + coord.writeback._stamp_current_best_measurement(evidence) + assert evidence.items() <= coord.shared_state.current_best_measurement.items() + assert coord.shared_state.current_best["measurement"] == coord.shared_state.current_best_measurement + + +def test_native_keep_preserves_exact_snapshot_and_json_argv(session_dir, tmp_path): + import json + import shlex + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + from hyperloom.orchestrator.state.shared_state import stack_base_params + + coord = _coord(session_dir) + coord.shared_state.baseline_tput = 1000.0 + coord.shared_state.current_best = {"tput": 1050.0, "extra_server_args": "--obsolete", "extra_envs": {"OLD": "1"}} + snapshot = { + "version": 1, + "append_args": ["--json-config", '{"nested": "value with spaces"}', "--enable-cache"], + "env": {"SGLANG_USE_AITER": "1"}, + "remove_args": ["--old-default"], + "unset_env": ["OLD"], + "replace_args": False, + "source_files": {"/source/kernel.py": "f" * 64}, + } + launch = { + "version": 1, + "framework": "sglang", + "base_argv": ["python", "-m", "sglang.launch_server", "--old-default", "old-value"], + "effective_argv": ["python", "-m", "sglang.launch_server", *snapshot["append_args"]], + "base_env": {"SGLANG_USE_AITER": None, "OLD": "1"}, + "effective_env": {**snapshot["env"], "OLD": None}, + "resolved_executable": "/usr/bin/python", + "runtime_environment": dict(snapshot["env"]), + "source_files": snapshot["source_files"], + "overrides_sha256": canonical_sha256(snapshot), + } + launch["evidence_sha256"] = canonical_sha256(launch) + materialized = tmp_path / "measured.yaml" + materialized.write_text( + json.dumps({"benchmark": {"framework": "sglang", "agentx": {"launch_overrides": snapshot}}}) + ) + evidence = { + "name": "native-candidate", + "agentx_launch_contract": 1, + "agentx_server_launch": launch, + "materialized_config": str(materialized), + } + assert coord.writeback.lift_to_current_best("explore", 1200.0, evidence) + current = coord.shared_state.current_best + assert shlex.split(current["extra_server_args"]) == snapshot["append_args"] + assert current["extra_envs"] == snapshot["env"] + assert current["args_mode"] == "append" + assert current["native_launch_overrides"] == snapshot + assert coord.shared_state.optimization_stack[-1]["native_launch_overrides"] == snapshot + base = stack_base_params(coord.writeback.current_best_launch_config()) + assert base["base_native_launch_overrides"] == snapshot + base["base_native_launch_overrides"]["append_args"].append("--later") + assert current["native_launch_overrides"] == snapshot + + +def test_native_keep_refuses_config_that_changed_after_measurement(tmp_path): + import json + from hyperloom.orchestrator.measurement.native_launch import native_launch_config + + path = tmp_path / "native.yaml" + path.write_text(json.dumps({"benchmark": {"framework": "sglang", "agentx": {"launch_overrides": {"version": 1}}}})) + with pytest.raises(ValueError, match="does not match its measured launch"): + native_launch_config( + {"agentx_launch_contract": 1, "materialized_config": str(path), "agentx_server_launch": {}} + ) + + def test_env_spec_hands_geak_the_import_root_not_the_snapshot_top(session_dir, tmp_path): """GEAK PYTHONPATHs this value; the snapshot top holds no importable module.""" coord = _coord(session_dir) diff --git a/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py b/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py index b219c7aeb2..787f4e7b5a 100644 --- a/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py @@ -56,6 +56,42 @@ def test_should_use_ray_backend_unset_multi_node_false(monkeypatch: pytest.Monke assert rb._should_use_ray_backend() is False +@pytest.mark.parametrize("legacy_switch", ["0", "1"]) +def test_native_agentx_session_forces_direct_local_execution(monkeypatch, tmp_path, legacy_switch): + source = tmp_path / "native.yaml" + source.write_text(yaml.safe_dump({"benchmark": {"agentx": "enable"}}), encoding="utf-8") + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + monkeypatch.setenv("HYPERLOOM_AGENTX", legacy_switch) + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", "1") + assert rb._should_use_ray_backend() is False + assert rb.ray_gpu_specialist_exec_enabled() is False + + +def test_fresh_agentx_switch_forces_direct_local_execution(monkeypatch): + monkeypatch.delenv("HYPERLOOM_BENCHMARK_CONFIG", raising=False) + monkeypatch.delenv("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", raising=False) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", "1") + assert rb._should_use_ray_backend() is False + assert rb.ray_gpu_specialist_exec_enabled() is False + + +@pytest.mark.parametrize("epoch, ray_enabled", [(1, True), (2, False), (3, False)]) +def test_saved_agentx_epoch_owns_ray_routing(monkeypatch, tmp_path, epoch, ray_enabled): + from hyperloom.orchestrator.state.shared_state import SharedState + + SharedState(session_id="saved-agentx", benchmark_mode="agentx", agentx_epoch=epoch).save(tmp_path) + source = tmp_path / "native.yaml" + source.write_text("benchmark:\n agentx: enable\n", encoding="utf-8") + monkeypatch.setenv("INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR", str(tmp_path)) + monkeypatch.setenv("HYPERLOOM_BENCHMARK_CONFIG", str(source)) + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("INFERENCE_OPTIMIZER_RAY_EXEC", "1") + monkeypatch.setenv("INFERENCE_OPTIMIZER_NODES", "1") + assert rb._should_use_ray_backend() is ray_enabled + assert rb.ray_gpu_specialist_exec_enabled() is ray_enabled + + # ── visible-device merge invariant ─────────────────────────────────────────── def test_merge_worker_env_preserves_ray_visible_devices(monkeypatch: pytest.MonkeyPatch): """Ray owns *_VISIBLE_DEVICES; the caller must never override them.""" @@ -286,6 +322,7 @@ def _pid_alive(pid: int) -> bool: return True +@pytest.mark.skipif(sys.platform != "linux", reason="process-tree supervision requires Linux procfs") def test_managed_process_start_and_reap(): """A supervised process is reaped on stop() — no detached GPU-proc escape.""" mgr = ManagedServerProcess() @@ -304,6 +341,7 @@ def test_managed_process_start_and_reap(): assert not _pid_alive(pid), "supervised process must not survive stop()" +@pytest.mark.skipif(sys.platform != "linux", reason="PR_SET_PDEATHSIG is Linux-only") def test_pdeathsig_arms_a_trappable_signal(tmp_path: Path): """The parent-death signal must be SIGTERM, never SIGKILL. @@ -330,6 +368,7 @@ def test_pdeathsig_arms_a_trappable_signal(tmp_path: Path): assert log_path.read_text(encoding="utf-8").strip() == str(int(signal.SIGTERM)) +@pytest.mark.skipif(sys.platform != "linux", reason="PR_SET_PDEATHSIG is Linux-only") def test_owner_death_still_reaps_the_wrappers_setsid_server(tmp_path: Path): """An abrupt owner death leaves no server behind. @@ -381,6 +420,7 @@ def test_owner_death_still_reaps_the_wrappers_setsid_server(tmp_path: Path): os.killpg(server_pgid, signal.SIGKILL) +@pytest.mark.skipif(sys.platform != "linux", reason="process-tree supervision requires Linux procfs") def test_managed_process_stop_idempotent(): mgr = ManagedServerProcess() mgr.start(["sleep", "5"]) @@ -572,6 +612,80 @@ def test_num_gpus_for_config_reads_tp(tmp_path: Path): assert gr._num_gpus_for_config(cfg) == 4.0 +def test_num_gpus_for_config_counts_tp_pp_and_pcp(tmp_path: Path): + from hyperloom.orchestrator.actions.executors import _grid_runner as gr + + cfg = tmp_path / "c.yaml" + cfg.write_text( + "benchmark:\n envs:\n TP: 2\n PP_SIZE: 2\n PCP_SIZE: 2\n", + encoding="utf-8", + ) + assert gr._num_gpus_for_config(cfg) == 8.0 + + +def test_num_gpus_for_config_prefers_resolved_agentx_topology(tmp_path: Path): + from hyperloom.orchestrator.actions.executors import _grid_runner as gr + + cfg = tmp_path / "c.yaml" + cfg.write_text( + "benchmark:\n" + " agentx: enable\n" + " envs:\n" + " TP: 1\n" + " workload_spec:\n" + " resolved_topology:\n" + " tp: 2\n" + " pp: 2\n" + " pcp_size: 2\n" + " gpu_count: 8\n", + encoding="utf-8", + ) + assert gr._num_gpus_for_config(cfg) == 8.0 + + +def test_num_gpus_for_config_rejects_inconsistent_agentx_topology(tmp_path: Path): + from hyperloom.orchestrator.actions.executors import _grid_runner as gr + + cfg = tmp_path / "c.yaml" + cfg.write_text( + "benchmark:\n" + " agentx: enable\n" + " envs:\n" + " TP: 16\n" + " workload_spec:\n" + " resolved_topology:\n" + " tp: 2\n" + " pp: 2\n" + " pcp_size: 2\n" + " gpu_count: 4\n", + encoding="utf-8", + ) + + with pytest.raises(ValueError, match="invalid native AgentX resolved topology"): + gr._num_gpus_for_config(cfg) + + +def test_num_gpus_for_config_ignores_stale_topology_outside_agentx(tmp_path: Path): + from hyperloom.orchestrator.actions.executors import _grid_runner as gr + + cfg = tmp_path / "c.yaml" + cfg.write_text( + "benchmark:\n" + " envs:\n" + " TP: 3\n" + " PP_SIZE: 2\n" + " workload_spec:\n" + " resolved_topology:\n" + " tp: 2\n" + " pp: 2\n" + " pcp_size: 2\n" + " gpu_count: 8\n", + encoding="utf-8", + ) + + assert gr._num_gpus_for_config(cfg) == 6.0 + + def test_num_gpus_for_config_defaults_to_one(tmp_path: Path): from hyperloom.orchestrator.actions.executors import _grid_runner as gr @@ -1826,6 +1940,7 @@ def test_serving_lease_close_swallows_kill_error(monkeypatch: pytest.MonkeyPatch assert lease._actor is None +@pytest.mark.skipif(sys.platform != "linux", reason="process-tree supervision requires Linux procfs") def test_releasing_a_lease_reaps_the_served_process(serving_lease_on_a_ray_double): """``ray.kill`` skips ``__ray_terminate__``, so the actor must be asked first.""" lease = serving_lease_on_a_ray_double @@ -1842,6 +1957,7 @@ def test_releasing_a_lease_reaps_the_served_process(serving_lease_on_a_ray_doubl assert lease._actor is None +@pytest.mark.skipif(sys.platform != "linux", reason="process-tree supervision requires Linux procfs") def test_releasing_a_specialist_lease_reaps_its_subprocess(monkeypatch: pytest.MonkeyPatch): """The same invariant on the specialist lease, which used to skip the stop. diff --git a/src/hyperloom/inference_optimizer/tests/test_run_optimize_exit_code_lock.py b/src/hyperloom/inference_optimizer/tests/test_run_optimize_exit_code_lock.py index 36c826b8c5..417cdfe911 100644 --- a/src/hyperloom/inference_optimizer/tests/test_run_optimize_exit_code_lock.py +++ b/src/hyperloom/inference_optimizer/tests/test_run_optimize_exit_code_lock.py @@ -5,7 +5,6 @@ from __future__ import annotations -import argparse import asyncio from pathlib import Path @@ -90,22 +89,28 @@ def test_optimize_has_no_supervisor_launcher_hooks() -> None: assert "orchestrator.supervisor" not in source -def test_multinode_tp_exceeds_total_gpus_exits_2() -> None: +def test_multinode_tp_exceeds_total_gpus_exits_2(capsys) -> None: """Gate 1: TP larger than nodes*gpus_per_node fails fast with exit code 2.""" # nodes=2, gpus_per_node=1 -> total_gpus=2 < tp=4. - args = argparse.Namespace(nodes=2, tp=4, ep=1, gpus_per_node=1) + args = ocli._build_parser().parse_args( + ["optimize", "--nodes", "2", "--tp", "4", "--ep", "1", "--gpus-per-node", "1"] + ) with pytest.raises(SystemExit) as exc: asyncio.run(ocli._run_optimize(args)) assert exc.value.code == 2 + assert "TP=4 exceeds total GPU count" in capsys.readouterr().err -def test_multinode_ep_exceeds_tp_exits_2() -> None: +def test_multinode_ep_exceeds_tp_exits_2(capsys) -> None: """Gate 2: EP greater than TP fails fast with exit code 2.""" # total_gpus=16 >= tp=2 so gate 1 passes; ep=4 > tp=2 trips gate 2. - args = argparse.Namespace(nodes=2, tp=2, ep=4, gpus_per_node=8) + args = ocli._build_parser().parse_args( + ["optimize", "--nodes", "2", "--tp", "2", "--ep", "4", "--gpus-per-node", "8"] + ) with pytest.raises(SystemExit) as exc: asyncio.run(ocli._run_optimize(args)) assert exc.value.code == 2 + assert "EP=4 > TP=2" in capsys.readouterr().err def test_session_busy_exits_with_session_busy_code(tmp_path: Path) -> None: diff --git a/src/hyperloom/inference_optimizer/tests/test_sbd_v6_initial.py b/src/hyperloom/inference_optimizer/tests/test_sbd_v6_initial.py index e8b4008a54..a559975bfd 100644 --- a/src/hyperloom/inference_optimizer/tests/test_sbd_v6_initial.py +++ b/src/hyperloom/inference_optimizer/tests/test_sbd_v6_initial.py @@ -710,6 +710,7 @@ def test_preflight_records_install_steps_in_execution_order(tmp_path, monkeypatc from hyperloom.inference_optimizer.session.sbd_v6 import pending_install_event from hyperloom.orchestrator.actions.executors import benchmark_backend + monkeypatch.setattr(os, "environ", os.environ.copy()) inferencex = tmp_path / "InferenceX" inferencex.mkdir() monkeypatch.setenv("INFERENCEX_PATH", str(inferencex)) diff --git a/src/hyperloom/inference_optimizer/tests/test_server_argv_seal.py b/src/hyperloom/inference_optimizer/tests/test_server_argv_seal.py index 4ae3aa20a2..dd7fe57b05 100644 --- a/src/hyperloom/inference_optimizer/tests/test_server_argv_seal.py +++ b/src/hyperloom/inference_optimizer/tests/test_server_argv_seal.py @@ -66,17 +66,26 @@ def _function(module, name: str) -> ast.FunctionDef: raise AssertionError(f"{name} not found in {module.__file__}") -def _seal_call(body: list[ast.stmt]) -> tuple[int, ast.Call]: - """Return the index and node of the sole seal call among a body's statements.""" +def _seal_call(body: list[ast.stmt]) -> ast.Call: + """Find the final seal, nested when its alternative uses native argv.""" + statements = body + if not any( + isinstance(stmt, ast.Expr) + and isinstance(stmt.value, ast.Call) + and isinstance(stmt.value.func, ast.Name) + and stmt.value.func.id == SEAL + for stmt in body + ): + statements = list(ast.walk(ast.Module(body=body, type_ignores=[]))) found = [ - (index, stmt.value) - for index, stmt in enumerate(body) + stmt.value + for stmt in statements if isinstance(stmt, ast.Expr) and isinstance(stmt.value, ast.Call) and isinstance(stmt.value.func, ast.Name) and stmt.value.func.id == SEAL ] - assert len(found) == 1, f"expected exactly one top-level {SEAL} call, found {len(found)}" + assert len(found) == 1, f"expected exactly one {SEAL} call, found {len(found)}" return found[0] @@ -125,14 +134,13 @@ def _is_argument_env(key: ast.expr) -> bool: def test_nothing_the_seal_was_given_is_touched_again_after_it(module, name): """The seal is the last statement naming the mapping it was handed.""" body = _function(module, name).body - index, call = _seal_call(body) + call = _seal_call(body) sealed = {node.id for argument in call.args for node in ast.walk(argument) if isinstance(node, ast.Name)} assert sealed, "the seal call names nothing this test could follow" late = [ node.lineno - for stmt in body[index + 1 :] - for node in ast.walk(stmt) - if isinstance(node, ast.Name) and node.id in sealed + for node in ast.walk(ast.Module(body=body, type_ignores=[])) + if isinstance(node, ast.Name) and node.id in sealed and node.lineno > call.end_lineno ] assert not late, f"{name} touches {sorted(sealed)} after the seal, at line(s) {late}" @@ -153,7 +161,7 @@ def test_every_write_to_the_argument_env_is_one_the_seal_settles(): def test_every_write_inside_a_composer_happens_above_the_seal(module, name): """Composition is what runs before the seal; nothing writes the env below it.""" source = Path(module.__file__).read_text(encoding="utf-8") - seal_line = _seal_call(_function(module, name).body)[1].lineno + seal_line = _seal_call(_function(module, name).body).lineno late = [lineno for function, lineno in _argument_env_writes(source) if function == name and lineno > seal_line] assert not late, f"{name} writes the argument env after the seal at line {seal_line}, at line(s) {late}" diff --git a/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py b/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py index 2a31230bc3..aecca02115 100644 --- a/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py @@ -50,6 +50,14 @@ def test_zero_when_nothing_established(self): assert resolve_grading_anchor_tput(SharedState()) == 0.0 +@pytest.mark.parametrize("epoch, expected", [(1, True), (2, False)]) +def test_agentx_resume_preserves_legacy_warm_replay_but_disables_native(tmp_path, epoch, expected): + state = SharedState(benchmark_mode="agentx", agentx_epoch=epoch, warm_replay_enabled=True) + state.save(tmp_path) + + assert SharedState.load_or_init(tmp_path).warm_replay_enabled is expected + + class TestInjectStackBaseParams: @staticmethod def _state(): diff --git a/src/hyperloom/inference_optimizer/tests/test_warm_replay.py b/src/hyperloom/inference_optimizer/tests/test_warm_replay.py index 8ff3c38c7b..74e2f86878 100644 --- a/src/hyperloom/inference_optimizer/tests/test_warm_replay.py +++ b/src/hyperloom/inference_optimizer/tests/test_warm_replay.py @@ -2176,8 +2176,15 @@ def _git_repo_for_required_patch(tmp_path: Path) -> tuple[Path, str]: return repo, patch +@pytest.mark.parametrize( + ("benchmark_mode", "agentx_epoch"), + [("synthetic", 0), ("agentx", 1), ("agentx", 2)], + ids=["synthetic", "legacy-agentx", "native-agentx"], +) def test_combined_keep_retains_validated_framework_root_without_reapply( tmp_path, + benchmark_mode, + agentx_epoch, ): checkout, patch_content = _git_repo_for_required_patch(tmp_path) subprocess.run( @@ -2189,6 +2196,27 @@ def test_combined_keep_retains_validated_framework_root_without_reapply( ) coord = _make_coord(tmp_path, warm_start_recipe=_warm_recipe_t1()) coord.shared_state.baseline_tput = 600.0 + coord.shared_state.benchmark_mode = benchmark_mode + coord.shared_state.agentx_epoch = agentx_epoch + candidate_perf = {"output_throughput": 612.0} + if benchmark_mode == "agentx": + coord.shared_state.baseline_perf = { + "output_throughput": 600.0, + "total_throughput": 600.0, + "e2e_norm_intvty_p90": 20.0, + "e2e_norm_intvty_p50": 50.0, + "duration_seconds": 900.0, + "request_error_rate": 0.0, + } + candidate_perf = { + **coord.shared_state.baseline_perf, + "output_throughput": 612.0, + "total_throughput": 612.0, + "e2e_norm_intvty_p90": 23.0, + "e2e_norm_intvty_p50": 57.5, + } + original_inferencex_path = str(tmp_path / "pinned-inferencex") + coord.shared_state.active_inferencex_path = original_inferencex_path coord.shared_state.warm_replay_outcome = {"expected_gain_pct": 0.0} task = _StubTask( params={ @@ -2212,7 +2240,7 @@ def test_combined_keep_retains_validated_framework_root_without_reapply( coord.phase_prelude.promote_warm_replay( { "status": "succeeded", - "output_throughput": 612.0, + **candidate_perf, "warm_patch_target": str(checkout), "warm_patch_pre_sha": "base-sha", "warm_patch_snapshot_manifest": { @@ -2232,6 +2260,9 @@ def test_combined_keep_retains_validated_framework_root_without_reapply( assert "persisted = True" in (checkout / "vllm" / "fp8.py").read_text() assert coord.shared_state.warm_replay_outcome["status"] == "reproduced" assert coord.shared_state.warm_replay_outcome["active_framework_root"] == str(checkout.resolve()) + assert coord.shared_state.active_inferencex_path == ( + original_inferencex_path if agentx_epoch == 2 else str(checkout.resolve()) + ) assert coord.shared_state.optimization_stack[-1]["framework_source_root"] == str(checkout.resolve()) entry = coord.shared_state.optimization_stack[-1] assert entry["recipe_delta"] == { diff --git a/src/hyperloom/inference_optimizer/tests/test_workload_envs.py b/src/hyperloom/inference_optimizer/tests/test_workload_envs.py index ada5452881..b630b46053 100644 --- a/src/hyperloom/inference_optimizer/tests/test_workload_envs.py +++ b/src/hyperloom/inference_optimizer/tests/test_workload_envs.py @@ -406,15 +406,28 @@ def test_profile_max_iters_override(monkeypatch, tmp_path): assert "NUM_PROMPTS" in bench["envs"] -def test_the_iters_override_keeps_the_delay_an_agentx_run_needs_at_zero(monkeypatch, tmp_path): - """Raising the capture bound must not reintroduce an iteration delay.""" +@pytest.mark.parametrize("native", [False, True]) +def test_the_iters_override_keeps_the_delay_an_agentx_run_needs_at_zero(monkeypatch, tmp_path, native): + """Both AgentX clients capture without an iteration delay; native capture is diagnostic.""" _clear_env(monkeypatch) monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "amd/GLM-5.2-MXFP4") monkeypatch.setenv("HYPERLOOM_PROFILE_MAX_ITERS", "128") monkeypatch.setenv("HYPERLOOM_PROFILE_DELAY_ITERS", "64") - src = _write(tmp_path / "cfg.yaml", framework="vllm", envs={"PROFILE": "1"}) - bench = _materialize(src, tmp_path / "out") + src = _write( + tmp_path / "cfg.yaml", + framework="vllm", + envs={"PROFILE": "1"}, + **({"agentx": "enable"} if native else {}), + ) + if native: + with pytest.raises(ValueError, match="diagnostic-only"): + _materialize(src, tmp_path / "measurement") + + bench = _materialize(src, tmp_path / "out", allow_agentx_profile_compat=native) + assert bench["workload_spec"].get("harness") == ("hyperloom-profiler-compat" if native else None) + assert "agentx" not in bench args = str(bench["envs"]["EXTRA_VLLM_ARGS"]) assert "--profiler-config.delay_iterations 0" in args assert "--profiler-config.max_iterations 128" in args diff --git a/src/hyperloom/inference_optimizer/tests/test_workload_envs_branches_unit.py b/src/hyperloom/inference_optimizer/tests/test_workload_envs_branches_unit.py index ff6dee9fd7..9417366be0 100644 --- a/src/hyperloom/inference_optimizer/tests/test_workload_envs_branches_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_workload_envs_branches_unit.py @@ -33,6 +33,11 @@ def _clear_env(monkeypatch): "PROFILE", "MODEL_PATH", "INFERENCEX_PATH", + "HYPERLOOM_AGENTX", + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_MODE", + "AGENTX_DATASET", "HYPERLOOM_PROFILE_MAX_ITERS", "HYPERLOOM_PROFILE_DELAY_ITERS", "HYPERLOOM_PROFILE_MAX_STEPS_CAP", @@ -45,6 +50,32 @@ def _clear_env(monkeypatch): monkeypatch.delenv(k, raising=False) +def _enable_native_agentx(monkeypatch, tmp_path: Path) -> None: + from hyperloom.inference_optimizer.agentx import native as native_agentx + + monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + monkeypatch.setenv("AGENTX_MODEL_ID", "acme/Test-Model") + monkeypatch.setenv( + "AGENTX_SERVER_SCRIPT", + "single_node/agentic/test_fp4_mi355x_sglang_mtp.sh", + ) + monkeypatch.setenv("INFERENCEX_PATH", str(tmp_path)) + + def _resolve(benchmark, *, expected_gpu_count, **_kwargs): + topology = { + "tp": expected_gpu_count, + "pp": 1, + "pcp_size": 1, + "ep": expected_gpu_count, + "gpu_count": expected_gpu_count, + "recipe_fingerprint": "a" * 64, + } + benchmark.setdefault("workload_spec", {})["resolved_topology"] = topology + return topology + + monkeypatch.setattr(native_agentx, "resolve_native_recipe", _resolve) + + def _write(path, **bench_extra): bench = {"framework": "sglang", "model": "/m", "envs": {}} bench.update(bench_extra) @@ -521,6 +552,39 @@ def test_profile_sglang_bad_extra_body(monkeypatch, tmp_path): assert "start_step" in body and "num_steps" in body +@pytest.mark.parametrize("framework", ["sglang", "vllm"]) +def test_agentx_diagnostic_profile_preserves_installed_framework(monkeypatch, tmp_path, framework): + from hyperloom.orchestrator.actions.executors import _server_patcher + + _clear_env(monkeypatch) + _enable_native_agentx(monkeypatch, tmp_path) + _stub_server_arg_injectors(monkeypatch) + monkeypatch.setenv("HYPERLOOM_ENABLE_PATCH", "1") + monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") + monkeypatch.setattr(_server_patcher, "resolve_sglang_shape_mode", lambda: "patched") + + def reject_source_patch(*args, **kwargs): + pytest.fail("diagnostic profiling must preserve the native framework install") + + monkeypatch.setattr(we, "ensure_sglang_patched_for_tracelens", reject_source_patch) + monkeypatch.setattr(we, "ensure_vllm_patched_for_tracelens", reject_source_patch) + src = _write( + tmp_path / "profile.yaml", + framework=framework, + agentx="enable", + envs={"PROFILE": "1"}, + profiler={"torch_profiler": {"enabled": True}}, + ) + + bench = _materialize(src, tmp_path / "out", allow_agentx_profile_compat=True) + + assert bench["workload_spec"]["harness"] == "hyperloom-profiler-compat" + assert bench["envs"]["HYPERLOOM_TRACELENS_PATCH_STATUS"] == "not_attempted" + assert bench["envs"]["HYPERLOOM_PROFILE_DEGRADED_REASON"] == "tracelens_runtime_patch_unavailable" + if framework == "sglang": + assert json.loads(bench["envs"]["PROFILE_EXTRA_BODY"])["detailed_annotations"] is False + + def _profile_num_steps(bench) -> int: """Captured-step count the sglang profile path writes into PROFILE_EXTRA_BODY.""" import json @@ -717,7 +781,7 @@ def test_agentx_workload_spec_concurrency_tracks_served_conc(monkeypatch, tmp_pa # process env instead, so the two agree by construction. _clear_env(monkeypatch) monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native_agentx(monkeypatch, tmp_path) monkeypatch.setenv("CONC", "8") src = _write(tmp_path / "cfg.yaml", envs={"CONC": 64}) bench = _materialize(src, tmp_path / "out") @@ -725,6 +789,56 @@ def test_agentx_workload_spec_concurrency_tracks_served_conc(monkeypatch, tmp_pa assert bench["workload_spec"]["concurrency"] == 8 +def test_native_agentx_rejects_final_extra_env_concurrency_change(monkeypatch, tmp_path): + _clear_env(monkeypatch) + monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") + _enable_native_agentx(monkeypatch, tmp_path) + src = _write( + tmp_path / "cfg.yaml", + envs={"CONC": 8}, + agentx={"enabled": True, "mode": "canonical", "concurrency": 8}, + ) + + with pytest.raises( + ValueError, + match=r"Native AgentX concurrency is fixed for the session: resolved CONC=8, candidate requested 16", + ): + _materialize(src, tmp_path / "out", extra_envs={"CONC": "16"}) + + +def test_native_agentx_filters_credentials_before_recipe_fingerprint(monkeypatch, tmp_path): + """The resolver and persisted YAML must see the same credential-free envs.""" + from hyperloom.inference_optimizer.agentx import native as native_agentx + + _clear_env(monkeypatch) + monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") + _enable_native_agentx(monkeypatch, tmp_path) + resolved_envs: dict[str, object] = {} + resolve = native_agentx.resolve_native_recipe + + def _capture_resolved_envs(benchmark, **kwargs): + resolved_envs.update(benchmark.get("envs") or {}) + return resolve(benchmark, **kwargs) + + monkeypatch.setattr(native_agentx, "resolve_native_recipe", _capture_resolved_envs) + src = _write( + tmp_path / "cfg.yaml", + agentx="enable", + envs={ + "CONC": 8, + "OPENAI_API_KEY": "must-not-be-fingerprinted", + "SAFE_WORKLOAD_KNOB": "preserved", + }, + ) + + bench = _materialize(src, tmp_path / "out") + + assert "OPENAI_API_KEY" not in resolved_envs + assert "OPENAI_API_KEY" not in bench["envs"] + assert resolved_envs["SAFE_WORKLOAD_KNOB"] == "preserved" + assert bench["envs"]["SAFE_WORKLOAD_KNOB"] == "preserved" + + def test_agentx_workload_spec_publishes_the_conc_scaled_warmup_grace(monkeypatch, tmp_path): """The spec must carry the grace the CLIENT is bounded by, not the raw knob. @@ -738,7 +852,7 @@ def test_agentx_workload_spec_publishes_the_conc_scaled_warmup_grace(monkeypatch _clear_env(monkeypatch) monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native_agentx(monkeypatch, tmp_path) monkeypatch.setenv("CONC", "32") # A grace declared as measured at CONC=8; warmup is linear in CONC, so the # bound this round runs at is 1800 * 32/8. @@ -762,7 +876,7 @@ def test_agentx_workload_spec_names_the_axis_the_session_is_graded_on(monkeypatc """ _clear_env(monkeypatch) monkeypatch.setenv("INFERENCE_OPTIMIZER_DISABLE_TP_CLAMP", "1") - monkeypatch.setenv("HYPERLOOM_AGENTX", "1") + _enable_native_agentx(monkeypatch, tmp_path) monkeypatch.delenv("HYPERLOOM_PERF_METRIC", raising=False) src = _write(tmp_path / "cfg.yaml", envs={}) bench = _materialize(src, tmp_path / "out") diff --git a/src/hyperloom/orchestrator/actions/executors/_accuracy_gate.py b/src/hyperloom/orchestrator/actions/executors/_accuracy_gate.py index 1aeaa08da8..8a84ac1887 100644 --- a/src/hyperloom/orchestrator/actions/executors/_accuracy_gate.py +++ b/src/hyperloom/orchestrator/actions/executors/_accuracy_gate.py @@ -16,6 +16,7 @@ import yaml +from hyperloom.common.agentx_accounting import measured_request_errors from hyperloom.common.env import env_flag, is_truthy from hyperloom.common.io import safe_mtime from hyperloom.common.perf_metric import is_agentx_mode @@ -337,6 +338,96 @@ def parse_quality_gate(workspace: Path | str) -> dict[str, Any]: return {"quality_gate": qg, "source_file": str(latest)} +def _parse_agentx_error_gate( + workspace: Path | str, +) -> tuple[float | None, float | None]: + """Read the newest AgentX error rate and its threshold, as percentages. + + Legacy ``map_aiperf`` artifacts expose a top-level percentage. Native + Magpie reports bind both values as ratios under + ``agentx_metrics.requests``. A native threshold is part of the accepted + measurement contract, so a missing, non-numeric, or out-of-range value is + returned as ``None`` and makes the quality gate fail closed. Legacy and + raw InferenceX artifacts do not carry that field and retain the historical + 10-percent default. + """ + # None rather than 0.0 so an export without the field is incomparable + # instead of a perfect score. + workspace = Path(workspace) + patterns = ("benchmark_report.json", "inferencex_result.json") + candidates = [ + Path(filename) + for pattern in patterns + for filename in glob.glob(str(workspace / "**" / pattern), recursive=True) + ] + for path in sorted(candidates, key=safe_mtime, reverse=True): + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError) as exc: + log.warning("accuracy_gate: unreadable %s: %s", path, exc) + continue + if not isinstance(data, dict): + continue + + legacy_rate = data.get("request_error_rate") + if isinstance(legacy_rate, (int, float)) and not isinstance(legacy_rate, bool): + return float(legacy_rate), AGENTX_ERROR_RATE_THRESHOLD_PCT + + metrics = data.get("agentx_metrics") + requests = metrics.get("requests") if isinstance(metrics, dict) else None + if isinstance(requests, dict): + native_rate = requests.get("error_rate") + if isinstance(native_rate, (int, float)) and not isinstance(native_rate, bool): + threshold = requests.get("threshold") + if ( + isinstance(threshold, (int, float)) + and not isinstance(threshold, bool) + and math.isfinite(float(threshold)) + and 0.0 <= float(threshold) <= 1.0 + ): + return 100.0 * float(native_rate), 100.0 * float(threshold) + log.warning( + "accuracy_gate: native AgentX report %s has an invalid request threshold: %r", + path, + threshold, + ) + return 100.0 * float(native_rate), None + + accounting = data.get("request_accounting") + if isinstance(accounting, dict): + errors = measured_request_errors(accounting) + successful = data.get("num_requests_successful") + if "records_dropped_total" in accounting: + dropped = accounting["records_dropped_total"] + total = data.get("num_requests_total") + if ( + errors is None + or not all(type(value) is int and value >= 0 for value in (successful, dropped, total)) + or successful + dropped != total + ): + return None, None + if ( + isinstance(errors, (int, float)) + and not isinstance(errors, bool) + and isinstance(successful, (int, float)) + and not isinstance(successful, bool) + and errors >= 0 + and successful >= 0 + and errors + successful > 0 + ): + return ( + 100.0 * float(errors) / float(errors + successful), + AGENTX_ERROR_RATE_THRESHOLD_PCT, + ) + return None, None + + +def parse_agentx_error_rate(workspace: Path | str) -> float | None: + """Read the newest legacy or native AgentX error rate, as a percentage.""" + rate, _threshold = _parse_agentx_error_gate(workspace) + return rate + + def _latest_agentx_result(workspace: Path | str) -> dict[str, Any] | None: """The newest ``inferencex_result.json`` under *workspace*, or None.""" results = [Path(f) for f in glob.glob(str(Path(workspace) / "**" / "inferencex_result.json"), recursive=True)] @@ -351,13 +442,6 @@ def _latest_agentx_result(workspace: Path | str) -> dict[str, Any] | None: return data if isinstance(data, dict) else None -def parse_agentx_error_rate(workspace: Path | str) -> float | None: - """Read ``request_error_rate`` from the newest ``inferencex_result.json``; None when no result reported one.""" - # None rather than 0.0 so an export without the field is incomparable instead of a perfect score. - rate = (_latest_agentx_result(workspace) or {}).get("request_error_rate") - return float(rate) if isinstance(rate, (int, float)) and not isinstance(rate, bool) else None - - def _mlperf_inline_accuracy(result: dict[str, Any] | None) -> float | None: """The harness's inline score when every turn was scored; None otherwise.""" result = result or {} @@ -423,9 +507,14 @@ def parse_eval_results( # that reported no rate is not comparable, and treating that as a pass is # how an incomparable measurement reaches the leaderboard set. if is_agentx_mode(benchmark_mode): - rate = parse_agentx_error_rate(workspace) - passed = rate is not None and rate <= AGENTX_ERROR_RATE_THRESHOLD_PCT - log.info("accuracy_gate: agentx request_error_rate=%s passed=%s", rate, passed) + rate, threshold = _parse_agentx_error_gate(workspace) + passed = rate is not None and threshold is not None and rate <= threshold + log.info( + "accuracy_gate: agentx request_error_rate=%s threshold=%s passed=%s", + rate, + threshold, + passed, + ) from hyperloom.common.agentx_workload import is_mlperf_backend if is_mlperf_backend(): @@ -445,6 +534,7 @@ def parse_eval_results( "task": "agentx_error_rate", "metric": "request_error_rate", "error_rate": rate, + "error_rate_threshold": threshold, } # Scriptable quality gate first: map passed->1.0 / fail->0.0. diff --git a/src/hyperloom/orchestrator/actions/executors/_benchmark_runtime.py b/src/hyperloom/orchestrator/actions/executors/_benchmark_runtime.py index 2a467c58ce..a869301c37 100644 --- a/src/hyperloom/orchestrator/actions/executors/_benchmark_runtime.py +++ b/src/hyperloom/orchestrator/actions/executors/_benchmark_runtime.py @@ -28,8 +28,13 @@ def apply_runtime_benchmark_overrides( grading: Mapping[str, Any] | None = None, ) -> dict[str, Any]: """Apply runtime env/CLI overrides to a Magpie benchmark YAML.""" - if agentx_mode is None and is_agentx_client_script(str(bench.get("benchmark_script") or "")): - agentx_mode = True + if agentx_mode is None: + from hyperloom.common.agentx_mode import native_agentx_enabled + + if is_agentx_client_script(str(bench.get("benchmark_script") or "")) or native_agentx_enabled( + bench.get("agentx") + ): + agentx_mode = True if model_path: bench["model"] = str(model_path) @@ -40,8 +45,9 @@ def apply_runtime_benchmark_overrides( if gpu_type: bench["runner_type"] = str(gpu_type) - # Force-pin the generic ``{framework}_{gpu_type}.sh`` so Magpie's resolver doesn't fall through to InferenceX - # native scripts that ignore ``EXTRA_*_ARGS``. + # The AgentX switch below restores its explicit native launcher. Other + # runs stay pinned to the generic script rather than depending on + # Magpie's resolver order. framework = str(bench.get("framework") or "").lower() if framework: bench["benchmark_script"] = f"{framework}_{gpu_type}.sh" @@ -55,6 +61,11 @@ def apply_runtime_benchmark_overrides( # {framework}_{gpu_type}.sh and silently reverts a materialize-time AgentX swap (grid/baseline/profile executors # rebuild via this function). apply_agentx_switch(bench, model_path, conc=conc, active=agentx_mode, grading=grading) + native_agentx = False + if "agentx" in bench: + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + native_agentx = native_agentx_enabled(bench.get("agentx")) envs: dict[str, Any] = bench.setdefault("envs", {}) # Same hazard as the AgentX swap above: the gpu_type block re-pins the bare {framework}_{gpu_type}.sh over the @@ -85,7 +96,7 @@ def apply_runtime_benchmark_overrides( tp_val = int(envs.get("TP", 1) or 1) existing_rocr = str(envs.get("ROCR_VISIBLE_DEVICES", "")).strip() existing_count = len([x for x in existing_rocr.split(",") if x.strip()]) if existing_rocr else 0 - if tp_val > 1 and existing_count < tp_val: + if not native_agentx and tp_val > 1 and existing_count < tp_val: envs["ROCR_VISIBLE_DEVICES"] = ",".join(str(i) for i in range(tp_val)) return envs diff --git a/src/hyperloom/orchestrator/actions/executors/_grid_base.py b/src/hyperloom/orchestrator/actions/executors/_grid_base.py index be35f436a6..e48d44b8fb 100644 --- a/src/hyperloom/orchestrator/actions/executors/_grid_base.py +++ b/src/hyperloom/orchestrator/actions/executors/_grid_base.py @@ -56,6 +56,7 @@ def __init__( extra_envs, allow_predicate=is_allowed_variant_env_key, ) + self.dropped_envs = list(dropped_envs) if dropped_envs: log.warning("Variant %s: dropping unsafe extra_envs %s", name, ", ".join(sorted(dropped_envs))) self.remove_args = to_str_list(remove_args) @@ -119,6 +120,7 @@ class VariantResult: output_throughput: float | None = None request_throughput: float | None = None total_token_throughput: float | None = None + agentx_gpu_count: int | None = None completed_requests: int | None = None duration_seconds: float | None = None ttft_mean_ms: float | None = None @@ -149,6 +151,8 @@ class VariantResult: server_log_path: str | None = None launch_evidence: dict[str, Any] = field(default_factory=dict) launch_evidence_path: str | None = None + materialized_config: str | None = None + native_measurement: dict[str, Any] = field(default_factory=dict) def to_dict(self) -> dict[str, Any]: """Serialize this result to a plain JSON-friendly dict.""" @@ -160,6 +164,7 @@ def to_dict(self) -> dict[str, Any]: "output_throughput": self.output_throughput, "request_throughput": self.request_throughput, "total_token_throughput": self.total_token_throughput, + "agentx_gpu_count": self.agentx_gpu_count, "completed_requests": self.completed_requests, "duration_seconds": self.duration_seconds, "ttft_mean_ms": self.ttft_mean_ms, @@ -185,4 +190,6 @@ def to_dict(self) -> dict[str, Any]: "server_log_path": self.server_log_path, "launch_evidence": self.launch_evidence, "launch_evidence_path": self.launch_evidence_path, + **({"materialized_config": self.materialized_config} if self.materialized_config else {}), + **self.native_measurement, } diff --git a/src/hyperloom/orchestrator/actions/executors/_grid_runner.py b/src/hyperloom/orchestrator/actions/executors/_grid_runner.py index 3d032c7079..48aecad254 100644 --- a/src/hyperloom/orchestrator/actions/executors/_grid_runner.py +++ b/src/hyperloom/orchestrator/actions/executors/_grid_runner.py @@ -412,126 +412,292 @@ def _build_variant_yaml( base_extra_envs: dict[str, str] | None = None, base_remove_args: list[str] | None = None, base_unset_envs: list[str] | None = None, + base_native_launch_overrides: dict[str, Any] | None = None, ) -> Path: """Materialize a per-variant Magpie YAML on disk.""" with base_yaml_path.open(encoding="utf-8") as f: cfg = yaml.safe_load(f) bench = cfg.setdefault("benchmark", {}) - replacing = str(base_args_mode).strip().lower() == "replace" - envs = apply_runtime_benchmark_overrides( - bench, - model_path=model_path, - gpu_type=gpu_type, - benchmark_script=benchmark_script, - conc=variant_conc(variant), - ) - extra_args_env = server_args_env_name(bench.get("framework")) - - variant_remove = to_str_list(getattr(variant, "remove_args", [])) - # A replacing base drops the inherited string wholesale, so only the - # variant's own removals still name flags that survive to be stripped. - effective_remove = ( - variant_remove if replacing else list(dict.fromkeys(to_str_list(base_remove_args) + variant_remove)) - ) - combined = compose_server_args( - inherited_args="" if replacing else str(envs.get(extra_args_env, "")), - base_extra_args=base_extra_args, - variant_extra_args=variant.extra_server_args, - remove_args=effective_remove, - args_mode=getattr(variant, "args_mode", "append"), - ) - # A grid variant never injects a MoE runner backend itself, but it does inherit one -- from the baseline recipe it - # was seeded with, or from an explicitly authored variant. - if combined and _SGLANG_MOE_RUNNER_BACKEND_RE.search(combined): - from ._workload_envs import _remove_moe_runner_backend_arg - if extra_args_env == "EXTRA_SGLANG_ARGS" and moe_runner_requires_aiter(combined, model_path): - log.warning( - "grid: dropping inherited --moe-runner-backend for variant %s: " - "this checkpoint's MoE quant scheme is only implemented on the " - "aiter runner and would crash on the first forward pass.", - variant.name, + # A native AgentX launcher owns the complete server command line. The + # pinned InferenceX revision has no optimizer-argv hook, so accepting a + # normal Hyperloom variant here would run the unchanged server and then + # falsely label its result as the candidate. The base materializer has the + # same guard, but grid/GEAK variants are layered on *after* that point and + # therefore need their own fail-closed boundary. + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + native_agentx = native_agentx_enabled(bench.get("agentx")) + native_optimizer = False + native_expected_gpu_count: int | None = None + native_expected_topology: dict[str, Any] = {} + native_outer_image = "" + if native_agentx: + from ._native_candidate import apply_native_candidate, has_launch_contract + + native_optimizer = has_launch_contract(bench) + raw_workload = bench.get("workload_spec") + workload = raw_workload if isinstance(raw_workload, dict) else {} + raw_topology = workload.get("resolved_topology") + topology = raw_topology if isinstance(raw_topology, dict) else {} + try: + native_expected_gpu_count = int(topology["gpu_count"]) + except (KeyError, TypeError, ValueError) as exc: + raise ValueError( + "Native AgentX grid variants require fingerprint-bound workload_spec.resolved_topology.gpu_count" + ) from exc + if native_expected_gpu_count <= 0: + raise ValueError("Native AgentX resolved GPU count must be positive") + try: + native_expected_topology = { + "tp": int(topology["tp"]), + "pp": int(topology["pp"]), + "pcp_size": int(topology["pcp_size"]), + "ep": int(topology["ep"]), + "conc": int(topology["conc"]), + "recipe_fingerprint": str(topology["recipe_fingerprint"]), + } + except (KeyError, TypeError, ValueError) as exc: + raise ValueError( + "Native AgentX grid variants require a complete, fingerprint-bound resolved topology" + ) from exc + if ( + min( + int(native_expected_topology["tp"]), + int(native_expected_topology["pp"]), + int(native_expected_topology["pcp_size"]), + int(native_expected_topology["ep"]), + int(native_expected_topology["conc"]), ) - combined = _remove_moe_runner_backend_arg(combined) - if combined: - envs[extra_args_env] = _shell_safe_dedupe(combined) - elif extra_args_env in envs or variant.args_mode == "replace" or base_args_mode == "replace": - envs[extra_args_env] = "" - # Composed base-then-variant, so a variant unsetting a key the base sets - # removes it: the last layer to name a key is the one that decides it. - for k in to_str_list(base_unset_envs): - if k.strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: - continue - envs.pop(k, None) - for k, v in (base_extra_envs or {}).items(): - envs[str(k)] = str(v) - for k in getattr(variant, "unset_envs", []) or []: - # Unsetting a pin retargets the benchmark rather than toggling a knob. - if str(k).strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: - log.warning("grid: refusing to unset pinned env %s for variant %s", k, variant.name) - continue - envs.pop(str(k), None) - for k, v in variant.extra_envs.items(): - envs[str(k)] = str(v) - from ._workload_envs import pin_mlperf_round_concurrency - - pin_mlperf_round_concurrency(envs) - # The launcher re-exports these unconditionally, so a value carried here is - # one the run never used. - for k in launcher_overwritten_envs(bench) & envs.keys(): - log.warning("grid: dropping %s for variant %s; the recipe overwrites it", k, variant.name) - envs.pop(k, None) - # The three AgentX bounds took this rung's CONC through ``variant_conc`` above, not through this merge: raising - # the client's grace alone would make the round wait inside a cap that did not move with it. - _overlay = str(getattr(variant, "overlay_pythonpath", "") or "").strip() - if _overlay: - # Structural containment on the overlay dir before it is prepended to PYTHONPATH: a legitimate authored-kernel - # overlay is a single existing directory (never a ``:``-joined list, never a ``..`` traversal or control - # char). - _overlay_ok = ( - ":" not in _overlay - and ".." not in Path(_overlay).parts - and not any(c in _overlay for c in ("\n", "\r", "\x00")) - and Path(_overlay).is_dir() + <= 0 + or not native_expected_topology["recipe_fingerprint"] + ): + raise ValueError("Native AgentX resolved topology values and recipe fingerprint must be present") + native_outer_image = str(workload.get("outer_image") or "").strip() + + unsupported: list[str] = [] + if str(base_extra_args or "").strip(): + unsupported.append("base_extra_args") + if str(variant.extra_server_args or "").strip(): + unsupported.append("variant.extra_server_args") + if str(base_args_mode or "append").strip().lower() == "replace": + unsupported.append("base_args_mode=replace") + if str(getattr(variant, "args_mode", "append") or "append").strip().lower() == "replace": + unsupported.append("variant.args_mode=replace") + if to_str_list(base_remove_args) or to_str_list(getattr(variant, "remove_args", [])): + unsupported.append("remove_args") + if to_str_list(base_unset_envs) or to_str_list(getattr(variant, "unset_envs", [])): + unsupported.append("unset_envs") + unsupported_envs = sorted( + { + str(key) + for mapping in (base_extra_envs or {}, getattr(variant, "extra_envs", {}) or {}) + for key in mapping + if str(key) != "CONC" + } ) - if _overlay_ok: - _cur_pp = str(envs.get("PYTHONPATH", "") or "") - envs["PYTHONPATH"] = f"{_overlay}:{_cur_pp}" if _cur_pp else _overlay + if unsupported_envs: + unsupported.append("extra_envs=" + ",".join(unsupported_envs)) + if str(getattr(variant, "overlay_pythonpath", "") or "").strip(): + unsupported.append("overlay_pythonpath") + if getattr(variant, "runtime_override", None): + unsupported.append("runtime_override") + if unsupported and not native_optimizer: + raise ValueError( + "Native AgentX cannot apply Hyperloom grid candidates with " + "the pinned InferenceX launcher (no optimizer-argv hook): " + "; ".join(unsupported) + ) + + if native_optimizer: + if getattr(variant, "dropped_envs", None): + raise ValueError(f"Native candidate contains unsupported environment keys: {variant.dropped_envs}") + if base_native_launch_overrides is not None: + from ._native_candidate import install_native_launch_snapshot + + install_native_launch_snapshot(bench, base_native_launch_overrides) else: - log.warning( - "grid: dropping unsafe overlay_pythonpath %r (not a single " - "existing directory / contains separator or traversal)", - _overlay, + apply_native_candidate( + bench, + extra_server_args=base_extra_args, + extra_envs=base_extra_envs, + remove_args=base_remove_args, + unset_envs=base_unset_envs, + args_mode=base_args_mode, ) + apply_native_candidate( + bench, + extra_server_args=variant.extra_server_args, + extra_envs=variant.extra_envs, + remove_args=getattr(variant, "remove_args", []), + unset_envs=getattr(variant, "unset_envs", []), + args_mode=getattr(variant, "args_mode", "append"), + runtime_override=getattr(variant, "runtime_override", None), + overlay_pythonpath=str(getattr(variant, "overlay_pythonpath", "") or ""), + source_files=getattr(variant, "native_source_files", None), + absent_source_files=getattr(variant, "native_absent_source_files", ()), + ) + envs = bench.setdefault("envs", {}) + if server_lifecycle is not None: + raise ValueError("Native AgentX optimizer candidates cannot change server lifecycle") + else: + replacing = str(base_args_mode).strip().lower() == "replace" + envs = apply_runtime_benchmark_overrides( + bench, + model_path=model_path, + gpu_type=gpu_type, + benchmark_script=benchmark_script, + conc=variant_conc(variant), + ) + extra_args_env = server_args_env_name(bench.get("framework")) + + replacing = str(base_args_mode).strip().lower() == "replace" + variant_remove = to_str_list(getattr(variant, "remove_args", [])) + # A replacing base drops the inherited string wholesale, so only the + # variant's own removals still name flags that survive to be stripped. + effective_remove = ( + variant_remove if replacing else list(dict.fromkeys(to_str_list(base_remove_args) + variant_remove)) + ) + combined = compose_server_args( + inherited_args="" if replacing else str(envs.get(extra_args_env, "")), + base_extra_args=base_extra_args, + variant_extra_args=variant.extra_server_args, + remove_args=effective_remove, + args_mode=getattr(variant, "args_mode", "append"), + ) + # A grid variant never injects a MoE runner backend itself, but it does inherit one -- from the baseline recipe it + # was seeded with, or from an explicitly authored variant. + if combined and _SGLANG_MOE_RUNNER_BACKEND_RE.search(combined): + from ._workload_envs import _remove_moe_runner_backend_arg - # Attempt runtime override: inject path_prefix/pythonpath_prefix/framework_bin etc. into benchmark.envs so the - # server subprocess resolves the attempt runtime. - _rt_override = getattr(variant, "runtime_override", None) or {} - if _rt_override: - apply_runtime_override(envs, _rt_override) - - # PATH guard: the xdit wrapper needs both `/venv/bin` (the `xdit` console script) and `/opt/rocm/bin` (`hipcc`); - # force-prepend both so an LLM-supplied PATH can't drop one. - if str(bench.get("framework", "")).strip().lower() == "xdit": - _cur_path = str(envs.get("PATH", "") or "") - _parts = [p for p in _cur_path.split(":") if p] - for _essential in ("/opt/rocm/bin", "/venv/bin"): - if _essential not in _parts: - _parts.insert(0, _essential) - envs["PATH"] = ":".join(_parts) - - if server_lifecycle is not None: - from ._server_lifecycle import inject_lifecycle - - inject_lifecycle( + if extra_args_env == "EXTRA_SGLANG_ARGS" and moe_runner_requires_aiter(combined, model_path): + log.warning( + "grid: dropping inherited --moe-runner-backend for variant %s: " + "this checkpoint's MoE quant scheme is only implemented on the " + "aiter runner and would crash on the first forward pass.", + variant.name, + ) + combined = _remove_moe_runner_backend_arg(combined) + if combined: + envs[extra_args_env] = _shell_safe_dedupe(combined) + elif extra_args_env in envs or variant.args_mode == "replace" or base_args_mode == "replace": + envs[extra_args_env] = "" + # Composed base-then-variant, so a variant unsetting a key the base sets + # removes it: the last layer to name a key is the one that decides it. + for k in to_str_list(base_unset_envs): + if k.strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: + continue + envs.pop(k, None) + for k, v in (base_extra_envs or {}).items(): + envs[str(k)] = str(v) + for k in getattr(variant, "unset_envs", []) or []: + # Unsetting a pin retargets the benchmark rather than toggling a knob. + if str(k).strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: + log.warning("grid: refusing to unset pinned env %s for variant %s", k, variant.name) + continue + envs.pop(str(k), None) + for k, v in variant.extra_envs.items(): + envs[str(k)] = str(v) + from ._workload_envs import pin_mlperf_round_concurrency + + pin_mlperf_round_concurrency(envs) + # The recipe re-exports these unconditionally, so a value carried here is + # one the run never used. + for k in launcher_overwritten_envs(bench) & envs.keys(): + log.warning("grid: dropping %s for variant %s; the recipe overwrites it", k, variant.name) + envs.pop(k, None) + # The three AgentX bounds took this rung's CONC through ``variant_conc`` above, not through this merge: raising + # the client's grace alone would make the round wait inside a cap that did not move with it. + _overlay = str(getattr(variant, "overlay_pythonpath", "") or "").strip() + if _overlay: + # Structural containment on the overlay dir before it is prepended to PYTHONPATH: a legitimate authored-kernel + # overlay is a single existing directory (never a ``:``-joined list, never a ``..`` traversal or control + # char). + _overlay_ok = ( + ":" not in _overlay + and ".." not in Path(_overlay).parts + and not any(c in _overlay for c in ("\n", "\r", "\x00")) + and Path(_overlay).is_dir() + ) + if _overlay_ok: + _cur_pp = str(envs.get("PYTHONPATH", "") or "") + envs["PYTHONPATH"] = f"{_overlay}:{_cur_pp}" if _cur_pp else _overlay + else: + log.warning( + "grid: dropping unsafe overlay_pythonpath %r (not a single " + "existing directory / contains separator or traversal)", + _overlay, + ) + + # Attempt runtime override: inject path_prefix/pythonpath_prefix/framework_bin etc. into benchmark.envs so the + # server subprocess resolves the attempt runtime. + _rt_override = getattr(variant, "runtime_override", None) or {} + if _rt_override: + apply_runtime_override(envs, _rt_override) + + # PATH guard: the xdit wrapper needs both `/venv/bin` (the `xdit` console script) and `/opt/rocm/bin` (`hipcc`); + # force-prepend both so an LLM-supplied PATH can't drop one. + if str(bench.get("framework", "")).strip().lower() == "xdit": + _cur_path = str(envs.get("PATH", "") or "") + _parts = [p for p in _cur_path.split(":") if p] + for _essential in ("/opt/rocm/bin", "/venv/bin"): + if _essential not in _parts: + _parts.insert(0, _essential) + envs["PATH"] = ":".join(_parts) + + if server_lifecycle is not None: + from ._server_lifecycle import inject_lifecycle + + inject_lifecycle( + bench, + cleanup=bool(server_lifecycle.get("cleanup", True)), + pid_dir=server_lifecycle["pid_dir"], + port=int(server_lifecycle["port"]), + ) + if native_agentx: + # Concurrency is part of the measurement contract even though the + # current upstream recipe fingerprint excludes it. Never compare a + # candidate at a different replay load with the accepted baseline. + from hyperloom.inference_optimizer.agentx.native import resolve_native_recipe + + inferencex_path = ( + os.environ.get("INFERENCEX_PATH", "").strip() or str(bench.get("inferencex_path") or "").strip() + ) + if not inferencex_path: + raise ValueError("Native AgentX grid variants require benchmark.inferencex_path") + actual_conc = int(envs.get("CONC") or 0) + expected_conc = int(native_expected_topology["conc"]) + if actual_conc != expected_conc: + raise ValueError( + "Native AgentX concurrency is fixed for the session: " + f"accepted CONC={expected_conc}, candidate requested {actual_conc}" + ) + assert native_expected_gpu_count is not None + resolved_topology = resolve_native_recipe( bench, - cleanup=bool(server_lifecycle.get("cleanup", True)), - pid_dir=server_lifecycle["pid_dir"], - port=int(server_lifecycle["port"]), + inferencex_path=inferencex_path, + expected_gpu_count=native_expected_gpu_count, + outer_image=native_outer_image or None, ) + changed = { + key: (native_expected_topology[key], resolved_topology.get(key)) + for key in native_expected_topology + if native_expected_topology[key] != resolved_topology.get(key) + } + if changed: + detail = ", ".join(f"{key}={before!r}->{after!r}" for key, (before, after) in sorted(changed.items())) + raise ValueError( + "Native AgentX concurrency changed the accepted recipe arm; " + "topology-changing rounds are unsupported: " + detail + ) + envs = bench.setdefault("envs", {}) + + if native_optimizer: + from ._native_source import verify_native_source_imports - # The final write to the argument env; nothing below may touch it. - seal_server_argv(envs, bench.get("framework")) + verify_native_source_imports(bench) + else: + # The final write to the argument env; nothing below may touch it. + seal_server_argv(envs, bench.get("framework")) output_subdir.mkdir(parents=True, exist_ok=True) out_path = output_subdir / "config.yaml" with out_path.open("w", encoding="utf-8") as f: @@ -563,6 +729,7 @@ async def _settled_measurement( subprocess_started_unix: float | None, settle_seconds: float = REPORT_SETTLE_SECONDS, poll_seconds: float = REPORT_SETTLE_POLL_SECONDS, + materialized_config_path: Path | None = None, ) -> tuple[dict[str, Any] | None, dict[str, Any]]: """Read the benchmark report, re-reading briefly while it is still settling.""" deadline = time.monotonic() + max(0.0, float(settle_seconds)) @@ -573,6 +740,7 @@ async def _settled_measurement( report, workspace=workspace, subprocess_started_unix=subprocess_started_unix, + materialized_config_path=materialized_config_path, ) attempts += 1 if measurement.get("valid_measurement") or time.monotonic() >= deadline: @@ -604,9 +772,15 @@ def _prepend_magpie_pythonpath(magpie_dir: str, current_pythonpath: str) -> str: def sync_benchmark_timeout(config_path: Path, timeout_sec: float) -> None: - """Give Magpie and bypass the same cap as the enclosing benchmark process.""" + """Sync ordinary benchmark caps; native configs retain their fixed identity.""" cfg = yaml.safe_load(config_path.read_text(encoding="utf-8")) bench = cfg["benchmark"] + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + if native_agentx_enabled(bench.get("agentx")): + # The subprocess watchdog still enforces timeout_sec. Mutating the + # resolved config here would invalidate its workload and recipe hashes. + return bench["timeout_seconds"] = timeout_sec if bench.get("server_lifecycle"): bench["server_lifecycle"]["server_ready_timeout_s"] = timeout_sec @@ -642,6 +816,11 @@ def _run_magpie( """Blocking subprocess wrapper. Returns (rc, stdout, stderr).""" sync_benchmark_timeout(config_path, timeout_sec) server_log_path = _benchmark_server_log(config_path, output_dir) + config_data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + config_benchmark = config_data.get("benchmark") if isinstance(config_data, dict) else {} + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + native_agentx = isinstance(config_benchmark, dict) and native_agentx_enabled(config_benchmark.get("agentx")) env = scrub_benchmark_process_env(os.environ.copy()) env["PYTHONUNBUFFERED"] = "1" from ._workload_envs import resolve_reference_launch @@ -667,14 +846,20 @@ def _run_magpie( env["MAGPIE_INFERENCEX_PATH"] = inferencex_path # Baseline patches its own checkout, but explore / sweep never pass through that hook: re-assert here so a # resumed session or a re-cloned checkout still emits the eval-start marker. - ensure_benchmark_lib_eval_start_patched(Path(inferencex_path)) + if not native_agentx: + ensure_benchmark_lib_eval_start_patched(Path(inferencex_path)) # The generation bounds + pathology probe are asserted whether or not ``$INFERENCEX_PATH`` is set: unset falls # back to the same env discovery the baseline arm uses ($MAGPIE_PATH/InferenceX). probe_root = Path(inferencex_path) if inferencex_path else None # Best-effort, unlike the probe below: a missing guard only costs the reason a failed round reports. - ensure_eval_unbound_outputs_patched(probe_root) - if not ensure_eval_probe_patched(probe_root) and not materialized_run_eval_disabled(config_path): + if not native_agentx: + ensure_eval_unbound_outputs_patched(probe_root) + if ( + not native_agentx + and not ensure_eval_probe_patched(probe_root) + and not materialized_run_eval_disabled(config_path) + ): eval_bounds_msg = ( "eval generation bounds + pathology probe are not installed " "(utils/evals/patches/lm_eval_sitecustomize.py, inferencex=" @@ -698,7 +883,8 @@ def _run_magpie( # RESULT_DIR default; leaks are picked up by the salvage path. env["RESULT_DIR"] = result_dir or str(output_dir) # InferenceX ``run_lm_eval`` cleans ``$EVAL_RESULT_DIR`` after processing lm-eval output. - env["EVAL_RESULT_DIR"] = str(Path(env["RESULT_DIR"]) / "eval_output") + if not native_agentx: + env["EVAL_RESULT_DIR"] = str(Path(env["RESULT_DIR"]) / "eval_output") # Pin SERVER_LOG / GPU_METRICS_CSV per-task so logs land alongside ``benchmark_report.json``. env["SERVER_LOG"] = str(output_dir / "server.log") env["GPU_METRICS_CSV"] = str(output_dir / "gpu_metrics.csv") @@ -752,15 +938,37 @@ def _run_magpie( def _num_gpus_for_config(config_path: Path) -> float: - """Read the tensor-parallel size (``TP``) from a materialized benchmark YAML.""" + """Read the physical serving GPU count from a materialized benchmark YAML.""" try: with Path(config_path).open(encoding="utf-8") as fp: cfg = yaml.safe_load(fp) or {} - envs = (cfg.get("benchmark") or {}).get("envs") or {} - return float(int(envs.get("TP", 1) or 1)) + benchmark = cfg.get("benchmark") or {} except Exception: # noqa: BLE001 — best-effort; default to 1 GPU return 1.0 + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + workload = benchmark.get("workload_spec") or {} + topology = workload.get("resolved_topology") or {} + if native_agentx_enabled(benchmark.get("agentx")): + # Invalid native metadata is not a reason to under-lease one GPU. It + # is an integrity failure and must stop before Ray schedules the run. + gpu_count = int(topology["gpu_count"]) + tp = int(topology["tp"]) + pp = int(topology["pp"]) + pcp = int(topology["pcp_size"]) + if min(gpu_count, tp, pp, pcp) <= 0 or gpu_count != tp * pp * pcp: + raise ValueError("invalid native AgentX resolved topology") + return float(gpu_count) + try: + envs = benchmark.get("envs") or {} + tp = int(envs.get("TP", 1) or 1) + pp = int(envs.get("PP_SIZE", 1) or 1) + pcp = int(envs.get("PCP_SIZE", 1) or 1) + return float(tp * pp * pcp) + except (AttributeError, OverflowError, TypeError, ValueError): + return 1.0 + def _mn_restart_env( reference_envs: dict[str, str], @@ -886,6 +1094,7 @@ async def run_grid( base_extra_envs: dict[str, str] | None = None, base_remove_args: list[str] | None = None, base_unset_envs: list[str] | None = None, + base_native_launch_overrides: dict[str, Any] | None = None, warmup_before_measure: bool | None = None, server_already_ready: bool = False, serving_lease: Any = None, @@ -1108,6 +1317,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant base_extra_envs=base_extra_envs, base_remove_args=base_remove_args, base_unset_envs=base_unset_envs, + base_native_launch_overrides=base_native_launch_overrides, ) except Exception as exc: # noqa: BLE001 build_error = "yaml_build_error" @@ -1174,6 +1384,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant base_extra_envs=base_extra_envs, base_remove_args=base_remove_args, base_unset_envs=base_unset_envs, + base_native_launch_overrides=base_native_launch_overrides, ) else: log.info( @@ -1210,6 +1421,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant base_extra_envs=base_extra_envs, base_remove_args=base_remove_args, base_unset_envs=base_unset_envs, + base_native_launch_overrides=base_native_launch_overrides, ) except Exception as exc: # noqa: BLE001 log.warning( @@ -1326,6 +1538,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant _, warmup_measurement = await _settled_measurement( warmup_workspace, subprocess_started_unix=warmup_started_unix, + materialized_config_path=warmup_cfg_path, settle_seconds=REPORT_SETTLE_SECONDS if warmup_rc == 0 else 0.0, ) else: @@ -1333,6 +1546,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant None, workspace=warmup_workspace, subprocess_started_unix=warmup_started_unix, + materialized_config_path=warmup_cfg_path, ) if warmup_rc != 0 or not warmup_measurement.get("valid_measurement"): _teardown_variant_server(slot, lifecycle) @@ -1839,8 +2053,13 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant report, measurement = await _settled_measurement( workspace, subprocess_started_unix=variant_started_unix, + materialized_config_path=cfg_path, settle_seconds=REPORT_SETTLE_SECONDS, ) + if measurement.get("agentx_launch_contract") == 1: + from ._native_candidate import record_launch_evidence + + record_launch_evidence(cfg_path, measurement) nonzero_kept_error: str | None = None warnings = list(measurement.pop("nonfatal_warnings", []) or []) for leak_src, _ in harvested: @@ -1990,6 +2209,7 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant output_throughput=measurement.get("output_throughput"), request_throughput=measurement.get("request_throughput"), total_token_throughput=measurement.get("total_token_throughput"), + agentx_gpu_count=measurement.get("agentx_gpu_count"), completed_requests=measurement.get("completed_requests"), duration_seconds=measurement.get("duration_seconds"), ttft_mean_ms=measurement.get("ttft_mean_ms"), @@ -2000,6 +2220,23 @@ def _skip_rest_for_budget(idx: int, *, spent_on: str, rounds_left: int = variant intvty_p90=measurement.get("e2e_norm_intvty_p90"), intvty_p50=measurement.get("e2e_norm_intvty_p50"), request_error_rate=measurement.get("request_error_rate"), + materialized_config=str(cfg_path) if measurement.get("agentx_launch_contract") == 1 else None, + native_measurement={ + key: value + for key, value in measurement.items() + if measurement.get("agentx_launch_contract") == 1 + and ( + key.startswith(("agentx_", "native_agentx_")) + or key + in { + "valid_measurement", + "benchmark_valid", + "publishable", + "submission_valid", + "invalid_measurement_reasons", + } + ) + }, workspace=str(workspace), report_path=str(report_path) if report_path.exists() else None, raw_result_path=measurement.get("raw_result_path"), diff --git a/src/hyperloom/orchestrator/actions/executors/_inferencex_patcher.py b/src/hyperloom/orchestrator/actions/executors/_inferencex_patcher.py index dafc00aca4..a2aa00b6fc 100644 --- a/src/hyperloom/orchestrator/actions/executors/_inferencex_patcher.py +++ b/src/hyperloom/orchestrator/actions/executors/_inferencex_patcher.py @@ -71,6 +71,35 @@ _EVAL_START_SENTINEL = "HYPERLOOM_EVAL_START" _EVAL_START_LOCK_PATH = str(Path(tempfile.gettempdir()) / "hyperloom_benchmark_lib_eval_start_patcher.lock") +# Exact alternatives for the infx package layout. Legacy sessions still use the +# original shell/Python text; selection is by content, never by a version guess. +_ANCHOR_ALTERNATIVES: dict[str, dict[str, str]] = { + _BENCH_SERVING_LEGACY: { + ' extra_body={\n "num_steps": 1,\n "merge_profiles": True,\n "profile_by_stage": True,\n },': ' extra_body=__import__("json").loads(__import__("os").environ.get("PROFILE_EXTRA_BODY") or ' + '\'{"num_steps": 1, "merge_profiles": true, "profile_by_stage": true}\'),', + }, + _EVAL_DEST_LEGACY: { + ' target="./${stem}_conc${eval_conc}${extension}"\n' + " suffix=2\n" + ' while [ -e "$target" ]; do\n' + ' target="./${stem}_conc${eval_conc}_${suffix}${extension}"': ' local target_dir="${RESULT_DIR:-.}/"\n' + ' target="${target_dir}${stem}_conc${eval_conc}${extension}"\n' + " suffix=2\n" + ' while [ -e "$target" ]; do\n' + ' target="${target_dir}${stem}_conc${eval_conc}_${suffix}${extension}"', + }, + _EVAL_START_LEGACY: { + " # Read by append_lm_eval_summary.\n" + _EVAL_START_LEGACY: " # Read by append_lm_eval_summary.\n" + + _EVAL_START_PATCHED, + }, +} + + +def _matching_patch_anchor(text: str, legacy: str) -> str: + """Select the supported upstream spelling before counting or replacing it.""" + return next((anchor for anchor in _ANCHOR_ALTERNATIVES.get(legacy, {}) if anchor in text), legacy) + + # Two independent answers to the same budget, injected together. _EVAL_PROBE_PY = """ # --- HYPERLOOM_EVAL_PROBE (early-exit probe + per-request bounds) ------------ @@ -542,12 +571,19 @@ async def _hl_amodel_call(self, *args, **kwargs): # Appending needs no anchor, but it does need this file: upstream renaming or moving it puts the probe and the bounds # back to warn-only, and the eval runs unbounded again. EVAL_PROBE_TARGET_PARTS = ("utils", "evals", "patches", "lm_eval_sitecustomize.py") +EVAL_PROBE_TARGETS = (("infx", *EVAL_PROBE_TARGET_PARTS[1:]), EVAL_PROBE_TARGET_PARTS) + + +def _inferencex_project_root(path: Path | str) -> Path: + root = Path(path) + nested = root / "inferencex-e2e" + return nested if (nested / "benchmarks" / "benchmark_lib.sh").is_file() else root def _discover_inferencex_roots( inferencex_path: Path | str | None, ) -> list[Path]: - """Return every InferenceX checkout root Hyperloom should patch.""" + """Use the explicit checkout, or discover defaults when none is supplied.""" roots: list[Path] = [] seen: set[Path] = set() @@ -556,7 +592,7 @@ def _add(candidate: Path | str | None) -> None: if not candidate: return try: - resolved = Path(candidate).expanduser().resolve() + resolved = _inferencex_project_root(Path(candidate).expanduser().resolve()) except OSError: return if not resolved.is_dir(): @@ -567,6 +603,8 @@ def _add(candidate: Path | str | None) -> None: roots.append(resolved) _add(inferencex_path) + if inferencex_path is not None: + return roots _add(os.environ.get("INFERENCEX_PATH", "").strip() or None) magpie_dir = (os.environ.get("MAGPIE_PATH") or "").strip() if magpie_dir: @@ -614,6 +652,9 @@ def _apply_line_replacement_atomic( log.warning("_inferencex_patcher: cannot read %s: %s", src, e) return False + selected = _matching_patch_anchor(original, legacy) + patched_line = _ANCHOR_ALTERNATIVES.get(legacy, {}).get(selected, patched_line) + legacy = selected if legacy not in original: log.warning(missing_msg, src) return False @@ -712,7 +753,7 @@ def _pick_benchmark_serving(candidates: list[Path]) -> Path: text = path.read_text(encoding="utf-8", errors="replace") except OSError: continue - if _BENCH_SERVING_SENTINEL in text or _BENCH_SERVING_LEGACY in text: + if _BENCH_SERVING_SENTINEL in text or count_anchor_hits(text, _BENCH_SERVING_LEGACY): return path return candidates[-1] @@ -726,7 +767,7 @@ def benchmark_serving_path_in(root: Path | str) -> Path: because a gate speaks for the tree the run will execute, not for whatever else the environment can reach. """ - base = Path(root) + base = _inferencex_project_root(root) existing = [path for path in (base.joinpath(*rel) for rel in _BENCH_SERVING_REL_PARTS) if path.is_file()] if not existing: return base.joinpath(*_BENCH_SERVING_REL_PARTS[-1]) @@ -866,8 +907,16 @@ def ensure_benchmark_lib_eval_start_patched( def _resolve_eval_sitecustomize_paths( inferencex_path: Path | str | None, ) -> list[Path]: - """Return every existing ``/utils/evals/patches/lm_eval_sitecustomize.py``.""" - return _resolve_inferencex_files(inferencex_path, *EVAL_PROBE_TARGET_PARTS) + """Resolve the active eval patch module in either supported project layout.""" + return [ + path + for root in _discover_inferencex_roots(inferencex_path) + if ( + path := next( + (root.joinpath(*parts) for parts in EVAL_PROBE_TARGETS if root.joinpath(*parts).is_file()), None + ) + ) + ] def eval_probe_targets_exist(inferencex_path: Path | str | None = None) -> bool: @@ -1043,7 +1092,7 @@ def describe(self) -> str: ("eval_start", ("benchmarks", "benchmark_lib.sh"), _EVAL_START_SENTINEL, _EVAL_START_LEGACY), ( _BENCH_SERVING_ANCHOR_NAME, - # The pinned revision's layout; newer checkouts are resolved by name, not by this path. + # Legacy sessions retain this path; newer layouts resolve the implementation by name. ("utils", "bench_serving", "benchmark_serving.py"), _BENCH_SERVING_SENTINEL, _BENCH_SERVING_LEGACY, @@ -1052,8 +1101,8 @@ def describe(self) -> str: def count_anchor_hits(text: str, anchor: str) -> int: - """Return how many sites in ``text`` the given anchor would rewrite.""" - return text.count(anchor) + """Count the supported upstream spelling that this patch would rewrite.""" + return text.count(_matching_patch_anchor(text, anchor)) def verify_patch_anchors( @@ -1138,7 +1187,7 @@ def failed_patch_anchors_in(root: Path | str) -> list[AnchorStatus]: Returns: The failing subset for that root alone. """ - base = Path(root) + base = _inferencex_project_root(root) def _resolve(name: str, parts: tuple[str, ...]) -> list[Path]: """Name the one file under ``base`` a patch targets, when it exists. diff --git a/src/hyperloom/orchestrator/actions/executors/_managed_profile_result.py b/src/hyperloom/orchestrator/actions/executors/_managed_profile_result.py new file mode 100644 index 0000000000..070eb65297 --- /dev/null +++ b/src/hyperloom/orchestrator/actions/executors/_managed_profile_result.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Validate Magpie diagnostic evidence without accepting a performance measurement.""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import yaml + + +def _capture_rounds(capture: Mapping[str, Any], workspace: Path, expected_ranks: int) -> list[dict[str, Any]]: + capture_id = capture.get("capture_id") + if not isinstance(capture_id, str) or not capture_id or Path(capture_id).name != capture_id: + raise ValueError("AgentX profile capture has no valid capture_id") + root = (workspace / "torch_trace" / capture_id).resolve() + artifact = json.loads((root / "capture.json").read_text(encoding="utf-8")) + if artifact != capture: + raise ValueError("AgentX profile report disagrees with its capture manifest") + if type(capture.get("expected_ranks")) is not int or capture["expected_ranks"] != expected_ranks: + raise ValueError("AgentX profile rank count disagrees with the accepted topology") + rounds = capture.get("profiles") + if rounds is None: + rounds = [dict(capture, profile_index=1, trace_dir=str(root))] + elif not isinstance(rounds, list) or not rounds: + raise ValueError("AgentX profile series has no completed captures") + if "completed_profiles" in capture and ( + capture["completed_profiles"] != len(rounds) or capture.get("effective_profiles") != len(rounds) + ): + raise ValueError("AgentX profile series completion accounting is inconsistent") + return [_validated_round(row, index, root, capture) for index, row in enumerate(rounds, 1)] + + +def _validated_round(row: Any, index: int, root: Path, capture: Mapping[str, Any]) -> dict[str, Any]: + if not isinstance(row, dict) or row.get("status") != "complete" or row.get("profile_index") != index: + raise ValueError("AgentX profile series contains an incomplete or misordered capture") + for key in ("capture_id", "framework", "num_steps", "expected_ranks"): + if type(row.get(key)) is not type(capture[key]) or row[key] != capture[key]: + raise ValueError(f"AgentX profile round {key} disagrees with its capture") + directory = Path(row["trace_dir"]).resolve() + if not directory.is_relative_to(root): + raise ValueError("AgentX profile round is outside this capture directory") + files = [Path(path).resolve() for path in row.get("trace_files") or []] + if not files or any(not path.is_relative_to(directory) or not path.is_file() for path in files): + raise ValueError("AgentX profile trace is missing or outside its capture round") + ranks = row.get("rank_trace_files") + expected = row.get("expected_ranks") + if type(expected) is not int or expected <= 0 or not isinstance(ranks, dict): + raise ValueError("AgentX profile capture has no verified rank manifest") + if set(ranks) != {str(rank) for rank in range(expected)} or any(not paths for paths in ranks.values()): + raise ValueError("AgentX profile capture is missing verified worker ranks") + rank_files = {rank: [str(Path(path).resolve()) for path in paths] for rank, paths in ranks.items()} + if sorted(path for paths in rank_files.values() for path in paths) != sorted(map(str, files)): + raise ValueError("AgentX profile rank manifest disagrees with the trace inventory") + return {**row, "trace_dir": str(directory), "trace_files": list(map(str, files)), "rank_trace_files": rank_files} + + +def diagnostic_profile_result( + report: Mapping[str, Any] | None, + *, + workspace: Path, + config_path: Path, + returncode: int, + subprocess_started_unix: float, +) -> dict[str, Any]: + """Require runtime, receipt and complete traces, while keeping KEEP gates false.""" + result: dict[str, Any] = { + "status": "failed", + "diagnostic_only": True, + "valid_measurement": False, + "benchmark_valid": False, + "publishable": False, + "submission_valid": False, + "trace_input_ready": False, + } + try: + metrics, rounds = _validate_diagnostic_report( + report, workspace, config_path, returncode, subprocess_started_unix + ) + capture = metrics["profile_capture"] + evidence = metrics["server_launch"] + selected = rounds[-1] + main_trace = max(selected["rank_trace_files"]["0"], key=lambda path: (Path(path).stat().st_size, path)) + result.update( + status="succeeded", + reported_success=True, + trace_input_ready=True, + agentx_server_launch=evidence, + profile_capture=capture, + profile_analyses=metrics.get("profile_analyses") or [], + profile_rounds=rounds, + selected_profile_index=selected["profile_index"], + trace_capture=capture, + trace_capture_status="complete", + trace_dir=selected["trace_dir"], + trace_files=selected["trace_files"], + rank_trace_paths=selected["rank_trace_files"], + main_trace_path=main_trace, + primary_rank=0, + profile_trace_selection_reason="latest_complete_profile_rank_0", + trace_manifest_path=str(workspace / "torch_trace" / capture["capture_id"] / "capture.json"), + ) + except (OSError, ValueError, TypeError, KeyError, yaml.YAMLError) as exc: + result.update(error_class="invalid_profile_diagnostic", error=str(exc)) + return result + + +def _validate_diagnostic_report( + report: Mapping[str, Any] | None, + workspace: Path, + config_path: Path, + returncode: int, + subprocess_started_unix: float, +) -> tuple[Mapping[str, Any], list[dict[str, Any]]]: + from hyperloom.inference_optimizer.agentx.identity import validate_server_launch + + if returncode != 0: + raise ValueError(f"Magpie diagnostic process exited with status {returncode}") + if not report or report.get("success") is not True or report.get("errors"): + raise ValueError(f"Magpie diagnostic runtime failed: {(report or {}).get('errors') or 'no successful report'}") + if (workspace / "benchmark_report.json").stat().st_mtime + 1 < subprocess_started_unix: + raise ValueError("Magpie diagnostic report predates this run") + metrics = report.get("agentx_metrics") or {} + capture = metrics.get("profile_capture") or {} + if metrics.get("diagnostic_only") is not True or capture.get("status") != "complete": + raise ValueError(f"Magpie diagnostic capture is incomplete: {capture.get('error') or capture.get('status')}") + benchmark = (yaml.safe_load(config_path.read_text(encoding="utf-8")) or {})["benchmark"] + evidence = metrics.get("server_launch") + errors = validate_server_launch(benchmark, evidence, workspace=workspace) + if errors: + raise ValueError("Magpie diagnostic launch is not verified: " + "; ".join(errors)) + receipt = json.loads((workspace / "agentx_server_launch.json").read_text(encoding="utf-8")) + if receipt != evidence: + raise ValueError("Magpie diagnostic report disagrees with its server launch receipt") + return metrics, _bound_capture_rounds(capture, evidence, benchmark, workspace) + + +def _bound_capture_rounds( + capture: Mapping[str, Any], evidence: Mapping[str, Any], benchmark: Mapping[str, Any], workspace: Path +) -> list[dict[str, Any]]: + """Bind trace inventory to the current launch and its accepted worker topology.""" + profiling = evidence.get("torch_profiler") or {} + if not capture.get("capture_id") or capture["capture_id"] != profiling.get("capture_id"): + raise ValueError("Magpie diagnostic capture ID disagrees with its server launch receipt") + directory = (workspace / "torch_trace" / capture["capture_id"]).resolve() + if not profiling.get("trace_dir") or Path(profiling["trace_dir"]).resolve() != directory: + raise ValueError("Magpie diagnostic capture directory disagrees with its server launch receipt") + if capture.get("framework") != evidence.get("framework"): + raise ValueError("Magpie diagnostic capture framework disagrees with its server launch receipt") + if type(capture.get("num_steps")) is not int or capture["num_steps"] != profiling.get("num_steps"): + raise ValueError("Magpie diagnostic capture step count disagrees with its server launch receipt") + expected_ranks = benchmark["workload_spec"]["resolved_topology"]["gpu_count"] + if type(expected_ranks) is not int or expected_ranks <= 0: + raise ValueError("Magpie diagnostic configuration has no trusted GPU count") + return _capture_rounds(capture, workspace, expected_ranks) diff --git a/src/hyperloom/orchestrator/actions/executors/_native_args.py b/src/hyperloom/orchestrator/actions/executors/_native_args.py new file mode 100644 index 0000000000..ca5e10ef1c --- /dev/null +++ b/src/hyperloom/orchestrator/actions/executors/_native_args.py @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Compose argv tokens for the native launch contract without shell rewriting.""" + +from __future__ import annotations + +import shlex +from collections.abc import Sequence + + +def compose_native_args(inherited: Sequence[str], extra: str, *, remove: Sequence[str], replace: bool) -> list[str]: + """Retain JSON/space operands and apply removals to the intended layer.""" + proposed = shlex.split(extra) + before = [] if replace else list(inherited) + if replace: + before, proposed = proposed, [] + kept: list[str] = [] + skipping = False + for token in before: + if token.startswith("--"): + skipping = token.partition("=")[0] in remove + if not skipping: + kept.append(token) + return kept + proposed diff --git a/src/hyperloom/orchestrator/actions/executors/_native_candidate.py b/src/hyperloom/orchestrator/actions/executors/_native_candidate.py new file mode 100644 index 0000000000..8a80334fe6 --- /dev/null +++ b/src/hyperloom/orchestrator/actions/executors/_native_candidate.py @@ -0,0 +1,270 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Translate optimizer candidates into Magpie's native server launch contract.""" + +from __future__ import annotations + +import os +import shlex +from copy import deepcopy +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +import yaml + +from hyperloom.common.coerce import to_str_list +from hyperloom.common.env_safety import BLOCKED_CHILD_ENV_NAMES, _ENV_KEY_RE +from hyperloom.inference_optimizer.agentx.identity import has_launch_contract +from ._native_args import compose_native_args +from ._native_source import source_file_hashes, verify_native_source_imports + +_FIXED_ENVS = frozenset( + { + "MODEL", + "MODEL_PATH", + "FRAMEWORK", + "TP", + "EP", + "PP", + "PCP_SIZE", + "DCP_SIZE", + "DP", + "CONC", + "ISL", + "OSL", + "NUM_PROMPTS", + "NUM_WARMUPS", + "RANDOM_RANGE_RATIO", + "RUN_EVAL", + "ROCR_VISIBLE_DEVICES", + "HIP_VISIBLE_DEVICES", + "CUDA_VISIBLE_DEVICES", + "AIPERF_BIN", + "WEKA_LOADER_OVERRIDE", + } +) +_RUNTIME_KEYS = frozenset( + { + "path_prefix", + "entrypoint_bin_dir", + "pythonpath_prefix", + "pythonpath_prefixes", + "ld_library_path_prefix", + "runtime_env", + "framework_bin", + "framework_python", + "framework_venv_root", + "runtime_python_exe", + } +) + + +def install_native_launch_snapshot(benchmark: dict[str, Any], snapshot: Mapping[str, Any]) -> None: + """Restore the accepted server implementation without replaying its deltas.""" + if not has_launch_contract(benchmark) or not has_launch_contract({"agentx": {"launch_overrides": snapshot}}): + raise ValueError("Native launch snapshots require an existing version 1 optimization contract") + benchmark["agentx"]["launch_overrides"] = deepcopy(dict(snapshot)) + + +def _fixed_env(name: str) -> bool: + return name in _FIXED_ENVS or name.startswith(("AGENTX_", "AIPERF_", "INFERENCEX_")) + + +def _runtime_candidate(benchmark: Mapping[str, Any], override: Mapping[str, Any], env: dict[str, str]) -> str: + """Resolve the exact server interpreter/entrypoint and runtime search paths.""" + unknown = set(override).difference(_RUNTIME_KEYS) + if unknown: + raise ValueError(f"Unsupported native runtime_override fields: {sorted(unknown)}") + from ._grid_runner import _RUNTIME_ENV_RESERVED, apply_runtime_override + + runtime_env = override.get("runtime_env") or {} + if not isinstance(runtime_env, Mapping): + raise ValueError("Native runtime_env must be an environment mapping") + for name in runtime_env: + if ( + _fixed_env(str(name)) + or str(name) in BLOCKED_CHILD_ENV_NAMES | _RUNTIME_ENV_RESERVED + or not _ENV_KEY_RE.fullmatch(str(name)) + ): + raise ValueError(f"Native runtime cannot override protected environment: {name}") + for key in ( + "path_prefix", + "entrypoint_bin_dir", + "pythonpath_prefix", + "pythonpath_prefixes", + "ld_library_path_prefix", + ): + raw = override.get(key) + for entry in [raw] if isinstance(raw, str) else raw or []: + path = Path(str(entry)).expanduser() + if not path.is_absolute() or not path.is_dir() or ":" in str(path) or any(c in str(path) for c in "\n\r\0"): + raise ValueError(f"Native runtime requires an existing absolute directory for {key}: {entry}") + combined = {**{str(k): str(v) for k, v in (benchmark.get("envs") or {}).items()}, **env} + before = dict(combined) + apply_runtime_override(combined, dict(override)) + env.update({k: v for k, v in combined.items() if before.get(k) != v and not k.startswith("HYPERLOOM_FRAMEWORK_")}) + framework = str(benchmark.get("framework") or "").lower() + python = str(override.get("runtime_python_exe") or override.get("framework_python") or "") + bin_path = str(override.get("framework_bin") or "") + venv = str(override.get("framework_venv_root") or "") + if framework == "sglang": + executable = python or (str(Path(venv) / "bin/python") if venv else "") + if not executable and bin_path: + root = Path(bin_path) if Path(bin_path).is_dir() else Path(bin_path).parent + executable = str(root / "python") + elif framework == "vllm": + bin_root = ( + Path(bin_path) if bin_path else Path(python).parent if python else Path(venv) / "bin" if venv else None + ) + executable = str(bin_root / "vllm") if bin_root and bin_root.is_dir() else str(bin_root or "") + else: + raise ValueError(f"Native runtime override is unsupported for framework {framework!r}") + if executable: + path = Path(executable).expanduser() + if not path.is_absolute() or not path.is_file() or not os.access(path, os.X_OK): + raise ValueError(f"Native server executable is not an absolute executable file: {executable}") + return str(path) + if python or bin_path or venv: + raise ValueError("Native runtime does not identify a usable server executable") + return "" + + +def _base_removals(benchmark: Mapping[str, Any], existing: Mapping[str, Any], requested: Any) -> list[str]: + removed = to_str_list(existing.get("remove_args")) + appended = {token.split("=", 1)[0] for token in existing.get("append_args", []) if token.startswith("--")} + workload = benchmark.get("workload_spec") or {} + observed = workload.get("server_launch") or {} + base = observed.get("base_argv") + base_flags = {token.split("=", 1)[0] for token in base or [] if token.startswith("--")} + for raw in to_str_list(requested): + tokens = shlex.split(raw) + if len(tokens) != 1 or not tokens[0].startswith("--") or "=" in tokens[0]: + raise ValueError(f"Native remove_args requires a flag name: {raw!r}") + name = tokens[0] + if name in appended and base is None: + raise ValueError("Removing an inherited native candidate flag requires verified base_argv evidence") + if name in appended and name not in base_flags: + continue + if name not in removed: + removed.append(name) + return removed + + +def apply_native_candidate( + benchmark: dict[str, Any], + *, + extra_server_args: str = "", + extra_envs: Mapping[str, Any] | None = None, + remove_args: Any = None, + unset_envs: Any = None, + args_mode: str = "append", + runtime_override: Mapping[str, Any] | None = None, + overlay_pythonpath: str = "", + source_files: Mapping[str, str] | None = None, + absent_source_files: Sequence[str] = (), +) -> None: + """Compose one candidate layer without changing the canonical benchmark inputs.""" + if not has_launch_contract(benchmark): + raise ValueError("Native optimization requires an accepted launch_overrides version 1 contract") + agentx = benchmark["agentx"] + existing = dict(agentx["launch_overrides"]) + env = {str(k): str(v) for k, v in (existing.get("env") or {}).items()} + fixed = benchmark.get("envs") or {} + removed_env = to_str_list(existing.get("unset_env")) + for name in to_str_list(unset_envs): + if _fixed_env(name) or name in BLOCKED_CHILD_ENV_NAMES or not _ENV_KEY_RE.fullmatch(name): + raise ValueError(f"Native candidate cannot unset protected environment: {name}") + env.pop(name, None) + if name not in removed_env: + removed_env.append(name) + for key, raw in (extra_envs or {}).items(): + name, value = str(key), str(raw) + if _fixed_env(name): + if name in fixed and str(fixed[name]) == value: + continue + raise ValueError(f"Native candidate cannot change canonical workload environment: {name}") + if name in BLOCKED_CHILD_ENV_NAMES or not _ENV_KEY_RE.fullmatch(name): + raise ValueError(f"Native candidate environment is not allowed: {name}") + env[name] = value + if name in removed_env: + removed_env.remove(name) + replace = str(args_mode).lower() == "replace" + removed_args = _base_removals(benchmark, existing, remove_args) + args = compose_native_args( + existing.get("append_args") or [], + str(extra_server_args or ""), + remove=to_str_list(remove_args), + replace=replace, + ) + files = dict(existing.get("source_files") or {}) + if overlay_pythonpath: + overlay = Path(overlay_pythonpath).expanduser() + if not overlay.is_absolute() or not overlay.is_dir() or not (overlay / "sitecustomize.py").is_file(): + raise ValueError("Native kernel overlay must be an existing absolute directory with sitecustomize.py") + if ":" in str(overlay) or any(c in str(overlay) for c in "\n\r\0"): + raise ValueError("Native kernel overlay must be one directory") + inherited = env.get("PYTHONPATH", str(fixed.get("PYTHONPATH") or "")) + env["PYTHONPATH"] = ":".join( + [str(overlay), *[part for part in inherited.split(":") if part and part != str(overlay)]] + ) + files.update( + source_file_hashes(sorted(p for p in overlay.rglob("*") if p.is_file() and "__pycache__" not in p.parts)) + ) + executable = str(existing.get("executable") or "") + if runtime_override: + executable = _runtime_candidate(benchmark, runtime_override, env) or executable + files.update(dict(source_files or {})) + absent = set(existing.get("absent_source_files") or []).difference(source_files or {}) + absent.update(absent_source_files) + for path in absent_source_files: + files.pop(path, None) + updated: dict[str, Any] = { + **existing, + "version": 1, + "append_args": args, + "remove_args": removed_args, + "replace_args": bool(existing.get("replace_args")) or replace, + "env": dict(sorted(env.items())), + "unset_env": sorted(removed_env), + "source_files": dict(sorted(files.items())), + "absent_source_files": sorted(absent), + } + if executable: + updated["executable"] = executable + agentx["launch_overrides"] = updated + + +def update_native_candidate_file(config_path: Path, **candidate: Any) -> None: + """Attach post-apply source/runtime evidence to the YAML consumed by Magpie.""" + from hyperloom.inference_optimizer.agentx.native import resolve_native_recipe + + config = yaml.safe_load(config_path.read_text(encoding="utf-8")) + benchmark = config["benchmark"] + apply_native_candidate(benchmark, **candidate) + topology = (benchmark.get("workload_spec") or {}).get("resolved_topology") or {} + resolve_native_recipe( + benchmark, + inferencex_path=str(benchmark.get("inferencex_path") or os.environ.get("INFERENCEX_PATH") or ""), + expected_gpu_count=int(os.environ.get("HYPERLOOM_AGENTX_GPU_COUNT") or topology.get("gpu_count") or 0), + ) + verify_native_source_imports(benchmark) + config_path.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + + +def record_launch_evidence(config_path: Path, measurement: Mapping[str, Any]) -> None: + """Retain verified base argv for composing later candidates against this recipe.""" + if measurement.get("valid_measurement") is not True or measurement.get("agentx_launch_contract") != 1: + return + evidence = measurement.get("agentx_server_launch") + if not isinstance(evidence, Mapping) or measurement.get("agentx_candidate_fingerprint") != evidence.get( + "evidence_sha256" + ): + return + config = yaml.safe_load(config_path.read_text(encoding="utf-8")) + benchmark = config["benchmark"] + if not has_launch_contract(benchmark): + return + benchmark.setdefault("workload_spec", {})["server_launch"] = dict(evidence) + config_path.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") diff --git a/src/hyperloom/orchestrator/actions/executors/_native_profile.py b/src/hyperloom/orchestrator/actions/executors/_native_profile.py new file mode 100644 index 0000000000..4b2cf373fe --- /dev/null +++ b/src/hyperloom/orchestrator/actions/executors/_native_profile.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Reproduce an accepted native server in the separate diagnostic profile harness.""" + +from __future__ import annotations + +import shlex +import shutil +import hashlib +from copy import deepcopy +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import yaml + +from hyperloom.common.env_safety import filter_untrusted_env_mapping, is_allowed_variant_env_key +from hyperloom.inference_optimizer.grid_server_args import merge_server_args, remove_server_args + +from ._native_source import verify_native_source_imports + + +def managed_profile_benchmark(benchmark: Mapping[str, Any]) -> bool: + """Identify a resolved Magpie-owned diagnostic configuration.""" + agentx = benchmark.get("agentx") or {} + return bool( + isinstance(agentx, Mapping) + and ( + (agentx.get("resolved") or {}).get("server-launch-spec") + or (benchmark.get("workload_spec") or {}).get("profile_parent") + ) + and ((benchmark.get("profiler") or {}).get("torch_profiler") or {}).get("enabled") is True + ) + + +def managed_profile_workload_params( + benchmark: Mapping[str, Any], *, config_path: Path, capture: Mapping[str, Any] +) -> dict[str, Any]: + """Bind diagnostic certification to its verified workload and capture settings.""" + envs = benchmark.get("envs") or {} + return { + "source": str(config_path.resolve()), + "num_steps": capture["num_steps"], + "conc": benchmark["workload_spec"]["resolved_topology"]["conc"], + **{field: envs[name] for field, name in (("osl", "OSL"), ("r", "RANDOM_RANGE_RATIO")) if name in envs}, + } + + +def materialize_managed_profile( + config: dict[str, Any], + output_dir: Path, + *, + out_name: str, + snapshot: Mapping[str, Any] | None, + **candidate: Any, +) -> Path: + """Derive a diagnostic without reapplying ambient workload or recipe defaults.""" + from ._native_candidate import install_native_launch_snapshot, update_native_candidate_file + + if snapshot is not None: + install_native_launch_snapshot(config["benchmark"], snapshot) + output_dir.mkdir(parents=True, exist_ok=True) + path = output_dir / out_name + path.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + update_native_candidate_file(path, **candidate) + return path + + +def prepare_managed_profile(params: dict[str, Any], state: Any, output_dir: Path) -> None: + """Profile the accepted native implementation without replaying candidate deltas.""" + from hyperloom.inference_optimizer.agentx.identity import validate_server_launch + from ._native_candidate import apply_native_candidate + + measurement = getattr(state, "current_best_measurement", None) or {} + accepted = str(measurement.get("materialized_config") or getattr(state, "baseline_config_path", "") or "") + if not accepted: + raise ValueError("Native diagnostic profiling requires an accepted benchmark configuration") + config = yaml.safe_load(Path(accepted).read_text(encoding="utf-8")) + benchmark = config["benchmark"] + evidence = measurement.get("agentx_server_launch") or (benchmark.get("workload_spec") or {}).get("server_launch") + errors = validate_server_launch(benchmark, evidence) + if errors: + raise ValueError("Accepted native server launch is not verified: " + "; ".join(errors)) + verify_native_source_imports(benchmark) + workload = benchmark.setdefault("workload_spec", {}) + workload["profile_parent"] = { + "materialized_config": str(Path(accepted).resolve()), + "config_sha256": hashlib.sha256(Path(accepted).read_bytes()).hexdigest(), + "workload_fingerprint": (workload.get("execution") or {}).get("workload_fingerprint"), + } + settings: dict[str, Any] = {"enabled": True, "detailed_annotations": True} + template = params.get("config_path") + if template and Path(template).resolve() != Path(accepted).resolve(): + template_benchmark = (yaml.safe_load(Path(template).read_text(encoding="utf-8")) or {}).get("benchmark") or {} + settings.update((template_benchmark.get("profiler") or {}).get("torch_profiler") or {}) + settings.update((params.get("profiler") or {}).get("torch_profiler") or {}) + settings.update(params.get("torch_profiler") or {}) + for key in ( + "num_steps", + "num_profiles", + "start_seconds", + "interval_seconds", + "capture_timeout_seconds", + "flush_timeout_seconds", + "detailed_annotations", + ): + if key in params: + settings[key] = params[key] + settings["enabled"] = True + benchmark["profiler"] = { + "torch_profiler": settings, + "system_profiler": {"enabled": False}, + "tracelens": {"enabled": False}, + } + apply_native_candidate( + benchmark, + extra_server_args=str(params.get("extra_server_args") or ""), + extra_envs=params.get("extra_envs"), + remove_args=params.get("remove_args"), + unset_envs=params.get("unset_envs"), + args_mode=str(params.get("args_mode") or "append"), + ) + output_dir.mkdir(parents=True, exist_ok=True) + destination = output_dir / "managed_profile.source.yaml" + destination.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + params["config_path"] = str(destination) + params["native_launch_overrides"] = deepcopy(benchmark["agentx"]["launch_overrides"]) + params["baseline_double_run"] = False + params["disable_run_eval"] = True + for key in ("extra_server_args", "extra_envs", "remove_args", "unset_envs", "args_mode"): + params.pop(key, None) + + +def project_native_profile(params: dict[str, Any], state: Any) -> None: + """Use observed server flags/runtime while retaining diagnostic client ownership.""" + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + + measurement = getattr(state, "current_best_measurement", None) or {} + path = str(measurement.get("materialized_config") or getattr(state, "baseline_config_path", "") or "") + if not path: + raise ValueError("Native diagnostic profiling requires an accepted benchmark configuration") + benchmark = (yaml.safe_load(Path(path).read_text(encoding="utf-8")) or {}).get("benchmark") or {} + evidence = measurement.get("agentx_server_launch") + if not isinstance(evidence, Mapping): + evidence = (benchmark.get("workload_spec") or {}).get("server_launch") + if not isinstance(evidence, Mapping): + raise ValueError("Native diagnostic profiling requires verified server launch evidence") + if evidence.get("evidence_sha256") != canonical_sha256( + {k: v for k, v in evidence.items() if k != "evidence_sha256"} + ): + raise ValueError("Native diagnostic profiling server launch evidence changed") + argv = evidence.get("effective_argv") or [] + framework = str(benchmark.get("framework") or "").lower() + if framework == "sglang" and argv[1:3] == ["-m", "sglang.launch_server"]: + flags = argv[3:] + elif framework == "vllm" and argv[1:2] == ["serve"] and len(argv) >= 3: + flags = argv[3:] + else: + raise ValueError("Native diagnostic profiling cannot reproduce this server entrypoint") + projected = remove_server_args( + shlex.join(flags), ["--model", "--model-path", "--served-model-name", "--host", "--port"] + ) + params["extra_server_args"] = merge_server_args(projected, str(params.get("extra_server_args") or "")) + runtime = {str(k): str(v) for k, v in (evidence.get("runtime_environment") or {}).items()} + safe_env, _ = filter_untrusted_env_mapping(runtime, allow_predicate=is_allowed_variant_env_key) + params["extra_envs"] = {**safe_env, **dict(params.get("extra_envs") or {})} + override = dict(params.get("runtime_override") or {}) + for env_key, field in (("PYTHONPATH", "pythonpath_prefixes"), ("LD_LIBRARY_PATH", "ld_library_path_prefix")): + if runtime.get(env_key): + override[field] = [entry for entry in runtime[env_key].split(":") if entry] + executable = str(evidence.get("resolved_executable") or argv[0]) + executable = shutil.which(executable, path=runtime.get("PATH")) or executable + if not Path(executable).is_absolute(): + raise ValueError("Native diagnostic profiling cannot resolve the accepted server executable") + if framework == "sglang": + override["runtime_python_exe"] = executable + else: + override["framework_bin"] = executable + override["framework_python"] = str(Path(executable).parent / "python") + params["runtime_override"] = override + verify_native_source_imports(benchmark) diff --git a/src/hyperloom/orchestrator/actions/executors/_native_source.py b/src/hyperloom/orchestrator/actions/executors/_native_source.py new file mode 100644 index 0000000000..8e29722fe3 --- /dev/null +++ b/src/hyperloom/orchestrator/actions/executors/_native_source.py @@ -0,0 +1,208 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Attest the deployed files and import roots of native optimizer candidates.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import subprocess +import tempfile +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + + +def source_file_hashes(paths: Sequence[str | Path]) -> dict[str, str]: + """Bind applied source bytes, refusing missing targets instead of omitting them.""" + files: dict[str, str] = {} + for raw in paths: + path = Path(raw).expanduser().absolute() + if not path.is_file(): + raise ValueError(f"Native candidate source target is not a file: {path}") + files[str(path)] = hashlib.sha256(path.read_bytes()).hexdigest() + return dict(sorted(files.items())) + + +def applied_source_evidence( + framework_root: Path | None, + patches: Sequence[Path], + artifacts: Sequence[Mapping[str, Any]] = (), +) -> dict[str, Any]: + """Describe the realized files of an already-applied source transaction.""" + from ._patch_snapshot import patch_declared_ops + + operations: dict[str, str] = {} + if patches and framework_root is None: + raise ValueError("Native source transaction requires its applied framework root") + if framework_root is not None: + for relative, operation in patch_declared_ops(framework_root, list(patches)).items(): + operations[str((framework_root / relative).absolute())] = operation + for artifact in artifacts: + target = Path(str(artifact["root"])) / str(artifact["rel_target"]) + operations[str(target.absolute())] = "upsert" + if (patches or artifacts) and not operations: + raise ValueError("Native source transaction did not identify any applied files") + return _attest_source_operations(operations) + + +def _attest_source_operations(operations: Mapping[str, str]) -> dict[str, Any]: + absent = sorted(path for path, operation in operations.items() if operation == "delete") + if any(os.path.lexists(path) for path in absent): + raise ValueError("Native source deletion was not applied") + return { + "source_files": source_file_hashes([path for path in operations if path not in absent]), + "absent_source_files": absent, + } + + +def _kernel_source_operations(applied: Mapping[str, Any]) -> dict[str, str]: + manifest = json.loads(Path(str(applied["manifest_path"])).read_text(encoding="utf-8")) + backups = manifest.get("source_backups") or [] + if backups: + return { + str(Path(row["path"]).absolute()): "delete" if row.get("disposition") == "deleted" else "upsert" + for row in backups + } + target = applied.get("target_file") or manifest.get("target_file") + if not target: + raise ValueError("Native kernel transaction did not record its deployed source paths") + return {str(Path(target).absolute()): "upsert"} + + +def warm_source_evidence(params: Mapping[str, Any], output_dir: Path) -> dict[str, Any]: + """Bind the successful patches retained by baseline's warm replay transaction.""" + from ._patch_snapshot import patch_declared_ops + + operations: dict[str, str] = {} + for applied in params.get("warm_kernel_apply_results") or []: + operations.update(_kernel_source_operations(applied)) + statuses = params.get("_warm_patch_statuses") or [] + if len(statuses) != len(params.get("patches") or []): + raise ValueError("Native warm replay has incomplete source patch application evidence") + for index, status in enumerate(statuses): + if status.get("status") not in { + "applied", + "applied_3way", + "applied_nogit", + "already_present", + "present_in_dirty_worktree", + }: + raise ValueError("Native warm replay cannot silently omit a requested source patch") + patch = output_dir / "warm_patches" / f"{index:03d}_{Path(status.get('patch_ref') or '').stem or 'patch'}.diff" + root = Path(status["target_repo"]) + operations.update({str((root / path).absolute()): op for path, op in patch_declared_ops(root, [patch]).items()}) + return _attest_source_operations(operations) + + +def kernel_source_evidence(applied: Mapping[str, Any]) -> dict[str, Any]: + """Bind every write/deletion recorded by the kernel deployment transaction.""" + return _attest_source_operations(_kernel_source_operations(applied)) + + +_SOURCE_PACKAGES = ("sglang", "vllm", "aiter", "sgl_kernel", "flydsl", "flashinfer", "torch") +_IMPORT_PROBE = """import importlib.util, json, sys +result = {} +for name in json.loads(sys.argv[1]): + spec = importlib.util.find_spec(name) + result[name] = { + 'origin': None if spec is None else spec.origin, + 'locations': [] if spec is None else list(spec.submodule_search_locations or []), + } + if name == 'sitecustomize': + result[name]['loaded'] = sys.modules.get(name) is not None +print(json.dumps(result)) +""" + + +def _source_import_anchor(path: Path, overlay_roots: Sequence[Path]) -> tuple[str, Path]: + for root in overlay_roots: + if path.is_relative_to(root): + return "sitecustomize", root / "sitecustomize.py" + for parent in path.parents: + if parent.name in _SOURCE_PACKAGES and (parent / "__init__.py").is_file(): + return parent.name, parent + for parent in path.parents: + if not any((parent / marker).is_file() for marker in ("pyproject.toml", "setup.py", "setup.cfg")): + continue + for name in _SOURCE_PACKAGES: + for package in (parent / name, parent / "python" / name): + if (package / "__init__.py").is_file(): + return name, package + raise ValueError(f"Native source file has no verifiable framework/package import root: {path}") + + +def _server_python(benchmark: Mapping[str, Any], overrides: Mapping[str, Any], env: Mapping[str, str]) -> str: + evidence = (benchmark.get("workload_spec") or {}).get("server_launch") or {} + argv = evidence.get("base_argv") or [] + executable = str(overrides.get("executable") or (argv[0] if argv else "")) + resolved = shutil.which(executable, path=env.get("PATH")) if executable else None + if not resolved: + raise ValueError("Native source validation requires the verified server executable or a runtime override") + path = Path(resolved) + if str(benchmark.get("framework") or "").lower() == "vllm": + # The vLLM console entrypoint belongs to the same environment as its Python. + python = path.parent / "python" + if not python.is_file(): + raise ValueError(f"Native vLLM entrypoint has no sibling Python: {path}") + return str(python) + return str(path) + + +def verify_native_source_imports(benchmark: dict[str, Any]) -> None: + """Resolve candidate package origins with the actual server interpreter and env.""" + overrides = (benchmark.get("agentx") or {}).get("launch_overrides") or {} + targets = list(overrides.get("source_files") or {}) + list(overrides.get("absent_source_files") or []) + if not targets: + return + for raw, digest in (overrides.get("source_files") or {}).items(): + if source_file_hashes([raw]).get(str(Path(raw).absolute())) != digest: + raise ValueError(f"Native source bytes changed after candidate application: {raw}") + if any(os.path.lexists(path) for path in overrides.get("absent_source_files") or []): + raise ValueError("Native deleted source reappeared before launch") + env = {**os.environ, **{str(k): str(v) for k, v in (benchmark.get("envs") or {}).items()}} + launch = (benchmark.get("workload_spec") or {}).get("server_launch") or {} + env.update({str(k): str(v) for k, v in (launch.get("runtime_environment") or {}).items()}) + env.update(overrides.get("env") or {}) + for name in overrides.get("unset_env") or []: + env.pop(name, None) + overlay_roots = [ + Path(entry).absolute() + for entry in env.get("PYTHONPATH", "").split(os.pathsep) + if entry and (Path(entry) / "sitecustomize.py").is_file() + ] + expected: dict[str, Path] = {} + for raw in targets: + module, anchor = _source_import_anchor(Path(raw).absolute(), overlay_roots) + if module in expected and expected[module] != anchor: + raise ValueError(f"Native candidate has conflicting import roots for {module}") + expected[module] = anchor + proc = subprocess.run( + [_server_python(benchmark, overrides, env), "-c", _IMPORT_PROBE, json.dumps(sorted(expected))], + env=env, + cwd=tempfile.gettempdir(), + capture_output=True, + text=True, + timeout=30, + check=True, + ) + lines = proc.stdout.strip().splitlines() + if not lines: + raise ValueError("Native source import probe returned no evidence") + observed = json.loads(lines[-1]) + for module, anchor in expected.items(): + record = observed.get(module) or {} + if module == "sitecustomize" and record.get("loaded") is not True: + raise ValueError(f"Native candidate overlay did not load successfully: {anchor}") + origin = record.get("origin") + locations = record.get("locations") or [] + matches = ( + bool(origin) + and (Path(origin).resolve() == anchor.resolve() or Path(origin).resolve().is_relative_to(anchor.resolve())) + ) or any(Path(location).resolve() == anchor.resolve() for location in locations) + if not matches: + raise ValueError(f"Native candidate source is not imported from {anchor}: {module} resolved to {record}") + benchmark.setdefault("workload_spec", {})["source_imports"] = observed diff --git a/src/hyperloom/orchestrator/actions/executors/_ray_backend.py b/src/hyperloom/orchestrator/actions/executors/_ray_backend.py index 1b5da65a7a..298b7c9197 100644 --- a/src/hyperloom/orchestrator/actions/executors/_ray_backend.py +++ b/src/hyperloom/orchestrator/actions/executors/_ray_backend.py @@ -25,8 +25,17 @@ def _should_use_ray_backend() -> bool: """Whether GPU/serving work runs through the Ray backend. - ``INFERENCE_OPTIMIZER_RAY_EXEC`` wins when set; otherwise single-node is ON, multi-node OFF, and pytest OFF. + Native AgentX stays local. Otherwise ``INFERENCE_OPTIMIZER_RAY_EXEC`` wins + when set; single-node defaults ON, multi-node and pytest default OFF. """ + # Magpie AgentX v1 is local-only. More importantly, its idle-GPU selector + # reads the materialized YAML rather than Ray's ambient actor mask, so + # nesting it in a Ray lease can escape the allocation. Keep the entire + # native session on the direct local backend. + from hyperloom.common.agentx_mode import native_agentx_session + + if native_agentx_session(): + return False from ._multi_node_env import is_multi_node auto = not os.environ.get("PYTEST_CURRENT_TEST") and not is_multi_node() diff --git a/src/hyperloom/orchestrator/actions/executors/_recipe_script.py b/src/hyperloom/orchestrator/actions/executors/_recipe_script.py index b82e4bcc10..ede4ba4c75 100644 --- a/src/hyperloom/orchestrator/actions/executors/_recipe_script.py +++ b/src/hyperloom/orchestrator/actions/executors/_recipe_script.py @@ -37,8 +37,19 @@ def resolve_launch_server_script(bench: Mapping[str, Any]) -> str: if not script: return "" - if is_agentx_client_script(script): - script = str(envs.get("AGENTX_SERVER_SCRIPT") or os.environ.get("AGENTX_SERVER_SCRIPT") or "").strip() + native_agentx = False + if "agentx" in bench: + from hyperloom.inference_optimizer.agentx.native import native_agentx_enabled + + native_agentx = native_agentx_enabled(bench["agentx"]) + if is_agentx_client_script(script) or native_agentx: + # Native launchers own the full replay lifecycle; only the framework's + # generic proxy implements GEAK's server-only phase. + script = ( + "" + if native_agentx + else str(envs.get("AGENTX_SERVER_SCRIPT") or os.environ.get("AGENTX_SERVER_SCRIPT") or "").strip() + ) if not script: framework = str(bench.get("framework") or envs.get("FRAMEWORK") or "").strip().lower() if not framework: diff --git a/src/hyperloom/orchestrator/actions/executors/_workload_envs.py b/src/hyperloom/orchestrator/actions/executors/_workload_envs.py index c261ae6535..9665c73ad1 100644 --- a/src/hyperloom/orchestrator/actions/executors/_workload_envs.py +++ b/src/hyperloom/orchestrator/actions/executors/_workload_envs.py @@ -33,7 +33,7 @@ import yaml -from hyperloom.common.coerce import to_str_list +from hyperloom.common.coerce import to_float, to_str_list from hyperloom.common.env import env_bool, env_flag, is_truthy from hyperloom.common.gpu_identity import AMD_GPU_DISPATCH_IDENTITIES from hyperloom.common.perf_metric import ( @@ -253,6 +253,20 @@ def agentx_env_for_conc(conc: int | None = None) -> "Mapping[str, str]": AGENTX_FULL_CONTEXT_FAMILIES = ("dsv4", "deepseekv4", "glm52", "minimaxm3", "kimik3") AGENTX_CORPUS_FULL = "semianalysis_cc_traces_weka_062126" AGENTX_CORPUS_256K = "semianalysis_cc_traces_weka_062126_256k" +_AGENTX_IDENTITY_ENVS = frozenset( + { + "AGENTX_MODEL_ID", + "AGENTX_SERVER_SCRIPT", + "AGENTX_RECIPE", + "AGENTX_CONFIG_FILE", + "AGENTX_SELECTOR", + "AGENTX_MODE", + "AGENTX_FAILED_REQUEST_THRESHOLD", + "AGENTX_DATASET", + "AGENTX_WARMUP_REQUESTS_PER_LANE", + "WEKA_LOADER_OVERRIDE", + } +) def _agentx_model_family(model: str) -> str: @@ -466,6 +480,165 @@ def served_knob(key: str, default: Any) -> str: } +def _config_enabled(value: Any) -> bool: + """Interpret one serialized Magpie ``enabled`` value.""" + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "on", "enable", "enabled"} + return bool(value) + + +def _serialized_native_agentx_enabled(value: Any) -> bool: + """Read ``benchmark.agentx`` without importing the optional AgentX package.""" + if isinstance(value, Mapping): + return _config_enabled(value.get("enabled", True)) + return _config_enabled(value) + + +def _agentx_profile_requested(bench: Mapping[str, Any]) -> bool: + """Return whether Magpie v1 would reject native AgentX profiling. + + Native AgentX rejects all four profiler/analysis integrations, not just + torch traces. Route every such leg through Hyperloom's phase-gated legacy + client so a custom YAML cannot survive materialization only to fail during + Magpie config validation. + """ + raw_envs = bench.get("envs") + envs = raw_envs if isinstance(raw_envs, Mapping) else {} + if str(envs.get("PROFILE") or "").strip().lower() in {"1", "true", "yes", "on"}: + return True + raw_profiler = bench.get("profiler") + profiler = raw_profiler if isinstance(raw_profiler, Mapping) else {} + for section_name in ("torch_profiler", "system_profiler", "tracelens"): + raw_section = profiler.get(section_name) + section = raw_section if isinstance(raw_section, Mapping) else {} + if _config_enabled(section.get("enabled", False)): + return True + raw_gap = bench.get("gap_analysis") + gap = raw_gap if isinstance(raw_gap, Mapping) else {} + return _config_enabled(gap.get("enabled", False)) + + +def _agentx_setting(envs: Mapping[str, Any], name: str) -> str: + """Resolve one AgentX setting from persisted YAML, then the process.""" + return str(envs.get(name) or os.environ.get(name) or "").strip() + + +def _native_agentx_config(envs: Mapping[str, Any], existing: Any = None) -> str | dict[str, Any]: + """Build Magpie's native AgentX value without erasing YAML-native pins.""" + existing_config = dict(existing) if isinstance(existing, Mapping) else {} + # Hyperloom changes CONC per baseline/grid/sweep round. Magpie gives an + # explicit agentx.concurrency precedence over benchmark.envs.CONC, so + # carrying a source-YAML value here would make every materialized rung run + # the same load while being labelled with a different concurrency. + existing_config.pop("concurrency", None) + # ``resolved`` describes the previous point. Magpie will recompute it for + # this round after Hyperloom settles CONC; carrying it forward makes a + # topology-changing input look pre-resolved when it is not. + previous_resolved = existing_config.pop("resolved", None) + if not existing_config.get("selector") and isinstance(previous_resolved, Mapping): + # A previously unique point can become ambiguous when only CONC + # changes (for example GLM's TP4/HiCache and TP8/no-offload arms both + # contain CONC=2). Preserve the accepted arm, not its old concurrency, + # as an explicit Magpie selector before asking it to resolve the rung. + stable_selector_keys = ( + "tp", + "pp", + "pcp-size", + "ep", + "dcp-size", + "dp-attn", + "spec-decoding", + "kv-offloading", + "kv-offload-backend", + ) + derived_selector = {key: previous_resolved[key] for key in stable_selector_keys if key in previous_resolved} + if derived_selector: + existing_config["selector"] = derived_selector + mode = _agentx_setting(envs, "AGENTX_MODE").lower() or str(existing_config.get("mode") or "canonical").lower() + if mode not in {"canonical", "fast"}: + raise ValueError("AGENTX_MODE must be 'canonical' or 'fast'") + recipe = _agentx_setting(envs, "AGENTX_RECIPE") or str(existing_config.get("recipe") or "").strip() + config_file = _agentx_setting(envs, "AGENTX_CONFIG_FILE") or str(existing_config.get("config_file") or "").strip() + selector_raw = _agentx_setting(envs, "AGENTX_SELECTOR") + threshold_raw = _agentx_setting(envs, "AGENTX_FAILED_REQUEST_THRESHOLD") + + existing_selector = existing_config.get("selector") + selector: dict[str, Any] = dict(existing_selector) if isinstance(existing_selector, Mapping) else {} + if selector_raw: + try: + parsed = json.loads(selector_raw) + except json.JSONDecodeError as exc: + raise ValueError(f"AGENTX_SELECTOR must be a JSON object: {exc}") from exc + if not isinstance(parsed, dict): + raise ValueError("AGENTX_SELECTOR must be a JSON object") + selector = parsed + + if ( + not existing_config + and mode == "canonical" + and not recipe + and not config_file + and not selector + and not threshold_raw + ): + # This is the public, zero-boilerplate Magpie switch the integration is + # designed around. Expand to an object only for an explicit override. + return "enable" + + config: dict[str, Any] = {**existing_config, "enabled": True, "mode": mode} + if recipe: + config["recipe"] = recipe + if config_file: + config["config_file"] = config_file + if selector: + config["selector"] = selector + if threshold_raw: + try: + threshold = float(threshold_raw) + except ValueError as exc: + raise ValueError("AGENTX_FAILED_REQUEST_THRESHOLD must be a number between 0 and 1") from exc + if not 0.0 <= threshold <= 1.0: + raise ValueError("AGENTX_FAILED_REQUEST_THRESHOLD must be between 0 and 1") + config["failed_request_threshold"] = threshold + elif "failed_request_threshold" in config: + configured_threshold = to_float(config["failed_request_threshold"]) + if configured_threshold is None or not 0.0 <= configured_threshold <= 1.0: + raise ValueError("benchmark.agentx.failed_request_threshold must be between 0 and 1") + config["failed_request_threshold"] = configured_threshold + return config + + +def _native_agentx_local_model_path(model_ref: str) -> str: + """Return a local checkpoint override, excluding Hugging Face repo IDs. + + InferenceX launchers interpret every non-empty ``MODEL_PATH`` as a local + download directory. Forwarding ``amd/GLM-5.2-MXFP4`` there would therefore + create a relative ``amd/...`` tree under the checkout instead of using the + normal Hugging Face cache. Relative local paths remain available with an + explicit ``./`` prefix; absolute and home-relative paths are unambiguous. + """ + candidate = str(model_ref or "").strip() + if candidate.count("/") == 1 and not candidate.startswith(("/", ".", "~")): + return "" + return str(Path(candidate).expanduser().resolve()) if candidate else "" + + +def _canonical_agentx_model(envs: Mapping[str, Any], model_path: str) -> str: + """Resolve the canonical InferenceX model id separately from local weights.""" + explicit = _agentx_setting(envs, "AGENTX_MODEL_ID") + if explicit: + return explicit + candidate = str(model_path or "").strip() + # A normal Hugging Face identity is safe to reuse when --model itself is + # remote. Absolute/relative filesystem paths are never guessed into an id. + if candidate.count("/") == 1 and not candidate.startswith(("/", ".", "~")): + return candidate + raise ValueError( + "Native AgentX requires AGENTX_MODEL_ID=; " + "keep the local checkpoint in --model (it is forwarded as MODEL_PATH)" + ) + + def pin_mlperf_round_concurrency(envs: dict[str, Any]) -> None: """Make ``AGENTIC_CONCURRENCY`` follow this round's ``CONC``. @@ -488,9 +661,11 @@ def apply_agentx_switch( *, conc: Any = None, active: bool | None = None, + native_agentx_mode: bool | None = None, grading: Mapping[str, Any] | None = None, + allow_profile_compat: bool = False, ) -> None: - """Switch serving-framework benchmarks to the AgentX aiperf client. + """Preserve the native or legacy AgentX harness selected by the session or YAML. ``conc`` is the concurrency this round will run at; the inner benchmark cap, the client's warmup grace and the published ``workload_spec.concurrency`` are @@ -501,8 +676,13 @@ def apply_agentx_switch( GEAK; callers that cannot reach the live state leave it ``None`` and the spec derives both from the environment as before. """ + workload = bench.get("workload_spec") + native_profile = isinstance(workload, Mapping) and workload.get("harness") == "hyperloom-profiler-compat" + native_selected = ( + bool(native_agentx_mode) or _serialized_native_agentx_enabled(bench.get("agentx")) or native_profile + ) if active is None: - active = agentx_enabled() + active = native_selected or agentx_enabled() if not active: return from hyperloom.inference_optimizer import framework_registry @@ -521,20 +701,67 @@ def apply_agentx_switch( mlperf_trajectories, ) + local_model = str( + model_path + or envs.get("MODEL_PATH") + or envs.get("MODEL") + or bench.get("model") + or os.environ.get("MODEL_PATH", "") + ).strip() from ._agentx_timeouts import agentx_warmup_grace_sec - _agentx_env = agentx_env_for_conc(conc) - bench["benchmark_script"] = agentx_client_script(_agentx_env) + resolved_conc: int | None = None + if conc not in (None, ""): + if isinstance(conc, bool): + raise ValueError("AgentX concurrency must be a positive integer") + try: + resolved_conc = int(conc) + except (TypeError, ValueError) as exc: + raise ValueError("AgentX concurrency must be a positive integer") from exc + if resolved_conc <= 0: + raise ValueError("AgentX concurrency must be a positive integer") + # This is the value the native launcher consumes. Publishing only the + # workload_spec value would make the result metadata disagree with the + # actual replay. + envs["CONC"] = resolved_conc + _agentx_env = agentx_env_for_conc(resolved_conc) envs["RUN_EVAL"] = "false" - envs["MODEL"] = str(model_path or bench.get("model") or os.environ.get("MODEL_PATH", "")).strip() envs["FRAMEWORK"] = framework - # WEKA_LOADER_OVERRIDE is upstream's own per-recipe corpus pin, so it has no - # AGENTX_ prefix and would not survive the loop below. aiperf_client.sh - # documents it as a supported knob; without forwarding it only works when - # the benchmark process happens to inherit the full parent environment, - # which is exactly the kind of silent difference this path exists to remove. + from ._native_profile import managed_profile_benchmark + + profile_requested = native_selected and _agentx_profile_requested(bench) + profile_compat = profile_requested and not managed_profile_benchmark(bench) + if profile_requested and not allow_profile_compat: + raise ValueError( + "Native AgentX profiler/gap-analysis settings are diagnostic-only " + "and may be materialized only by ProfileExecutor; they cannot " + "become a baseline, grid, or optimization measurement." + ) + forbidden_native_corpus_envs = ( + "AGENTX_DATASET", + "AGENTX_WARMUP_REQUESTS_PER_LANE", + "WEKA_LOADER_OVERRIDE", + ) + if native_selected and not profile_compat: + overridden = [ + name for name in forbidden_native_corpus_envs if str(envs.get(name) or os.environ.get(name) or "").strip() + ] + if overridden: + raise ValueError( + "Native AgentX corpus selection belongs to the resolved " + "InferenceX recipe; remove " + ", ".join(overridden) + ) + + # The legacy and profiler clients consume the AGENTX_* controls. + # Native Magpie must not inherit them: duration, entry count, + # dataset, or a foreign AIPerf binary would change the replay without + # changing InferenceX's recipe fingerprint. for key, value in os.environ.items(): - if key.startswith("AGENTX_") or key in ("AIPERF_BIN", "WEKA_LOADER_OVERRIDE"): + if (not native_selected or profile_compat) and ( + key.startswith("AGENTX_") or key in ("AIPERF_BIN", "WEKA_LOADER_OVERRIDE") + ): + if native_selected and key in _AGENTX_IDENTITY_ENVS and str(envs.get(key) or "").strip(): + continue envs[key] = value if is_mlperf_backend(_agentx_env) and (key.startswith(("MLPERF_", "AGENTIC_")) or key == BACKEND_ENV): envs[key] = value @@ -566,7 +793,7 @@ def apply_agentx_switch( _grace = agentx_warmup_grace_sec(_agentx_env) _raw_grace = (os.environ.get("AGENTX_WARMUP_GRACE_PERIOD") or "").strip() envs["AGENTX_WARMUP_GRACE_PERIOD"] = str(_grace) - if bench.get("timeout_seconds") is not None: + if (not native_selected or profile_compat) and bench.get("timeout_seconds") is not None: envs["AGENTX_PHASE_WAIT_TIMEOUT_S"] = str(bench["timeout_seconds"]) if _raw_grace != str(_grace): log.info( @@ -574,6 +801,69 @@ def apply_agentx_switch( _grace, _raw_grace or "unset", ) + + canonical_model = _canonical_agentx_model(envs, local_model) if native_selected else local_model + if not native_selected: + bench["benchmark_script"] = agentx_client_script(_agentx_env) + envs["MODEL"] = local_model + native = False + elif profile_compat: + # Magpie AgentX v1 deliberately rejects torch/system profiling. Keep + # Hyperloom's phase-gated client for this one leg, and clear the native + # launcher pin so the client cannot recursively invoke a full AgentX + # replay as though it were a server-only script. + bench.pop("agentx", None) + bench["benchmark_script"] = "aiperf_client.sh" + bench["model"] = local_model + envs["MODEL"] = local_model + # The compatibility server still needs the local checkpoint path, but + # corpus selection must follow the InferenceX recipe identity. A local + # directory name is arbitrary and cannot identify the 1M/256k corpus. + envs["AGENTX_MODEL_ID"] = canonical_model + envs["AGENTX_SERVER_SCRIPT"] = "" + native = False + else: + from hyperloom.inference_optimizer.agentx.native import validate_native_launcher_name + + script = _agentx_setting(envs, "AGENTX_SERVER_SCRIPT") + validate_native_launcher_name(script) + agentx_config = _native_agentx_config(envs, bench.get("agentx")) + bench["agentx"] = agentx_config + bench["benchmark_script"] = script + bench["model"] = canonical_model + # The generic Magpie path maps MI325X/MI308X to the MI300X launcher + # family. Native AgentX recipes validate the physical hardware encoded + # in their model-specific launcher, so retain the probed target identity + # instead of that compatibility alias. + physical_runner = str(os.environ.get("TARGET_GPU_TYPE") or "").strip().lower() + if physical_runner: + bench["runner_type"] = physical_runner + raw_gpu_selection = bench.get("gpu_selection") + gpu_selection = dict(raw_gpu_selection) if isinstance(raw_gpu_selection, Mapping) else {} + # The bundled Hyperloom baselines disable Magpie GPU selection because + # the generic path derives a mask from CLI TP. AgentX resolves its + # physical rank count only after loading the InferenceX recipe, so let + # Magpie select that count unless an explicit visible-device mask (or + # candidates list) constrains it. A stale generic ``count`` must not + # override the resolved recipe topology. + gpu_selection["auto"] = True + gpu_selection.pop("count", None) + bench["gpu_selection"] = gpu_selection + envs.pop("MODEL", None) + envs.pop("MODEL_PATH", None) + local_model_path = _native_agentx_local_model_path(local_model) + if local_model_path: + envs["MODEL_PATH"] = local_model_path + # Persist both identity pins so a resumed/grid rebuild remains native + # even if its subprocess did not inherit the operator's shell. + envs["AGENTX_MODEL_ID"] = canonical_model + envs["AGENTX_SERVER_SCRIPT"] = script + # The launcher owns native replay shape. Hyperloom only supplies a + # deterministic grace bound; legacy corpus/warmup controls are rejected + # above because they are not part of the InferenceX recipe fingerprint. + envs["AGENTIC_WARMUP_GRACE_PERIOD"] = str(_grace) + native = True + # Publish LAST, and only now: the spec is what GEAK replays, so every value # in it must be the one this function settled on. ``warmup_grace_period_s`` # reads AGENTX_WARMUP_GRACE_PERIOD back off ``envs`` above, so publishing @@ -581,13 +871,26 @@ def apply_agentx_switch( # while the client ran with the scaled one. ``_agentx_env`` carries this # round's CONC, so the spec's concurrency is the served concurrency by # construction rather than by later repair. - bench["workload_spec"] = build_agentx_workload_spec( + spec = build_agentx_workload_spec( bench, envs, - model_path=model_path, + model_path=canonical_model, env=_agentx_env, grading=grading, ) + if native_selected: + spec["harness"] = "magpie-native-agentx" if native else "hyperloom-profiler-compat" + if native: + mode = str(bench["agentx"].get("mode") or "canonical") if isinstance(bench["agentx"], dict) else "canonical" + # InferenceX owns these values on the native path. Do not advertise + # legacy AGENTX_DURATION/NUM_ENTRIES knobs that it does not consume. + spec["duration_s"] = 1200 if mode == "fast" else 3600 + spec["geak_loop_duration_s"] = min(int(spec["duration_s"]), 900) + spec["num_entries"] = 393 + spec["warmup_requests_per_lane"] = 1 if mode == "fast" else 10 + if isinstance(workload, Mapping) and "profile_parent" in workload: + spec["profile_parent"] = workload["profile_parent"] + bench["workload_spec"] = spec def prepare_agentx_runtime( @@ -597,9 +900,10 @@ def prepare_agentx_runtime( config_path: Path | str | None = None, output_dir: Path | str | None = None, active: bool | None = None, + allow_profile_compat: bool = False, ) -> str | None: """Deploy and preflight AgentX assets for baseline/profile runs.""" - runtime_env = env or os.environ + runtime_env = env if env is not None else os.environ if active is None: active = agentx_enabled(runtime_env) if not active and config_path: @@ -607,9 +911,11 @@ def prepare_agentx_runtime( materialized = yaml.safe_load(Path(config_path).read_text(encoding="utf-8")) or {} benchmark = materialized.get("benchmark") if isinstance(materialized, dict) else {} from hyperloom.common.agentx_workload import is_agentx_client_script + from hyperloom.common.agentx_mode import native_agentx_enabled - active = isinstance(benchmark, dict) and is_agentx_client_script( - str(benchmark.get("benchmark_script") or "") + active = isinstance(benchmark, dict) and ( + is_agentx_client_script(str(benchmark.get("benchmark_script") or "")) + or native_agentx_enabled(benchmark.get("agentx")) ) except (OSError, ValueError, TypeError): active = False @@ -625,6 +931,7 @@ def prepare_agentx_runtime( env=runtime_env, inferencex_path=str(inferencex_path or ""), config_path=Path(config_path) if config_path else "", + allow_profile_compat=allow_profile_compat, ) except AgentXPreflightError as exc: return f"AgentX preflight failed: {exc}" @@ -1103,6 +1410,9 @@ def default_baseline_config() -> Path: Returns: Path: The shipped Magpie YAML config path for the resolved framework. """ + explicit = os.environ.get("HYPERLOOM_BENCHMARK_CONFIG", "").strip() + if explicit: + return Path(explicit).expanduser().resolve() fw = os.environ.get("FRAMEWORK") or framework_registry.DEFAULT_FRAMEWORK return asset_root() / "assets" / "configs" / framework_registry.shipped_config_name("baseline", fw) @@ -1351,10 +1661,13 @@ def materialize_config_with_envs( benchmark_script: str | None = None, out_name: str = "baseline_config.with_envs.yaml", establish_quality_ref: bool = False, + native_launch_overrides: Mapping[str, Any] | None = None, drop_moe_runner_backend: bool = False, flydsl_source_dirs: bool = False, agentx_mode: bool | None = None, + native_agentx_mode: bool | None = None, grading: Mapping[str, Any] | None = None, + allow_agentx_profile_compat: bool = False, ) -> Path: """Render a per-run Magpie YAML with caller-provided overrides. @@ -1404,9 +1717,14 @@ def materialize_config_with_envs( cache key. Off by default: only a run that applied such a patch needs it. agentx_mode: Explicit session-level AgentX decision. ``None`` preserves the legacy environment-based fallback. + native_agentx_mode: Native session identity retained when a diagnostic + profile uses a generic template without ``benchmark.agentx``. + ``None`` derives identity from the source YAML. grading: The session's ``SharedState.grading``, which settles the axis and noise band the AgentX ``workload_spec`` publishes to GEAK. ``None`` preserves the environment-derived fallback. + allow_agentx_profile_compat: Permit the non-native, diagnostic AgentX + profile shim. Only ``ProfileExecutor`` may set this. Returns: The materialized YAML path (stable file name across calls). @@ -1423,6 +1741,31 @@ def materialize_config_with_envs( with config_path.open(encoding="utf-8") as f: cfg = yaml.safe_load(f) or {} bench = cfg.setdefault("benchmark", {}) + source_native_agentx = _serialized_native_agentx_enabled(bench.get("agentx")) + if source_native_agentx and agentx_mode is False: + raise ValueError( + "The benchmark YAML enables agentx, but the Hyperloom session is " + "synthetic. Set HYPERLOOM_AGENTX=1 so grading, epochs, budgets, and " + "runtime preparation use the AgentX contract." + ) + from ._native_profile import managed_profile_benchmark, materialize_managed_profile + + if managed_profile_benchmark(bench): + if not allow_agentx_profile_compat: + raise ValueError("Managed AgentX diagnostic profiling may only be materialized by ProfileExecutor") + return materialize_managed_profile( + cfg, + output_dir, + out_name=out_name, + snapshot=native_launch_overrides, + extra_server_args=extra_server_args, + extra_envs=extra_envs, + remove_args=remove_args, + unset_envs=unset_envs, + args_mode=args_mode, + ) + source_agentx_model = str(bench.get("model") or "").strip() if source_native_agentx else "" + source_agentx_script = str(bench.get("benchmark_script") or "").strip() if source_native_agentx else "" if model_path: bench["model"] = str(model_path) precision = os.environ.get("PRECISION", "").strip() @@ -1438,6 +1781,10 @@ def materialize_config_with_envs( if benchmark_script: bench["benchmark_script"] = str(benchmark_script) envs = bench.setdefault("envs", {}) + if source_agentx_model: + envs.setdefault("AGENTX_MODEL_ID", source_agentx_model) + if source_agentx_script: + envs.setdefault("AGENTX_SERVER_SCRIPT", source_agentx_script) runtime_path = os.environ.get("PATH", "").strip() if "PATH" in envs and runtime_path: envs["PATH"] = runtime_path @@ -1447,7 +1794,22 @@ def materialize_config_with_envs( gpu_type=gpu_type, explicit_benchmark_script=bool(benchmark_script), ) - apply_agentx_switch(bench, model_path, active=agentx_mode, grading=grading) + apply_agentx_switch( + bench, + model_path, + active=True if source_native_agentx else agentx_mode, + native_agentx_mode=native_agentx_mode, + grading=grading, + allow_profile_compat=allow_agentx_profile_compat, + ) + native_agentx = False + if "agentx" in bench: + native_agentx = _serialized_native_agentx_enabled(bench.get("agentx")) + agentx_workload = ( + isinstance(bench.get("workload_spec"), Mapping) + and str(bench["workload_spec"].get("kind") or "").strip() == "agentx_trace_replay" + ) + native_profile_compat = agentx_workload and bench["workload_spec"].get("harness") == "hyperloom-profiler-compat" # Fail fast on framework/script mismatch (e.g. vllm image + sglang script). # Only trip when the script carries a DIFFERENT known framework's prefix, so # custom/non-prefixed scripts are not falsely rejected. @@ -1464,8 +1826,31 @@ def materialize_config_with_envs( f"benchmark_script={_script!r} targets {_other[0]!r}; refusing " f"to boot server (would launch the wrong framework's entrypoint)" ) - effective_inferencex_path = str(inferencex_path or "").strip() or os.environ.get("INFERENCEX_PATH", "").strip() + explicit_inferencex_path = str(inferencex_path or "").strip() + configured_inferencex_path = str(bench.get("inferencex_path") or "").strip() + runtime_inferencex_path = os.environ.get("INFERENCEX_PATH", "").strip() + if native_agentx: + # Preflight may replace an unwritable or wrong-revision source checkout + # with the verified pinned checkout. That post-preflight runtime path + # is authoritative for native AgentX; letting the original YAML win + # here would launch and fingerprint a different tree than preflight + # validated. A task-scoped explicit path remains highest priority. + effective_inferencex_path = explicit_inferencex_path or runtime_inferencex_path or configured_inferencex_path + if ( + runtime_inferencex_path + and configured_inferencex_path + and Path(runtime_inferencex_path).expanduser().resolve() + != Path(configured_inferencex_path).expanduser().resolve() + ): + log.warning( + "Native AgentX: replacing benchmark.inferencex_path=%s with the preflight-validated INFERENCEX_PATH=%s", + configured_inferencex_path, + runtime_inferencex_path, + ) + else: + effective_inferencex_path = explicit_inferencex_path or configured_inferencex_path or runtime_inferencex_path if effective_inferencex_path: + effective_inferencex_path = str(Path(effective_inferencex_path).expanduser().resolve()) # Persist the resolved InferenceX checkout so Magpie's runtime checkout # matches Hyperloom's patch target. bench["inferencex_path"] = effective_inferencex_path @@ -1477,6 +1862,13 @@ def materialize_config_with_envs( "TP", "PORT", ): + # Hyperloom's process TP is the number of physical ranks reserved by + # the outer scheduler. In an AgentX recipe, envs.TP is inner tensor + # parallelism and physical ranks are TP x PP x PCP. Do not overwrite + # the recipe selector with the outer count when PP/PCP is greater than + # one; Magpie will persist the resolved inner TP below. + if native_agentx and env_key == "TP": + continue val = os.environ.get(env_key, "").strip() if val: envs[env_key] = _coerce_workload_int_env(env_key, val) @@ -1492,12 +1884,16 @@ def materialize_config_with_envs( tp_from_yaml = envs.get("TP") rocr_yaml = str(envs.get("ROCR_VISIBLE_DEVICES") or "").strip() rocr_devices = [d.strip() for d in rocr_yaml.split(",") if d.strip()] - if tp_from_env: + native_gpu_count = os.environ.get("HYPERLOOM_AGENTX_GPU_COUNT", "").strip() + if native_agentx and native_gpu_count: + resolved_tp = _coerce_workload_int_env("HYPERLOOM_AGENTX_GPU_COUNT", native_gpu_count) + elif tp_from_env: resolved_tp = int(tp_from_env) elif rocr_yaml and not tp_from_yaml: # Derive TP from the user-pinned GPU list when the YAML doesn't set TP. resolved_tp = len(rocr_devices) - envs["TP"] = resolved_tp + if not native_agentx: + envs["TP"] = resolved_tp else: resolved_tp = int(tp_from_yaml or 1) # Auto-clamp TP to the visible GPU count. Override via @@ -1515,8 +1911,15 @@ def materialize_config_with_envs( visible, ) resolved_tp = visible - envs["TP"] = resolved_tp - if not rocr_yaml or len(rocr_devices) < resolved_tp: + if not native_agentx: + envs["TP"] = resolved_tp + if native_agentx and not rocr_yaml: + # The pinned InferenceX agentic launchers copy ROCR's physical values + # back into HIP_VISIBLE_DEVICES after ROCR has re-indexed the devices. + # Pin the only mapping that is safe for those scripts; recipe resolution + # below verifies the count and rejects nonzero masks. + envs["ROCR_VISIBLE_DEVICES"] = ",".join(str(i) for i in range(resolved_tp)) + elif not native_agentx and (not rocr_yaml or len(rocr_devices) < resolved_tp): derived = ",".join(str(i) for i in range(resolved_tp)) if rocr_yaml and rocr_yaml != derived: log.warning( @@ -1539,6 +1942,182 @@ def materialize_config_with_envs( osl_val = int(envs.get("OSL") or _default_osl) conc_val = int(envs.get("CONC") or _default_conc) + if native_agentx: + # Persisted epoch-two recipes have no server launch contract. Keep + # their measurement-only boundary; epoch-three candidates use the + # formal launcher interface instead of generic EXTRA_*_ARGS hooks. + framework_env = server_args_env_name(bench.get("framework")) + source_server_args = str(envs.get(framework_env) or "").strip() + ref_args, reference_envs, reference_controls = resolve_reference_launch() + unsupported: list[str] = [] + if source_server_args: + unsupported.append(f"benchmark.envs.{framework_env}") + if server_args: + unsupported.append("extra_server_args") + if ref_args or reference_envs or any(reference_controls.values()): + unsupported.append("reference launch recipe") + if remove_args: + unsupported.append("remove_args") + if unset_envs: + unsupported.append("unset_envs") + if replace_args: + unsupported.append("args_mode=replace") + if drop_moe_runner_backend: + unsupported.append("drop_moe_runner_backend") + if flydsl_source_dirs: + unsupported.append("flydsl_source_dirs") + + combined_extra: dict[str, Any] = {} if native_launch_overrides is not None else dict(_operator_extra_env()) + if extra_envs: + combined_extra.update(extra_envs) + safe_extra_envs, dropped_extra_envs = filter_untrusted_env_mapping( + combined_extra, + allow_predicate=is_allowed_variant_env_key, + ) + for key in dropped_extra_envs: + log.warning( + "Dropping unsafe extra_envs key %s before AgentX materialization", + key, + ) + allowed_round_envs = {"CONC", "NUM_PROMPTS", "NUM_WARMUPS"} + unsupported_extra = sorted(set(safe_extra_envs).difference(allowed_round_envs)) + if unsupported_extra: + unsupported.append("extra_envs=" + ",".join(unsupported_extra)) + from ._native_candidate import apply_native_candidate, has_launch_contract + + optimizer_contract = has_launch_contract(bench) + if native_launch_overrides is not None and not optimizer_contract: + raise ValueError("Native launch snapshots cannot upgrade a measurement-only session") + if optimizer_contract: + if native_launch_overrides is not None: + from ._native_candidate import install_native_launch_snapshot + + install_native_launch_snapshot(bench, native_launch_overrides) + source_server_args, ref_args, reference_envs, reference_controls = "", "", {}, {} + server_args = str(extra_server_args or "").strip() + if dropped_extra_envs: + raise ValueError( + f"Native candidate contains unsupported environment keys: {sorted(dropped_extra_envs)}" + ) + if flydsl_source_dirs: + source_dirs = flydsl_extra_source_dirs() + if not source_dirs: + raise ValueError("Native FlyDSL candidates require discoverable source roots") + safe_extra_envs.setdefault(ENV_FLYDSL_EXTRA_SOURCE_DIRS, source_dirs) + if source_server_args: + apply_native_candidate(bench, extra_server_args=source_server_args) + apply_native_candidate( + bench, + extra_server_args=ref_args, + extra_envs=reference_envs, + remove_args=reference_controls.get("remove_args"), + unset_envs=reference_controls.get("unset_envs"), + args_mode=str(reference_controls.get("args_mode") or "append"), + overlay_pythonpath=str(reference_controls.get("overlay_pythonpath") or ""), + ) + candidate_remove = to_str_list(remove_args) + if drop_moe_runner_backend: + candidate_remove.append("--moe-runner-backend") + apply_native_candidate( + bench, + extra_server_args=server_args, + extra_envs=safe_extra_envs, + remove_args=candidate_remove, + unset_envs=unset_envs, + args_mode="replace" if replace_args else "append", + ) + elif unsupported: + raise ValueError( + "Native AgentX cannot apply Hyperloom server candidates with " + "the pinned InferenceX launcher (no optimizer-argv hook): " + "; ".join(unsupported) + ) + + if "CONC" in safe_extra_envs: + requested_conc = _coerce_workload_int_env( + "CONC", + str(safe_extra_envs["CONC"]), + ) + if requested_conc != conc_val: + raise ValueError( + "Native AgentX concurrency is fixed for the session: " + f"resolved CONC={conc_val}, candidate requested {requested_conc}" + ) + envs["CONC"] = requested_conc + # NUM_PROMPTS/NUM_WARMUPS are synthetic-client controls and are allowed + # in generic sweep payloads only so they can be discarded explicitly. + envs.pop("NUM_PROMPTS", None) + envs.pop("NUM_WARMUPS", None) + envs.pop(framework_env, None) + + # Re-settle the public switch after the fixed CONC has landed, then ask + # the pinned Magpie implementation to resolve and persist the exact + # InferenceX recipe point before any caller can acquire GPU resources. + apply_agentx_switch( + bench, + model_path, + conc=envs.get("CONC"), + active=True, + native_agentx_mode=native_agentx_mode, + grading=grading, + allow_profile_compat=allow_agentx_profile_compat, + ) + if not effective_inferencex_path: + raise ValueError( + "Native AgentX requires inferencex_path or INFERENCEX_PATH during benchmark materialization" + ) + # Magpie overlays benchmark.envs on its inherited environment before + # invoking the InferenceX launcher. Persist the runtime search paths + # so a later resume uses (or rejects drift from) the same ROCm/Python + # libraries instead of silently inheriting a different login shell. + from hyperloom.inference_optimizer.agentx.native import native_runtime_search_paths + + envs.update(native_runtime_search_paths()) + # The native resolver fingerprints the complete BenchmarkConfig it is + # handed. Remove credentials before that boundary so the persisted + # YAML and its launch-config fingerprint describe the same config. + # Filtering only after resolution makes every source YAML carrying a + # control-plane credential fail launch-boundary revalidation even + # though the credential itself is correctly omitted from the file. + envs = bench.setdefault("envs", {}) + filtered_envs, dropped_credentials = filter_untrusted_env_mapping( + envs, + allow_predicate=lambda key: key not in BENCHMARK_SECRET_ENV_NAMES, + ) + if dropped_credentials: + log.warning( + "Dropping control-plane credentials from benchmark envs: %s", + ", ".join(sorted(dropped_credentials)), + ) + envs.clear() + envs.update(filtered_envs) + + from hyperloom.inference_optimizer.agentx.native import resolve_native_recipe + + resolve_native_recipe( + bench, + inferencex_path=effective_inferencex_path, + expected_gpu_count=resolved_tp, + ) + envs = bench.setdefault("envs", {}) + # Pinned recipe inputs do not add credentials. Keep that invariant + # explicit so a future resolver change fails closed instead of either + # persisting a secret or mutating the already-fingerprinted config. + _, unsafe_resolved_envs = filter_untrusted_env_mapping( + envs, + allow_predicate=lambda key: key not in BENCHMARK_SECRET_ENV_NAMES, + ) + if unsafe_resolved_envs: + raise ValueError( + "Native AgentX resolver produced unsafe benchmark envs after " + "execution fingerprinting: " + ", ".join(sorted(unsafe_resolved_envs)) + ) + seal_server_argv(envs, bench.get("framework")) + output_dir.mkdir(parents=True, exist_ok=True) + materialized = output_dir / out_name + with materialized.open("w", encoding="utf-8") as f: + yaml.safe_dump(cfg, f, sort_keys=False) + return materialized + # Steady-state window for profiling configs (detected by YAML # ``benchmark.envs.PROFILE`` or ``profiler.torch_profiler.enabled``, not the process env). # The captured-step count is capped at @@ -1655,10 +2234,17 @@ def materialize_config_with_envs( is_sglang = "sglang" in fw sglang_sitecustomize = is_sglang and resolve_sglang_shape_mode() == "sitecustomize" - patch_attempted = _tracelens_patch_enabled() and not is_atom and not sglang_sitecustomize - # Written in every branch, not only the failing one: "patched fine" and "never tried because the image - # already carries it" call for different reactions when a trace later turns up without annotations. + patch_requested = _tracelens_patch_enabled() and not is_atom and not sglang_sitecustomize + patch_attempted = patch_requested and not native_profile_compat + # Written in every branch, not only the failing one. "No status" used to mean both "patched fine" and + # "never tried because the image already carries it", and those two call for different reactions when a + # trace later turns up without annotations. envs["HYPERLOOM_TRACELENS_PATCH_STATUS"] = "not_attempted" + if patch_requested and native_profile_compat: + # Diagnostic capture shares the recipe's installed framework. + # Patching it would change subsequent native measurements. + envs["HYPERLOOM_PROFILE_DEGRADED_REASON"] = _TRACELENS_PATCH_UNAVAILABLE + log.info("AgentX diagnostic profiling preserves the installed framework; source patches are skipped.") if patch_attempted: if "vllm" in fw: tracelens_patch_ok = ensure_vllm_patched_for_tracelens() @@ -2240,6 +2826,21 @@ def materialize_config_with_envs( if "--mark-trace" not in extra: envs["EXTRA_ATOM_ARGS"] = f"{extra} --mark-trace".strip() _apply_vllm_source_runtime(bench, envs) + if agentx_workload: + # Re-settle after process envs, reference controls, variant extra_envs, + # and unset_envs have all landed. In particular this keeps the launcher's + # CONC, the native AgentX selector, warmup grace, and the GEAK handoff on + # one round value. The switch is idempotent and deliberately removes a + # stale source ``agentx.concurrency`` in favor of envs.CONC. + apply_agentx_switch( + bench, + model_path, + conc=envs.get("CONC"), + active=True, + native_agentx_mode=native_agentx_mode, + grading=grading, + allow_profile_compat=allow_agentx_profile_compat, + ) # The rendered YAML is persisted, so credentials must not reach it. filtered_envs, dropped_credentials = filter_untrusted_env_mapping( envs, diff --git a/src/hyperloom/orchestrator/actions/executors/baseline.py b/src/hyperloom/orchestrator/actions/executors/baseline.py index 14b9f4d419..23ee6f7bb1 100644 --- a/src/hyperloom/orchestrator/actions/executors/baseline.py +++ b/src/hyperloom/orchestrator/actions/executors/baseline.py @@ -34,6 +34,12 @@ from hyperloom.common.model_paths import resolve_session_model_path from hyperloom.common.perf_metric import is_agentx_mode from hyperloom.common.timeutil import now_iso +from hyperloom.common.agentx_mode import ( + config_enables_native_agentx, + managed_native_agentx_session, + native_agentx_optimization_session, + native_agentx_session, +) from hyperloom.inference_optimizer.breakdown.recorder.baseline_event import ( ROUND_ACCURACY, ROUND_MEASURE, @@ -1631,6 +1637,7 @@ class BenchmarkRunExecutor: """One benchmark action: materialize the config, launch the backend, retry, parse the report.""" benchmark_watchdog = False + allow_agentx_profile_compat = False session_dir = SessionDirField() def __init__( @@ -1659,6 +1666,10 @@ def _resolve_default_config(self) -> Path: """Resolve the benchmark YAML the round runs when the task names none.""" raise NotImplementedError + def _agentx_profile_compatibility_error(self, shared_state: Any) -> str: + """Return a subclass-specific diagnostic profile constraint.""" + return "" + def _resolve_workspace(self, ctx: RunnerContext, action: str) -> Path: """Pick the per-task workspace dir.""" params = ctx.task.params or {} @@ -1713,6 +1724,23 @@ def _after_materialize_config( """Hook after YAML materialization, before launch; a returned dict aborts the round with it.""" return None + def _agentx_runtime_checkout( + self, + *, + config_path: Path, + output_dir: Path, + inferencex_path: str, + agentx_session: bool, + ) -> tuple[str, dict[str, Any] | None]: + """Select the checkout runtime preparation may mutate. + + Baseline/grid native AgentX uses its pinned checkout directly and the + native runtime path is read-only. ``ProfileExecutor`` overrides this + hook because its generic compatibility client deploys and patches + files that must never touch that pinned tree. + """ + return inferencex_path, None + @staticmethod def _failure_carries_markers( result: dict[str, Any], @@ -2306,6 +2334,35 @@ async def _run_once( effective_extra_server_args = str(params.get("extra_server_args") or "") extra = getattr(ctx, "extra", None) or {} live_shared_state = extra.get("shared_state") or self.shared_state + native_agentx = native_agentx_session(live_shared_state) + native_optimizer = native_agentx_optimization_session(live_shared_state) + native_launch = native_optimizer and ( + not self.allow_agentx_profile_compat or managed_native_agentx_session(live_shared_state) + ) + mutation_replay = ( + str(getattr(ctx.task, "kind", "") or "") == "replay_warm_recipe" + or bool(params.get("patches")) + or bool(params.get("warm_kernel_plan")) + ) + if native_agentx and not native_optimizer and mutation_replay: + return { + "status": "skipped", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "Native AgentX cannot replay source or kernel mutations until " + "the pinned InferenceX launcher exposes a fingerprinted " + "optimizer hook." + ), + "output_dir": str(self._resolve_workspace(ctx, "baseline")), + } + profile_compat_error = self._agentx_profile_compatibility_error(live_shared_state) if native_agentx else "" + if profile_compat_error: + return { + "status": "skipped", + "error_class": "agentx_profile_topology_unsupported", + "error": profile_compat_error, + "output_dir": str(self._resolve_workspace(ctx, "baseline")), + } fw = str(params.get("framework") or "").strip() or os.environ.get("FRAMEWORK", "").strip() if not fw and self._eager_fallback_armed(live_shared_state): log.warning( @@ -2365,6 +2422,18 @@ async def _run_once( # Accuracy eval (GSM8K) opt-out: ``--no-eval``, the ``disable_run_eval`` param and the eval-failure fallback # force ``RUN_EVAL=false``. base_extra_envs = dict(params.get("extra_envs") or {}) + _rt_from_params = params.get("runtime_override") + if native_agentx and not native_optimizer and isinstance(_rt_from_params, dict) and _rt_from_params: + return { + "status": "failed", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "Native AgentX cannot apply runtime_override with the pinned " + "InferenceX launcher. PATH/PYTHONPATH/framework runtime " + "changes would not be bound by the recipe fingerprint." + ), + "output_dir": str(output_dir), + } eval_disabled = self._eval_disabled(ctx) # The staged accuracy round is itself an eval, so ``--no-eval`` cancels it. defer_accuracy_until_after_measure = not eval_disabled and is_truthy( @@ -2373,15 +2442,18 @@ async def _run_once( if force_disable_eval or is_truthy(params.get("disable_run_eval")) or eval_disabled: base_extra_envs["RUN_EVAL"] = "false" await _prepare_aiter_serving_so(base_extra_envs, output_dir) + complete_native_snapshot = params.get("native_launch_overrides") if native_launch else None try: config_path = materialize_config_with_envs( config_path, output_dir, - extra_server_args=effective_extra_server_args, - extra_envs=base_extra_envs, - remove_args=params.get("remove_args"), - unset_envs=params.get("unset_envs"), - args_mode=str(params.get("args_mode") or "append"), + extra_server_args="" if complete_native_snapshot is not None else effective_extra_server_args, + extra_envs=None if complete_native_snapshot is not None else base_extra_envs, + remove_args=None if complete_native_snapshot is not None else params.get("remove_args"), + unset_envs=None if complete_native_snapshot is not None else params.get("unset_envs"), + args_mode="append" + if complete_native_snapshot is not None + else str(params.get("args_mode") or "append"), model_path=resolved_model, gpu_type=resolved_gpu, benchmark_script=override_script, @@ -2389,7 +2461,12 @@ async def _run_once( drop_moe_runner_backend=force_drop_moe_runner_backend, flydsl_source_dirs=is_truthy(params.get("flydsl_source_dirs")), agentx_mode=agentx_active(live_shared_state), + native_agentx_mode=native_agentx, + native_launch_overrides=None + if not native_launch + else params.get("native_launch_overrides") or params.get("base_native_launch_overrides"), grading=getattr(live_shared_state, "grading", None), + allow_agentx_profile_compat=self.allow_agentx_profile_compat, ) except FrameworkScriptMismatchError as exc: # Cross-framework script override: return a structured failure. @@ -2404,8 +2481,17 @@ async def _run_once( effective_inferencex_path = os.environ.get("INFERENCEX_PATH", "").strip() # Apply runtime_override from params into the materialized YAML so the revalidation baseline boots under the # same framework runtime as the KEEP'd candidate (PATH/PYTHONPATH/framework_bin etc.). - _rt_from_params = params.get("runtime_override") - if isinstance(_rt_from_params, dict) and _rt_from_params: + if native_launch: + from ._native_candidate import update_native_candidate_file + + update_native_candidate_file( + config_path, + runtime_override=_rt_from_params, + overlay_pythonpath=str(params.get("overlay_pythonpath") or params.get("final_overlay") or ""), + source_files=params.get("native_source_files"), + absent_source_files=params.get("native_absent_source_files") or (), + ) + elif isinstance(_rt_from_params, dict) and _rt_from_params: try: import yaml as _yaml @@ -2433,6 +2519,17 @@ async def _run_once( model_path=resolved_model, args_mode=str(params.get("args_mode") or "append"), ) + effective_inferencex_path, checkout_error = await asyncio.to_thread( + self._agentx_runtime_checkout, + config_path=config_path, + output_dir=output_dir, + inferencex_path=effective_inferencex_path, + agentx_session=native_agentx, + ) + if checkout_error is not None: + checkout_error.setdefault("materialized_config", str(config_path)) + checkout_error.setdefault("output_dir", str(output_dir)) + return checkout_error # AgentX: deploy the aiperf client into InferenceX benchmarks/ and # capability-preflight aiperf before Magpie runs the materialized config. # Baseline/profile shell out here (not via _run_magpie), so without this the @@ -2447,10 +2544,14 @@ async def _run_once( # ``_grid_runner`` already runs its copy of this through ``to_thread``. _agx_err = await asyncio.to_thread( prepare_agentx_runtime, - env=os.environ, + # Runtime preparation may scrub AgentX-only controls and a remote + # MODEL_PATH. Validate/deploy with a copy here; the exact child + # environment is prepared again immediately before spawn. + env=dict(os.environ), inferencex_path=effective_inferencex_path, config_path=config_path, active=agentx_active(live_shared_state), + allow_profile_compat=self.allow_agentx_profile_compat, ) if _agx_err: return { @@ -2656,6 +2757,27 @@ def _persist_recipe_snapshot(tree_records: list[dict[str, Any]]) -> bool: else: applied_patches = patch_application _pre_patch_sha = before_apply_sha + if native_optimizer and (params.get("patches") or params.get("warm_kernel_apply_results")): + from ._native_source import warm_source_evidence + + try: + evidence = warm_source_evidence(params, output_dir) + update_native_candidate_file(config_path, **evidence) + except (OSError, ValueError, KeyError, RuntimeError, subprocess.SubprocessError) as exc: + rollback = _revert_warm_patch_trees(params.get("_warm_patch_trees") or []) + kernel_rollback = _rollback_warm_kernel_apply_results( + params.get("warm_kernel_apply_results") or [], params.get("warm_kernel_snapshots") + ) + rollback["errors"] = list(rollback.get("errors") or []) + list(kernel_rollback.get("errors") or []) + rollback["ok"] = bool(rollback.get("ok") and kernel_rollback.get("ok")) + return { + "status": "failed", + "error_class": "native_source_attestation_failed", + "error": str(exc), + "rollback": rollback, + "output_dir": str(output_dir), + } + if applied_patches: log.info( "baseline_executor: prepared %d warm-replay code patches: %s", @@ -3388,7 +3510,8 @@ async def _run_single_benchmark( "run_eval_disabled": bool(run_eval_disabled), } # InferenceX ``run_lm_eval`` cleans ``$EVAL_RESULT_DIR`` after processing lm-eval output. - env["EVAL_RESULT_DIR"] = str(result_dir / "eval_output") + if not config_enables_native_agentx(config_path): + env["EVAL_RESULT_DIR"] = str(result_dir / "eval_output") # Pin SERVER_LOG / GPU_METRICS_CSV per-task so wrappers write into the task workspace; # ``harvest_leaked_artifacts`` is the defense-in-depth net. env["SERVER_LOG"] = str(output_dir / "server.log") @@ -3425,6 +3548,25 @@ async def _run_single_benchmark( silence_timeout_sec, timeout_sec = resolve_benchmark_timeouts() env["PYTHONUNBUFFERED"] = "1" sync_benchmark_timeout(config_path, timeout_sec) + try: + from hyperloom.inference_optimizer.agentx.runtime import ( + maybe_prepare_agentx, + ) + + maybe_prepare_agentx( + env=env, + inferencex_path=inferencex_path, + config_path=config_path, + allow_profile_compat=self.allow_agentx_profile_compat, + ) + except (ImportError, OSError, RuntimeError, TypeError, ValueError) as exc: + return { + "status": "failed", + "error_class": AGENTX_PREFLIGHT_ERROR_CLASS, + "error": (f"AgentX launch-boundary validation failed: {type(exc).__name__}: {exc}"), + "output_dir": str(output_dir), + "materialized_config": str(materialized_config_path), + } if not ctx_extra.get("mn_round_restarted"): try: # Merge the reference base UNDER the per-task args (last-wins) so a multi-node per-round restart @@ -3795,11 +3937,40 @@ def record_bringup(*, wrapper_stderr: str = "", wrapper_stdout: str = "") -> str except (OSError, json.JSONDecodeError): report = None + if self.allow_agentx_profile_compat: + from ._native_profile import managed_profile_benchmark + from ._managed_profile_result import diagnostic_profile_result + + configured = (yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}).get("benchmark") or {} + if managed_profile_benchmark(configured): + return { + **diagnostic_profile_result( + report, + workspace=workspace, + config_path=config_path, + returncode=proc_returncode, + subprocess_started_unix=subprocess_started_unix, + ), + "returncode": proc_returncode, + "output_dir": str(output_dir), + "workspace": str(workspace), + "report_path": str(report_path), + "materialized_config": str(materialized_config_path), + "subprocess_runtime_sec": round(subprocess_runtime_sec, 2), + "nonfatal_warnings": round_warnings, + **capture_meta, + } + measurement = extract_benchmark_measurement( report, workspace=workspace, subprocess_started_unix=subprocess_started_unix, + materialized_config_path=config_path, ) + if measurement.get("agentx_launch_contract") == 1: + from ._native_candidate import record_launch_evidence + + record_launch_evidence(config_path, measurement) warnings = round_warnings + list(measurement.pop("nonfatal_warnings", []) or []) for leak_src, _ in harvested: warnings.append(f"harvested_leaked_artifact:{leak_src}") @@ -3827,7 +3998,12 @@ def record_bringup(*, wrapper_stderr: str = "", wrapper_stdout: str = "") -> str error = f"benchmark_report.json missing under {workspace}" else: error_class = "invalid_measurement" - error = "benchmark report did not contain positive throughput and completed requests" + protocol_errors = measurement.get("native_agentx_protocol_errors") or [] + error = ( + "native AgentX report failed protocol validation: " + "; ".join(protocol_errors) + if protocol_errors + else "benchmark report did not contain positive throughput and completed requests" + ) error = redact_secret_values(error) return { "status": "failed", @@ -3944,11 +4120,12 @@ def record_bringup(*, wrapper_stderr: str = "", wrapper_stdout: str = "") -> str result["eval_probe"] = eval_probe log.warning("baseline_executor: %s", eval_probe_summary(eval_probe)) + from hyperloom.inference_optimizer.performance_display import format_measurement_metric + log.info( - "baseline_executor: %s %.1f %s (output) e2el=%.1fms", + "baseline_executor: %s %s (output) e2el=%.1fms", "success_with_warning" if warnings else "success", - result["output_throughput"], - framework_registry.throughput_unit(eval_framework), + format_measurement_metric(eval_framework, result), result["e2el_mean_ms"] or 0.0, ) return result @@ -4013,6 +4190,21 @@ def _after_materialize_config( output_dir: Path, ) -> dict[str, Any] | None: """Hook after YAML materialization, before launch.""" + try: + cfg = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError): + cfg = {} + bench = cfg.get("benchmark") if isinstance(cfg, dict) else {} + if isinstance(bench, dict): + from hyperloom.inference_optimizer.agentx.native import ( + native_agentx_enabled, + ) + + if native_agentx_enabled(bench.get("agentx")): + # Native AgentX executes the pinned upstream launcher and + # benchmark_lib.sh with RUN_EVAL=false. Generic-client/eval + # compatibility patchers must not mutate that checkout. + return None ix_root = self._inferencex_root_from_config(config_path) if ix_root: ensure_benchmark_lib_eval_dest_patched(Path(ix_root)) diff --git a/src/hyperloom/orchestrator/actions/executors/benchmark_result.py b/src/hyperloom/orchestrator/actions/executors/benchmark_result.py index b2406aa7e4..faba178fa0 100644 --- a/src/hyperloom/orchestrator/actions/executors/benchmark_result.py +++ b/src/hyperloom/orchestrator/actions/executors/benchmark_result.py @@ -8,6 +8,7 @@ import logging import csv import json +import math import os import time import re @@ -15,6 +16,9 @@ from pathlib import Path from typing import Any +import yaml + +from hyperloom.common.agentx_accounting import measured_request_errors from hyperloom.common.coerce import first_float, first_int, to_float, to_int from hyperloom.common.jsonio import read_json @@ -544,6 +548,7 @@ def _merge_raw_result( measurement["raw_result_path"] = str(source_path) # AgentX scenario verdict. if "submission_valid" in raw and "submission_valid" not in measurement: + measurement["scenario"] = "agentx" measurement["submission_valid"] = raw.get("submission_valid") reasons = raw.get("submission_invalid_reasons") or [] measurement["submission_invalid_reasons"] = ( @@ -566,6 +571,1018 @@ def _merge_raw_result( LATENCY_DERIVED = "derived_from_e2el_ttft" LATENCY_UNAVAILABLE = "unavailable" +_TRUSTED_NATIVE_AGENTX_GPU_COUNT_SOURCES = frozenset( + { + "magpie_report_recipe", + "magpie_config_snapshot", + "materialized_workload_metadata", + "inferencex_raw", + } +) + + +def _valid_recipe_fingerprint(value: Any) -> str | None: + """Return an InferenceX recipe fingerprint only when strictly valid.""" + if not isinstance(value, str): + return None + fingerprint = value + if len(fingerprint) != 64 or any(character not in "0123456789abcdef" for character in fingerprint): + return None + return fingerprint + + +def _native_agentx_explicit_topology_fields(value: Any) -> tuple[dict[str, int], bool]: + """Return explicit native topology fields and whether all were valid. + + InferenceX's aggregate currently serializes only ``tp`` and ``ep`` even + though the fingerprint-bound Magpie snapshot retains TP x PP x PCP. Keep + partial fields available for consistency checks without ever deriving a + GPU count from them. + """ + if not isinstance(value, dict): + return {}, False + + def explicit_positive_integer(*keys: str) -> tuple[int | None, bool]: + present_values = [value[key] for key in keys if key in value] + if not present_values: + return None, False + parsed_values: list[int] = [] + for raw in present_values: + parsed = to_float(raw) + if parsed is None or parsed <= 0 or not parsed.is_integer(): + return None, True + parsed_values.append(int(parsed)) + if len(set(parsed_values)) != 1: + return None, True + return parsed_values[0], True + + aliases = { + "tp": ("tp",), + "pp": ("pp",), + "pcp": ("pcp_size", "pcp-size", "pcp"), + "ep": ("ep",), + "gpu_count": ("gpu_count",), + } + fields: dict[str, int] = {} + for canonical, keys in aliases.items(): + parsed, present = explicit_positive_integer(*keys) + if present and parsed is None: + return {}, False + if parsed is not None: + fields[canonical] = parsed + return fields, True + + +def _native_agentx_topology(value: Any) -> tuple[dict[str, int], int] | None: + """Extract an explicit AgentX topology without inventing PP/PCP defaults.""" + fields, fields_valid = _native_agentx_explicit_topology_fields(value) + if not fields_valid: + return None + tp = fields.get("tp") + pp = fields.get("pp") + pcp = fields.get("pcp") + explicit_gpu_count = fields.get("gpu_count") + dimensions = (tp, pp, pcp) + if all(dimension is not None for dimension in dimensions): + assert tp is not None and pp is not None and pcp is not None + calculated_gpu_count = tp * pp * pcp + if explicit_gpu_count is not None and explicit_gpu_count != calculated_gpu_count: + return None + topology = {"tp": tp, "pp": pp, "pcp": pcp} + if "ep" in fields: + topology["ep"] = fields["ep"] + return topology, calculated_gpu_count + if explicit_gpu_count is not None and explicit_gpu_count > 0 and all(dimension is None for dimension in dimensions): + topology = {"gpu_count": explicit_gpu_count} + if "ep" in fields: + topology["ep"] = fields["ep"] + return topology, explicit_gpu_count + return None + + +def _set_native_agentx_topology( + measurement: dict[str, Any], + value: Any, + *, + source: str, +) -> bool: + """Record a validated native topology and its provenance.""" + parsed = _native_agentx_topology(value) + if parsed is None: + return False + topology, gpu_count = parsed + existing_gpu_count = first_int(measurement.get("agentx_gpu_count")) + if existing_gpu_count is not None and existing_gpu_count != gpu_count: + return False + existing_topology = measurement.get("agentx_gpu_topology") + if isinstance(existing_topology, dict): + for dimension in ("tp", "pp", "pcp", "ep", "gpu_count"): + if ( + dimension in existing_topology + and dimension in topology + and existing_topology[dimension] != topology[dimension] + ): + return False + topology = {**existing_topology, **topology} + measurement["agentx_gpu_topology"] = topology + measurement["agentx_gpu_count"] = gpu_count + measurement["agentx_gpu_count_source"] = source + return True + + +def _merge_native_agentx_report( + measurement: dict[str, Any], + report: dict[str, Any], +) -> None: + """Normalize Magpie's native ``agentx_metrics`` into Hyperloom fields.""" + metrics = report.get("agentx_metrics") + if not isinstance(metrics, dict): + return + + raw_throughput = metrics.get("throughput") + throughput = raw_throughput if isinstance(raw_throughput, dict) else {} + raw_requests = metrics.get("requests") + requests = raw_requests if isinstance(raw_requests, dict) else {} + raw_latency = metrics.get("latency_seconds") + latency = raw_latency if isinstance(raw_latency, dict) else {} + + schema_errors: list[str] = [] + + def positive_number(value: Any) -> bool: + parsed = to_float(value) + return parsed is not None and math.isfinite(parsed) and parsed > 0 + + if not isinstance(raw_throughput, dict): + schema_errors.append("agentx_metrics.throughput_missing") + else: + for key in ("output_tokens_per_second", "total_tokens_per_second", "duration_seconds"): + if not positive_number(throughput.get(key)): + schema_errors.append(f"agentx_metrics.throughput.{key}_invalid") + if not isinstance(raw_requests, dict): + schema_errors.append("agentx_metrics.requests_missing") + else: + successful_for_schema = first_int(requests.get("successful")) + profiled_for_schema = first_int(requests.get("profiled_total")) + errors_for_schema = first_int(requests.get("errors")) + error_rate_for_schema = to_float(requests.get("error_rate")) + if successful_for_schema is None or successful_for_schema <= 0: + schema_errors.append("agentx_metrics.requests.successful_invalid") + if profiled_for_schema is None or profiled_for_schema <= 0: + schema_errors.append("agentx_metrics.requests.profiled_total_invalid") + if errors_for_schema is None or errors_for_schema < 0: + schema_errors.append("agentx_metrics.requests.errors_invalid") + if ( + error_rate_for_schema is None + or not math.isfinite(error_rate_for_schema) + or not 0.0 <= error_rate_for_schema <= 1.0 + ): + schema_errors.append("agentx_metrics.requests.error_rate_invalid") + if not isinstance(raw_latency, dict): + schema_errors.append("agentx_metrics.latency_seconds_missing") + interactivity_for_schema = latency.get("e2e_norm_intvty") + if not isinstance(interactivity_for_schema, dict) or not positive_number(interactivity_for_schema.get("p90")): + schema_errors.append("agentx_metrics.latency_seconds.e2e_norm_intvty.p90_invalid") + recipe_for_schema = metrics.get("recipe") + recipe_fingerprint: str | None = None + if not isinstance(recipe_for_schema, dict): + schema_errors.append("agentx_metrics.recipe_missing") + else: + recipe_fingerprint = _valid_recipe_fingerprint(recipe_for_schema.get("recipe_fingerprint")) + if recipe_fingerprint is None: + schema_errors.append("agentx_metrics.recipe.recipe_fingerprint_invalid") + recipe_tp = first_int(recipe_for_schema.get("tp")) + if recipe_tp is None or recipe_tp <= 0: + schema_errors.append("agentx_metrics.recipe.tp_invalid") + launch_for_schema = metrics.get("launch") + launch_fingerprint: str | None = None + if not isinstance(launch_for_schema, dict): + schema_errors.append("agentx_metrics.launch_missing") + else: + launch_fingerprint = _valid_recipe_fingerprint(launch_for_schema.get("recipe_fingerprint")) + if launch_fingerprint is None: + schema_errors.append("agentx_metrics.launch.recipe_fingerprint_invalid") + elif recipe_fingerprint is not None and launch_fingerprint != recipe_fingerprint: + schema_errors.append("agentx_metrics.recipe_launch_fingerprint_mismatch") + if str(metrics.get("mode") or "").strip().lower() not in {"canonical", "fast"}: + schema_errors.append("agentx_metrics.mode_invalid") + if str(metrics.get("scenario_type") or "").strip().lower() != "agentic-coding": + schema_errors.append("agentx_metrics.scenario_type_invalid") + if metrics.get("recipe_fingerprint_valid") is not True: + schema_errors.append("agentx_metrics.recipe_fingerprint_valid_invalid") + if not isinstance(report.get("benchmark_valid"), bool): + schema_errors.append("benchmark_valid_missing") + if not isinstance(report.get("publishable"), bool): + schema_errors.append("publishable_missing") + measurement["native_agentx_schema_errors"] = schema_errors + measurement["native_agentx_schema_valid"] = not schema_errors + + native_throughput_fields = { + "request_throughput": "request_throughput", + "input_throughput": "input_tokens_per_second", + "output_throughput": "output_tokens_per_second", + "total_token_throughput": "total_tokens_per_second", + "duration_seconds": "duration_seconds", + } + for target, source in native_throughput_fields.items(): + value = to_float(throughput.get(source)) + if value is not None: + measurement[target] = value + + successful = first_int(requests.get("successful")) + if successful is not None: + measurement["completed_requests"] = successful + # AgentX is duration-bounded. Magpie's ``profiled_total`` is successful + # responses plus error-dropped records, not a requested-session target. + # Treating it as ``requested_requests`` makes the generic fixed-request + # completeness check reject an otherwise complete canonical replay. + measurement["requested_requests"] = None + measurement["request_errors"] = first_int(requests.get("errors")) + native_error_rate = to_float(requests.get("error_rate")) + # Hyperloom's legacy result contract records request error rate as a + # percentage. Magpie's native AgentX schema uses a 0..1 ratio. + measurement["request_error_rate"] = native_error_rate * 100.0 if native_error_rate is not None else None + + def latency_stat_ms(metric: str, stat_name: str) -> float | None: + values = latency.get(metric) + if not isinstance(values, dict): + return None + seconds = to_float(values.get(stat_name)) + return seconds * 1000.0 if seconds is not None else None + + # Magpie's generic LatencyMetrics serializer emits p99=0 even though the + # pinned InferenceX AgentX aggregate supplies p95, not p99. Clear those + # placeholders; a missing percentile is not a zero-latency observation. + measurement["ttft_p99_ms"] = None + measurement["e2el_p99_ms"] = None + for target, metric, stat_name in ( + ("ttft_mean_ms", "ttft", "mean"), + ("ttft_p50_ms", "ttft", "p50"), + ("ttft_p90_ms", "ttft", "p90"), + ("ttft_p95_ms", "ttft", "p95"), + ("tpot_mean_ms", "tpot", "mean"), + ("tpot_p50_ms", "tpot", "p50"), + ("tpot_p90_ms", "tpot", "p90"), + ("tpot_p95_ms", "tpot", "p95"), + ("e2el_mean_ms", "e2el", "mean"), + ("e2el_p95_ms", "e2el", "p95"), + ): + value = latency_stat_ms(metric, stat_name) + if value is not None: + measurement[target] = value + + interactivity = latency.get("e2e_norm_intvty") + if isinstance(interactivity, dict): + # This is a rate (1 / normalized E2E seconds), not a latency. Keep the + # native unit; multiplying it by 1000 would corrupt Hyperloom's 2-D + # AgentX objective. + measurement["e2e_norm_intvty_p90"] = to_float(interactivity.get("p90")) + measurement["e2e_norm_intvty_p50"] = to_float(interactivity.get("p50")) + + benchmark_valid = report.get("benchmark_valid") + publishable = report.get("publishable") + measurement["benchmark_valid"] = benchmark_valid + measurement["publishable"] = publishable + # Canonical Hyperloom measurements must pass both native validity and the + # recipe-fingerprint/canonical-mode gate represented by ``publishable``. + measurement["submission_valid"] = ( + benchmark_valid is True and publishable is True + if isinstance(benchmark_valid, bool) and isinstance(publishable, bool) + else None + ) + reasons = report.get("errors") or [] + invalid_reasons = [str(reason) for reason in reasons] if isinstance(reasons, list) else [str(reasons)] + if benchmark_valid is not True: + invalid_reasons.append("native_agentx_benchmark_invalid") + if publishable is not True: + mode = str(metrics.get("mode") or "").strip().lower() + invalid_reasons.append("native_agentx_fast_mode" if mode == "fast" else "native_agentx_not_publishable") + measurement["submission_invalid_reasons"] = list(dict.fromkeys(invalid_reasons)) + measurement["agentx_mode"] = metrics.get("mode") + measurement["agentx_requests"] = requests + # Keep the compact payloads until the protocol validator has cross-bound + # them to InferenceX's aggregate. The normalized fields omit the complete + # latency/accounting maps and cannot by themselves prove a single run. + measurement["agentx_report_throughput"] = throughput + measurement["agentx_report_latency_seconds"] = latency + recipe = metrics.get("recipe") + measurement["agentx_recipe"] = recipe + measurement["agentx_launch"] = metrics.get("launch") + measurement["agentx_server_launch"] = metrics.get("server_launch") + measurement["agentx_request_accounting"] = metrics.get("request_accounting") + if recipe_fingerprint is not None: + measurement["agentx_recipe_fingerprint"] = recipe_fingerprint + if launch_fingerprint is not None: + measurement["agentx_launch_recipe_fingerprint"] = launch_fingerprint + if recipe_fingerprint is not None and launch_fingerprint == recipe_fingerprint: + # Current Magpie omits PCP from agentx_metrics.recipe. Only accept the + # compact report itself when every parallelism dimension is explicit. + _set_native_agentx_topology(measurement, recipe, source="magpie_report_recipe") + measurement["agentx_report_dataset"] = metrics.get("dataset") + measurement["agentx_dataset"] = metrics.get("dataset") + + +def _merge_native_agentx_workspace_topology(measurement: dict[str, Any], workspace: Path) -> None: + """Use Magpie's resolved config snapshot as a fingerprint-bound topology.""" + expected_fingerprint = _valid_recipe_fingerprint(measurement.get("agentx_recipe_fingerprint")) + if expected_fingerprint is None: + return + config_path = workspace / "config.yaml" + if not config_path.is_file(): + return + try: + config = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError): + measurement["nonfatal_warnings"].append("native_agentx_config_snapshot_unreadable") + return + if not isinstance(config, dict): + return + + wrapped = config.get("benchmark") + if isinstance(wrapped, dict): + config = wrapped + + agentx = config.get("agentx") + resolved = agentx.get("resolved") if isinstance(agentx, dict) else None + if isinstance(resolved, dict): + resolved_fingerprint = _valid_recipe_fingerprint( + resolved.get("recipe-fingerprint", resolved.get("recipe_fingerprint")) + ) + if resolved_fingerprint != expected_fingerprint: + measurement["nonfatal_warnings"].append("native_agentx_config_recipe_fingerprint_mismatch") + measurement["native_agentx_topology_conflict"] = True + return + if not _set_native_agentx_topology(measurement, resolved, source="magpie_config_snapshot"): + measurement["nonfatal_warnings"].append("native_agentx_config_topology_invalid") + measurement["native_agentx_topology_conflict"] = True + return + + # Forward-compatible path for a materializer that persists an explicitly + # fingerprint-bound workload topology in Magpie's workspace snapshot. + workload_spec = config.get("workload_spec") + topology = workload_spec.get("resolved_topology") if isinstance(workload_spec, dict) else None + if not isinstance(topology, dict): + return + topology_fingerprint = _valid_recipe_fingerprint( + topology.get("recipe_fingerprint", topology.get("recipe-fingerprint")) + ) + if topology_fingerprint != expected_fingerprint: + measurement["nonfatal_warnings"].append("native_agentx_workload_recipe_fingerprint_mismatch") + measurement["native_agentx_topology_conflict"] = True + return + if not _set_native_agentx_topology(measurement, topology, source="materialized_workload_metadata"): + measurement["nonfatal_warnings"].append("native_agentx_workload_topology_invalid") + measurement["native_agentx_topology_conflict"] = True + + +def _native_distribution(value: Any) -> dict[str, int]: + """Map InferenceX's native token distribution onto Hyperloom names.""" + if not isinstance(value, dict): + return {} + out: dict[str, int] = {} + for source, target in (("mean", "avg"), ("p50", "p50"), ("p75", "p75"), ("p90", "p90"), ("p95", "p95")): + parsed = to_float(value.get(source)) + if parsed is not None and math.isfinite(parsed) and parsed >= 0: + out[target] = int(round(parsed)) + return out + + +def _native_agentx_raw_matches_report( + measurement: dict[str, Any], + raw: dict[str, Any], + *, + source_path: Path, +) -> bool: + """Validate raw aggregate identity before any of its payload is consumed.""" + warnings = measurement["nonfatal_warnings"] + expected_fingerprint = _valid_recipe_fingerprint(measurement.get("agentx_recipe_fingerprint")) + raw_fingerprint = _valid_recipe_fingerprint(raw.get("recipe_fingerprint")) + if expected_fingerprint is None or raw_fingerprint is None: + warnings.append(f"native_agentx_raw_recipe_fingerprint_missing:{source_path}") + return False + if raw_fingerprint != expected_fingerprint: + warnings.append(f"native_agentx_raw_recipe_fingerprint_mismatch:{source_path}") + return False + + raw_fields, raw_fields_valid = _native_agentx_explicit_topology_fields(raw) + raw_has_topology_field = any(key in raw for key in ("tp", "pp", "pcp", "pcp_size", "pcp-size", "ep", "gpu_count")) + if raw_has_topology_field and not raw_fields_valid: + warnings.append(f"native_agentx_raw_topology_invalid:{source_path}") + return False + + raw_topology = _native_agentx_topology(raw) + if raw_topology is not None: + parsed_raw_topology, raw_gpu_count = raw_topology + expected_gpu_count = first_int(measurement.get("agentx_gpu_count")) + if expected_gpu_count is not None and expected_gpu_count != raw_gpu_count: + warnings.append(f"native_agentx_raw_topology_mismatch:{source_path}") + return False + expected_topology = measurement.get("agentx_gpu_topology") + if isinstance(expected_topology, dict): + for dimension in ("tp", "pp", "pcp", "ep"): + if ( + dimension in expected_topology + and dimension in parsed_raw_topology + and expected_topology[dimension] != parsed_raw_topology[dimension] + ): + warnings.append(f"native_agentx_raw_topology_mismatch:{source_path}") + return False + return True + + # Pinned InferenceX aggregates currently expose TP + EP, not PP + PCP. + # Such a raw file may enrich corpus/cache telemetry only after Magpie's + # fingerprint-bound snapshot established the complete physical topology. + # In particular, TP alone must never become the per-GPU divisor. + expected_gpu_count = first_int(measurement.get("agentx_gpu_count")) + expected_topology = measurement.get("agentx_gpu_topology") + expected_dimensions = ( + { + dimension: value + for dimension in ("tp", "pp", "pcp") + if (value := first_int(expected_topology.get(dimension))) is not None and value > 0 + } + if isinstance(expected_topology, dict) + else {} + ) + trusted_topology = ( + expected_gpu_count is not None + and expected_gpu_count > 0 + and measurement.get("agentx_gpu_count_source") in _TRUSTED_NATIVE_AGENTX_GPU_COUNT_SOURCES + and len(expected_dimensions) == 3 + and expected_gpu_count == expected_dimensions["tp"] * expected_dimensions["pp"] * expected_dimensions["pcp"] + ) + if not trusted_topology: + warnings.append(f"native_agentx_raw_topology_invalid:{source_path}") + return False + + assert isinstance(expected_topology, dict) + for dimension in ("tp", "pp", "pcp"): + if dimension in raw_fields and raw_fields[dimension] != expected_dimensions[dimension]: + warnings.append(f"native_agentx_raw_topology_mismatch:{source_path}") + return False + if "gpu_count" in raw_fields and raw_fields["gpu_count"] != expected_gpu_count: + warnings.append(f"native_agentx_raw_topology_mismatch:{source_path}") + return False + if "ep" in raw_fields: + expected_ep = first_int(expected_topology.get("ep")) + if expected_ep is None: + recipe = measurement.get("agentx_recipe") + if isinstance(recipe, dict): + expected_ep = first_int(recipe.get("ep")) + if expected_ep is not None and raw_fields["ep"] != expected_ep: + warnings.append(f"native_agentx_raw_topology_mismatch:{source_path}") + return False + return True + + +def _merge_native_agentx_raw( + measurement: dict[str, Any], + raw: dict[str, Any], + *, + source_path: Path, +) -> bool: + """Enrich a native Magpie report from InferenceX's aggregate JSON. + + Magpie intentionally exposes a compact AgentX summary. The underlying + aggregate retains corpus distributions, cache telemetry, and PCP topology + that Hyperloom needs for its prompts and per-GPU comparisons. + """ + if str(raw.get("scenario_type") or "").strip().lower() != "agentic-coding": + return False + if not _native_agentx_raw_matches_report(measurement, raw, source_path=source_path): + return False + request_metrics = raw.get("request_metrics") + request_metrics = request_metrics if isinstance(request_metrics, dict) else {} + tokens = request_metrics.get("tokens") + tokens = tokens if isinstance(tokens, dict) else {} + measurement["isl_distribution"] = _native_distribution(tokens.get("input")) + measurement["osl_distribution"] = _native_distribution(tokens.get("output_actual")) + + dataset = raw.get("dataset") + if isinstance(dataset, dict): + measurement["agentx_dataset"] = dataset + measurement["corpus_loader"] = str(dataset.get("loader") or dataset.get("name") or "") + cache = request_metrics.get("cache") + cache = cache if isinstance(cache, dict) else {} + measurement["theoretical_prefix_cache_hit"] = to_float(cache.get("theoretical_cache_hit_rate")) + server_metrics = raw.get("server_metrics") + if isinstance(server_metrics, dict): + measurement["agentx_server_cache"] = server_metrics.get("cache") + + _set_native_agentx_topology(measurement, raw, source="inferencex_raw") + measurement["raw_result_path"] = str(source_path) + return True + + +def _validate_native_agentx_protocol( + measurement: dict[str, Any], + *, + workspace: Path | None, + raw: dict[str, Any] | None, + raw_path: Path | None, + materialized_config_path: Path | None = None, +) -> None: + """Cross-bind the native report to its exact AIPerf protocol artifacts. + + Magpie's compact report is deliberately convenient, but its current + ``benchmark_valid`` gate only proves positive throughput and an acceptable + error ratio. A one-second or cancelled run can therefore look publishable + if that report is considered alone. Native AgentX is accepted only when + the Magpie summary, InferenceX aggregate, materialized recipe snapshot and + AIPerf's own scenario export all describe the same complete run. + """ + + errors: list[str] = [] + + def reject(code: str) -> None: + if code not in errors: + errors.append(code) + + def strict_number(value: Any) -> float | None: + if isinstance(value, bool) or not isinstance(value, int | float): + return None + parsed = float(value) + return parsed if math.isfinite(parsed) else None + + def strict_non_negative_int(value: Any) -> int | None: + if isinstance(value, bool) or not isinstance(value, int): + return None + return value if value >= 0 else None + + def metric_avg(payload: dict[str, Any], key: str) -> float | None: + metric = payload.get(key) + if not isinstance(metric, dict): + return None + return strict_number(metric.get("avg")) + + config: dict[str, Any] = {} + if workspace is None: + reject("workspace_missing") + else: + config_path = workspace / "config.yaml" + try: + loaded = yaml.safe_load(config_path.read_text(encoding="utf-8")) + except (OSError, yaml.YAMLError): + loaded = None + reject("config_snapshot_unreadable") + if isinstance(loaded, dict): + wrapped = loaded.get("benchmark") + config = wrapped if isinstance(wrapped, dict) else loaded + else: + reject("config_snapshot_invalid") + + # Magpie intentionally snapshots only its own BenchmarkConfig fields. + # The launch authority and fixed session identity remain in Hyperloom's + # materialized input; bind the two before using that private metadata. + from hyperloom.inference_optimizer.agentx.identity import has_launch_contract + + expected: dict[str, Any] = {} + if materialized_config_path is not None: + try: + loaded_expected = yaml.safe_load(Path(materialized_config_path).read_text(encoding="utf-8")) + except (OSError, yaml.YAMLError): + loaded_expected = None + if isinstance(loaded_expected, dict): + wrapped = loaded_expected.get("benchmark") + expected = wrapped if isinstance(wrapped, dict) else loaded_expected + else: + reject("materialized_config_unreadable") + if has_launch_contract(config) or has_launch_contract(expected): + if not expected: + reject("materialized_config_missing") + else: + observed_config = {key: value for key, value in config.items() if key != "workload_spec"} + expected_config = {key: value for key, value in expected.items() if key != "workload_spec"} + if observed_config != expected_config: + reject("materialized_config_snapshot_mismatch") + config = expected + + raw_agentx = config.get("agentx") + agentx = raw_agentx if isinstance(raw_agentx, dict) else {} + raw_resolved = agentx.get("resolved") + resolved = raw_resolved if isinstance(raw_resolved, dict) else {} + if not resolved: + reject("config_resolved_recipe_missing") + workload = config.get("workload_spec") + workload = workload if isinstance(workload, dict) else {} + + expected_fingerprint = _valid_recipe_fingerprint( + resolved.get("recipe-fingerprint", resolved.get("recipe_fingerprint")) + ) + if expected_fingerprint is None: + reject("config_recipe_fingerprint_invalid") + elif expected_fingerprint != _valid_recipe_fingerprint(measurement.get("agentx_recipe_fingerprint")): + reject("config_report_recipe_fingerprint_mismatch") + + expected_concurrency = first_int( + resolved.get("conc"), + workload.get("concurrency"), + ) + expected_duration = first_int( + resolved.get("duration"), + workload.get("duration_s"), + ) + if expected_concurrency is None or expected_concurrency <= 0: + reject("config_concurrency_invalid") + if expected_duration is None or expected_duration <= 0: + reject("config_duration_invalid") + if str(measurement.get("agentx_mode") or "").strip().lower() == "canonical" and expected_duration != 3600: + reject("canonical_duration_not_3600") + + report_recipe = measurement.get("agentx_recipe") + report_recipe = report_recipe if isinstance(report_recipe, dict) else {} + report_launch = measurement.get("agentx_launch") + report_launch = report_launch if isinstance(report_launch, dict) else {} + + def require_same(label: str, *values: Any, fold: bool = False) -> None: + normalized: list[Any] = [] + for value in values: + if value is None or value == "": + reject(f"{label}_missing") + return + item = str(value).strip().lower() if fold else value + normalized.append(item) + if len(set(normalized)) != 1: + reject(f"{label}_mismatch") + + if raw is None or raw_path is None: + reject("inferencex_aggregate_missing") + raw = {} + else: + measurement["native_agentx_aggregate_path"] = str(raw_path) + + if resolved.get("custom") is True: + if report_recipe.get("custom_recipe") is not True or raw.get("custom_recipe") is not True: + reject("custom_recipe_identity_missing") + for output_field, config_field in ( + ("native_context_length", "native-context-length"), + ("max_model_len", "max-model-len"), + ("model_config_sha256", "model-config-sha256"), + ): + require_same( + output_field, resolved.get(config_field), report_recipe.get(output_field), raw.get(output_field) + ) + + require_same( + "recipe_fingerprint", + expected_fingerprint, + report_recipe.get("recipe_fingerprint"), + report_launch.get("recipe_fingerprint"), + raw.get("recipe_fingerprint"), + ) + require_same("concurrency", expected_concurrency, report_recipe.get("conc"), raw.get("conc")) + require_same("model", config.get("model"), resolved.get("model"), report_recipe.get("model"), raw.get("model")) + require_same( + "framework", + config.get("framework"), + resolved.get("framework"), + report_recipe.get("framework"), + raw.get("framework"), + fold=True, + ) + require_same( + "precision", + config.get("precision"), + resolved.get("precision"), + report_recipe.get("precision"), + raw.get("precision"), + fold=True, + ) + require_same( + "image", + config.get("docker_image"), + resolved.get("image"), + report_recipe.get("image"), + report_launch.get("docker_image"), + raw.get("image"), + ) + require_same( + "model_prefix", + resolved.get("model-prefix"), + report_recipe.get("infmax_model_prefix"), + raw.get("infmax_model_prefix"), + ) + require_same( + "launcher", + config.get("benchmark_script"), + report_launch.get("benchmark_script"), + ) + require_same("recipe_name", agentx.get("recipe"), report_launch.get("recipe")) + for label, config_key, report_key, raw_key in ( + ("tp", "tp", "tp", "tp"), + ("pp", "pp", "pp", "pp"), + ("ep", "ep", "ep", "ep"), + ("pcp", "pcp-size", None, "pcp_size"), + ): + values = [resolved.get(config_key), raw.get(raw_key)] + if report_key is not None: + values.append(report_recipe.get(report_key)) + require_same(label, *values) + + accounting = raw.get("request_accounting") + accounting = accounting if isinstance(accounting, dict) else {} + records_total = strict_non_negative_int(accounting.get("records_total")) + records_profiled = strict_non_negative_int(accounting.get("records_profiled")) + records_dropped = strict_non_negative_int(accounting.get("records_dropped_total")) + records_warmup = strict_non_negative_int(accounting.get("records_warmup_dropped")) + records_errors = strict_non_negative_int(accounting.get("records_error_dropped")) + successful = strict_non_negative_int(raw.get("num_requests_successful")) + total = strict_non_negative_int(raw.get("num_requests_total")) + if None in ( + records_total, + records_profiled, + records_dropped, + records_warmup, + records_errors, + successful, + total, + ): + reject("request_accounting_invalid") + else: + assert records_total is not None + assert records_profiled is not None + assert records_dropped is not None + assert records_warmup is not None + assert records_errors is not None + assert successful is not None + assert total is not None + if records_total != records_profiled + records_dropped: + reject("request_accounting_sum_mismatch") + if successful != records_profiled: + reject("request_accounting_success_mismatch") + if total != records_total: + reject("request_accounting_total_mismatch") + if records_profiled <= 0: + reject("request_accounting_empty") + measured_errors = measured_request_errors(accounting) + if measured_errors is None: + reject("request_accounting_drop_bounds_mismatch") + error_categories = accounting.get("error_categories") + if not isinstance(error_categories, dict): + reject("request_accounting_error_categories_invalid") + else: + category_counts = [strict_non_negative_int(value) for value in error_categories.values()] + if ( + any(value is None for value in category_counts) + or sum(value for value in category_counts if value is not None) != records_errors + ): + reject("request_accounting_error_categories_mismatch") + metrics_requests = measurement.get("completed_requests") + if first_int(metrics_requests) != successful: + reject("report_success_count_mismatch") + + report_accounting = measurement.get("agentx_request_accounting") + if report_accounting != accounting: + reject("report_request_accounting_mismatch") + report_requests = measurement.get("agentx_requests") + report_requests = report_requests if isinstance(report_requests, dict) else {} + if strict_non_negative_int(report_requests.get("total")) != records_total: + reject("report_request_total_mismatch") + if strict_non_negative_int(report_requests.get("records_total")) != records_total: + reject("report_records_total_mismatch") + if strict_non_negative_int(report_requests.get("warmup_dropped")) != records_warmup: + reject("report_warmup_count_mismatch") + if measured_errors is not None: + if strict_non_negative_int(report_requests.get("profiled_total")) != (successful + measured_errors): + reject("report_profiled_total_mismatch") + if strict_non_negative_int(report_requests.get("errors")) != measured_errors: + reject("report_error_count_mismatch") + error_rate_denominator = successful + measured_errors + report_error_rate = strict_number(report_requests.get("error_rate")) + if error_rate_denominator <= 0: + reject("report_error_rate_invalid") + elif report_error_rate is None or not math.isclose( + report_error_rate, + measured_errors / error_rate_denominator, + rel_tol=1e-12, + abs_tol=1e-12, + ): + reject("report_error_rate_mismatch") + + request_metrics = raw.get("request_metrics") + request_metrics = request_metrics if isinstance(request_metrics, dict) else {} + raw_throughput = request_metrics.get("throughput") + raw_throughput = raw_throughput if isinstance(raw_throughput, dict) else {} + raw_qps = request_metrics.get("qps") + raw_qps = raw_qps if isinstance(raw_qps, dict) else {} + report_throughput = measurement.get("agentx_report_throughput") + report_throughput = report_throughput if isinstance(report_throughput, dict) else {} + + def nested_metric(payload: dict[str, Any], section: str, key: str) -> float | None: + nested = payload.get(section) + if not isinstance(nested, dict): + return None + return strict_number(nested.get(key)) + + throughput_pairs = ( + ( + "request_throughput", + strict_number(report_throughput.get("request_throughput")), + strict_number(raw_qps.get("mean")), + ), + ( + "input_tokens_per_second", + strict_number(report_throughput.get("input_tokens_per_second")), + nested_metric(raw_throughput, "input", "tokens_per_second"), + ), + ( + "output_tokens_per_second", + strict_number(report_throughput.get("output_tokens_per_second")), + nested_metric(raw_throughput, "output", "tokens_per_second"), + ), + ( + "total_tokens_per_second", + strict_number(report_throughput.get("total_tokens_per_second")), + nested_metric(raw_throughput, "total", "tokens_per_second"), + ), + ( + "duration_seconds", + strict_number(report_throughput.get("duration_seconds")), + strict_number(raw_throughput.get("duration_seconds")), + ), + ) + for label, report_value, raw_value in throughput_pairs: + if report_value is None or raw_value is None: + reject(f"report_{label}_missing") + elif report_value != raw_value: + reject(f"report_{label}_mismatch") + + raw_latency = request_metrics.get("latency") + report_latency = measurement.get("agentx_report_latency_seconds") + if not isinstance(raw_latency, dict) or not isinstance(report_latency, dict): + reject("report_latency_missing") + elif report_latency != raw_latency: + reject("report_latency_mismatch") + + aiperf: dict[str, Any] = {} + artifact_path: Path | None = None + if workspace is not None: + try: + artifacts = sorted(workspace.rglob("profile_export_aiperf.json")) + except OSError: + artifacts = [] + if len(artifacts) != 1: + reject("aiperf_artifact_missing" if not artifacts else "aiperf_artifact_ambiguous") + else: + artifact_path = artifacts[0] + payload = read_json(artifact_path, default=None, require_dict=True) + if isinstance(payload, dict): + aiperf = payload + measurement["native_agentx_aiperf_path"] = str(artifact_path) + else: + reject("aiperf_artifact_unreadable") + + metadata = aiperf.get("metadata") + metadata = metadata if isinstance(metadata, dict) else {} + if metadata.get("scenario") != "inferencex-agentx-mvp": + reject("aiperf_scenario_mismatch") + if metadata.get("submission_valid") is not True: + reject("aiperf_submission_invalid") + raw_reasons = metadata.get("submission_invalid_reasons") + if isinstance(raw_reasons, list): + for reason in raw_reasons: + reject(f"aiperf:{reason}") + if aiperf.get("was_cancelled") is not False: + reject("aiperf_cancelled_or_unknown") + + input_config = aiperf.get("input_config") + input_config = input_config if isinstance(input_config, dict) else {} + if input_config.get("scenario") != "inferencex-agentx-mvp": + reject("aiperf_input_scenario_mismatch") + expected_model = str(config.get("model") or "").strip() + models = input_config.get("models") + model_items = models.get("items") if isinstance(models, dict) else None + if not isinstance(model_items, list) or len(model_items) != 1 or not isinstance(model_items[0], dict): + reject("aiperf_models_invalid") + elif str(model_items[0].get("name") or "").strip() != expected_model: + reject("aiperf_model_mismatch") + tokenizer = input_config.get("tokenizer") + if not isinstance(tokenizer, dict) or not str(tokenizer.get("name") or "").strip(): + reject("aiperf_tokenizer_invalid") + elif str(tokenizer.get("name") or "").strip() != expected_model: + reject("aiperf_tokenizer_mismatch") + phases = input_config.get("phases") + profiling_phases = ( + [ + phase + for phase in phases + if isinstance(phase, dict) and str(phase.get("kind") or phase.get("name") or "").lower() == "profiling" + ] + if isinstance(phases, list) + else [] + ) + if len(profiling_phases) != 1: + reject("aiperf_profiling_phase_invalid") + phase: dict[str, Any] = {} + else: + phase = profiling_phases[0] + if first_int(phase.get("concurrency")) != expected_concurrency: + reject("aiperf_concurrency_mismatch") + phase_duration = strict_number(phase.get("duration")) + if expected_duration is None or phase_duration != float(expected_duration): + reject("aiperf_configured_duration_mismatch") + if str(phase.get("timing_mode") or "").strip().lower() != "agentic_replay": + reject("aiperf_timing_mode_mismatch") + + coverage = metadata.get("metric_duration_coverage") + coverage_rows = coverage if isinstance(coverage, list) else [] + if len(coverage_rows) != 1 or not isinstance(coverage_rows[0], dict): + reject("aiperf_duration_coverage_missing") + else: + row = coverage_rows[0] + required_ratio = strict_number(row.get("required_ratio")) + ttft_ratio = strict_number(row.get("ttft_ratio")) + itl_ratio = strict_number(row.get("inter_token_latency_ratio")) + coverage_duration = strict_number(row.get("expected_duration_seconds")) + if profiling_phases and row.get("phase_name") != profiling_phases[0].get("name"): + reject("aiperf_duration_coverage_phase_mismatch") + if expected_duration is None or coverage_duration != float(expected_duration): + reject("aiperf_duration_coverage_expected_mismatch") + if required_ratio is None or required_ratio < 0.95 or required_ratio > 1.0: + reject("aiperf_duration_coverage_threshold_invalid") + if ( + required_ratio is None + or ttft_ratio is None + or itl_ratio is None + or max(ttft_ratio, itl_ratio) < required_ratio + ): + reject("aiperf_duration_coverage_failed") + + observed_duration = metric_avg(aiperf, "benchmark_duration") + if observed_duration is None or expected_duration is None or observed_duration < 0.95 * expected_duration: + reject("aiperf_benchmark_duration_incomplete") + aiperf_successes = metric_avg(aiperf, "request_count") + if successful is None or aiperf_successes != float(successful): + reject("aiperf_success_count_mismatch") + + aiperf_dataset = metadata.get("dataset") + report_dataset = measurement.get("agentx_report_dataset") + raw_dataset = raw.get("dataset") + if not all(isinstance(value, dict) for value in (aiperf_dataset, report_dataset, raw_dataset)): + reject("dataset_provenance_missing") + else: + assert isinstance(aiperf_dataset, dict) + assert isinstance(report_dataset, dict) + assert isinstance(raw_dataset, dict) + if aiperf_dataset != report_dataset or aiperf_dataset != raw_dataset: + reject("dataset_provenance_mismatch") + if aiperf_dataset.get("source_type") != "public_dataset": + reject("dataset_not_public") + if first_int(aiperf_dataset.get("num_dataset_entries")) != 393: + reject("dataset_entry_count_mismatch") + loader = str(aiperf_dataset.get("loader") or "") + if not loader.startswith("semianalysis_cc_traces_weka"): + reject("dataset_loader_invalid") + configured_corpus = str(workload.get("corpus") or "").strip() + if configured_corpus and loader != configured_corpus: + reject("dataset_config_mismatch") + + if measurement.get("native_agentx_aggregate_ambiguous") is True: + reject("inferencex_aggregate_ambiguous") + + from hyperloom.inference_optimizer.agentx.identity import ( + canonical_sha256, + has_launch_contract, + validate_server_launch, + verified_workload_fingerprint, + ) + + if has_launch_contract(config): + measurement["agentx_launch_contract"] = 1 + measurement.pop("agentx_workload_fingerprint", None) + measurement.pop("agentx_candidate_fingerprint", None) + for error in validate_server_launch(config, measurement.get("agentx_server_launch"), workspace=workspace): + reject(error) + try: + launch_artifact = json.loads((workspace / "agentx_server_launch.json").read_text(encoding="utf-8")) + except (OSError, ValueError, TypeError): + reject("server_launch_artifact_missing") + else: + if launch_artifact != measurement.get("agentx_server_launch"): + reject("server_launch_artifact_mismatch") + workload_fingerprint = verified_workload_fingerprint(config) + if workload_fingerprint is None: + reject("workload_fingerprint_invalid") + elif not errors: + measurement["agentx_workload_fingerprint"] = canonical_sha256( + {"workload": workload_fingerprint, "dataset": aiperf_dataset} + ) + measurement["agentx_candidate_fingerprint"] = measurement["agentx_server_launch"]["evidence_sha256"] + + measurement["native_agentx_protocol_errors"] = errors + measurement["native_agentx_protocol_valid"] = not errors + reasons = measurement.get("submission_invalid_reasons") + reasons = ( + [str(reason) for reason in reasons if not str(reason).startswith("native_agentx_protocol:")] + if isinstance(reasons, list) + else [] + ) + if errors: + reasons.extend(f"native_agentx_protocol:{error}" for error in errors) + measurement["submission_invalid_reasons"] = list(dict.fromkeys(reasons)) + measurement["submission_valid"] = bool( + measurement.get("benchmark_valid") is True and measurement.get("publishable") is True and not errors + ) + def _latency_snapshot(measurement: dict[str, Any]) -> dict[str, Any]: """The latency fields as they stand, for comparing across a fill pass. @@ -624,6 +1641,7 @@ def extract_benchmark_measurement( *, workspace: Path | None = None, subprocess_started_unix: float | None = None, + materialized_config_path: Path | None = None, ) -> dict[str, Any]: """Extract a normalized measurement from Magpie and InferenceX outputs. @@ -637,6 +1655,8 @@ def extract_benchmark_measurement( and (as a fallback) salvageable leaks. subprocess_started_unix: Optional launch time enabling the mtime-gated leak salvage pass. + materialized_config_path: Exact Hyperloom input submitted to Magpie; + required to bind native optimization results to their session. Returns: A normalized measurement dict (including ``valid_measurement``, any @@ -652,6 +1672,7 @@ def extract_benchmark_measurement( measurement: dict[str, Any] = { "reported_success": report.get("success") if report else None, + "scenario": report.get("scenario"), "framework": report.get("framework"), "model": report.get("model"), # Scriptable (server-less) workloads tag the report with workload_kind/unit and ship a quality_gate block @@ -684,8 +1705,16 @@ def extract_benchmark_measurement( "raw_result_path": None, "nonfatal_warnings": [], } + if str(report.get("scenario") or "").strip().lower() == "agentx": + measurement["native_agentx_report"] = True + measurement["native_agentx_schema_valid"] = False + _merge_native_agentx_report(measurement, report) + if workspace is not None and measurement.get("native_agentx_report") is True: + _merge_native_agentx_workspace_topology(measurement, workspace) origins: dict[str, str] = {} + native_aggregate: dict[str, Any] | None = None + native_aggregate_path: Path | None = None _tag_latency_origins( measurement, origins, @@ -696,7 +1725,22 @@ def extract_benchmark_measurement( if workspace is not None: for raw_path in _candidate_raw_jsons(workspace): raw = read_json(raw_path, default=None, require_dict=True) - if not raw or to_float(raw.get("output_throughput")) is None: + if not raw: + continue + if measurement.get("native_agentx_report") is True: + if str(raw.get("scenario_type") or "").strip().lower() != "agentic-coding": + continue + accepted = _merge_native_agentx_raw(measurement, raw, source_path=raw_path) + if accepted: + if native_aggregate is not None: + measurement["native_agentx_aggregate_ambiguous"] = True + native_aggregate = None + native_aggregate_path = None + break + native_aggregate = raw + native_aggregate_path = raw_path + continue + if to_float(raw.get("output_throughput")) is None: continue before = _latency_snapshot(measurement) _merge_raw_result(measurement, raw, source_path=raw_path) @@ -713,6 +1757,14 @@ def extract_benchmark_measurement( before = _latency_snapshot(measurement) _derive_tpot_if_missing(measurement, report) _tag_latency_origins(measurement, origins, label=LATENCY_DERIVED, before=before) + if measurement.get("native_agentx_report") is True: + _validate_native_agentx_protocol( + measurement, + workspace=workspace, + raw=native_aggregate, + raw_path=native_aggregate_path, + materialized_config_path=materialized_config_path, + ) measurement["valid_measurement"] = is_valid_measurement(measurement) # Second-chance salvage from Magpie leak destinations when the in-workspace search found no usable measurement @@ -723,7 +1775,19 @@ def extract_benchmark_measurement( subprocess_started_unix=subprocess_started_unix, ): raw = read_json(rescue_path, default=None, require_dict=True) - if not raw or to_float(raw.get("output_throughput")) is None: + if not raw: + continue + native_report = measurement.get("native_agentx_report") is True + native_raw = str(raw.get("scenario_type") or "").strip().lower() == "agentic-coding" + if native_report and not native_raw: + continue + if native_report and not _native_agentx_raw_matches_report( + measurement, + raw, + source_path=rescue_path, + ): + continue + if not native_report and to_float(raw.get("output_throughput")) is None: continue # Copy the leak into the workspace BEFORE merging so the NFS clone stays self-contained. materialized = _materialize_rescue_into_workspace( @@ -731,6 +1795,16 @@ def extract_benchmark_measurement( workspace, ) recorded_path = materialized if materialized is not None else rescue_path + if native_report: + accepted = _merge_native_agentx_raw(measurement, raw, source_path=recorded_path) + if accepted: + native_aggregate = raw + native_aggregate_path = recorded_path + warnings.append(f"rescued_from_leaked_path:{rescue_path}") + if materialized is None: + warnings.append(f"rescued_copy_into_workspace_failed: {rescue_path}") + break + continue before = _latency_snapshot(measurement) _merge_raw_result(measurement, raw, source_path=recorded_path) _tag_latency_origins(measurement, origins, label=LATENCY_FROM_RESCUED_RAW, before=before) @@ -742,6 +1816,14 @@ def extract_benchmark_measurement( before = _latency_snapshot(measurement) _derive_tpot_if_missing(measurement, report) _tag_latency_origins(measurement, origins, label=LATENCY_DERIVED, before=before) + if measurement.get("native_agentx_report") is True: + _validate_native_agentx_protocol( + measurement, + workspace=workspace, + raw=native_aggregate, + raw_path=native_aggregate_path, + materialized_config_path=materialized_config_path, + ) measurement["valid_measurement"] = is_valid_measurement(measurement) # One label for the pair, keyed on TTFT and falling back to E2EL, because @@ -807,28 +1889,45 @@ def is_valid_measurement(result: dict[str, Any] | None) -> bool: output_tput = to_float(result.get("output_throughput")) if output_tput is None or output_tput <= 0: return False - # Gated on BOTH the mode and the key's presence, and each half earns its keep. + if _is_scriptable_measurement(result): + # Scriptable workloads have their own image-quality contract and never + # run the AgentX replay. An ambient HYPERLOOM_AGENTX left in the parent + # shell must not make their otherwise valid result require an AgentX + # submission verdict. + from ._accuracy_gate import quality_gate_passed + + qg = result.get("quality_gate") + return quality_gate_passed(qg, require=False) + # Native reports and merged legacy AgentX raw results serialize their + # scenario, so a resumed subprocess need not inherit HYPERLOOM_AGENTX. + # A stray similarly named key on a synthetic result remains inert. from ._workload_envs import agentx_enabled - if agentx_enabled() and "submission_valid" in result: + agentx_result = str(result.get("scenario") or "").strip().lower() == "agentx" or agentx_enabled() + if result.get("native_agentx_report") is True and result.get("native_agentx_schema_valid") is not True: + return False + if result.get("native_agentx_report") is True: + if result.get("native_agentx_protocol_valid") is not True: + return False + gpu_count = first_int(result.get("agentx_gpu_count")) + if ( + result.get("native_agentx_topology_conflict") is True + or gpu_count is None + or gpu_count <= 0 + or result.get("agentx_gpu_count_source") not in _TRUSTED_NATIVE_AGENTX_GPU_COUNT_SOURCES + ): + return False + if agentx_result: verdict = result.get("submission_valid") if verdict is False: return False - if verdict is None: + if verdict is not True: # The verdict is unknown: no --scenario was requested or the aiperf build predates the field. map_aiperf # writes the key unconditionally, so None arrives as a present key. from hyperloom.common.env import env_bool if not env_bool("HYPERLOOM_ALLOW_UNVERIFIED_SUBMISSION"): return False - if _is_scriptable_measurement(result): - # A scriptable run whose image-quality gate failed is not selectable, regardless of throughput. - from ._accuracy_gate import quality_gate_passed - - qg = result.get("quality_gate") - if not quality_gate_passed(qg, require=False): - return False - return True completed = to_int(result.get("completed_requests")) return completed is not None and completed > 0 diff --git a/src/hyperloom/orchestrator/actions/executors/explore.py b/src/hyperloom/orchestrator/actions/executors/explore.py index e13051c2d7..d4a033118a 100644 --- a/src/hyperloom/orchestrator/actions/executors/explore.py +++ b/src/hyperloom/orchestrator/actions/executors/explore.py @@ -18,6 +18,7 @@ import yaml from hyperloom.common.coerce import to_str_list +from hyperloom.common.agentx_mode import native_agentx_optimization_session from hyperloom.common.env import is_truthy from hyperloom.common.gain_math import gain_pct from hyperloom.common.launch_log_evidence import launch_flag_setting_name, settings_the_engine_rewrote @@ -129,6 +130,8 @@ def is_stack_revalidation(params: dict | None) -> bool: "kb_evidence", "pr_evidence", "source_evidence", + "runtime_override", + "dropped_envs", "experience_citations", ) @@ -238,6 +241,8 @@ def _grid_variants_from_payload(payload: list[Any]) -> list[GridVariant]: gv.kb_evidence = list(raw.get("kb_evidence") or []) # type: ignore[attr-defined] gv.pr_evidence = list(raw.get("pr_evidence") or []) # type: ignore[attr-defined] gv.source_evidence = list(raw.get("source_evidence") or []) # type: ignore[attr-defined] + if isinstance(raw.get("runtime_override"), dict): + gv.runtime_override = dict(raw["runtime_override"]) # Validated against what the proposing agent was shown before the grid reached this executor. gv.experience_citations = [ # type: ignore[attr-defined] dict(item) for item in (raw.get("experience_citations") or []) if isinstance(item, dict) @@ -740,6 +745,7 @@ async def __call__(self, ctx) -> dict[str, Any]: } extra = getattr(ctx, "extra", None) or {} shared_state = extra.get("shared_state") or extra.get("state") + native_optimizer = native_agentx_optimization_session(shared_state) eval_disabled = _explore_eval_disabled(shared_state, params) output_root = Path( params.get("output_dir") @@ -777,6 +783,7 @@ async def __call__(self, ctx) -> dict[str, Any]: extra_envs={"RUN_EVAL": "false"} if eval_disabled else None, out_name="explore_base.with_envs.yaml", grading=getattr(shared_state, "grading", None), + native_launch_overrides=params.get("base_native_launch_overrides"), ) except FrameworkScriptMismatchError as exc: return { @@ -974,6 +981,16 @@ async def __call__(self, ctx) -> dict[str, Any]: base_unset_envs=base_unset_envs, base_args_mode=base_args_mode, ) + if native_optimizer and (gv.runtime_override or getattr(gv, "overlay_pythonpath", "")): + from hyperloom.inference_optimizer.agentx.identity import canonical_sha256 + + fp = canonical_sha256( + { + "config": fp, + "runtime_override": gv.runtime_override, + "overlay_pythonpath": getattr(gv, "overlay_pythonpath", ""), + } + )[:16] gv.canonical_fp = fp # type: ignore[attr-defined] if fp in unique_in_round: # In-round duplicate — keep the first occurrence. @@ -1057,6 +1074,7 @@ async def __call__(self, ctx) -> dict[str, Any]: stack_remove_args = list(dict.fromkeys(base_remove_args)) stack_unset_envs = list(dict.fromkeys(base_unset_envs)) stack_base_args_mode = base_args_mode + stack_native_snapshot = params.get("base_native_launch_overrides") running_base_tput = base_tput def _measured_against() -> dict[str, Any]: @@ -1188,6 +1206,12 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la args_mode=str(getattr(gv, "args_mode", "append") or "append"), ) _carry_variant_metadata(gv, run_gv) + if stack_native_snapshot is not None: + # The snapshot already contains the accepted base; only + # this proposal's delta may be layered onto it again. + run_gv.extra_envs = dict(gv.extra_envs) + run_gv.remove_args = list(gv.remove_args) + run_gv.unset_envs = list(gv.unset_envs) # ``--no-eval`` opted the session out of accuracy entirely, so it # holds for every round. The decision round additionally skips # eval when a warmup preceded it: it is timed against a @@ -1225,6 +1249,7 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la base_extra_envs=dict(stack_extra_envs), base_remove_args=list(stack_remove_args), base_unset_envs=list(stack_unset_envs), + base_native_launch_overrides=stack_native_snapshot, serving_lease=variant_lease, session_deadline_sec=session_deadline_sec, variant_expected_sec=warmup_expected_sec, @@ -1322,6 +1347,7 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la base_extra_envs=dict(stack_extra_envs), base_remove_args=list(stack_remove_args), base_unset_envs=list(stack_unset_envs), + base_native_launch_overrides=stack_native_snapshot, server_already_ready=use_warm_decision, serving_lease=variant_lease, session_deadline_sec=session_deadline_sec, @@ -1341,6 +1367,7 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la # A variant KEEPs when it clears the graded verdict and the accuracy gate. The axes and the # threshold floor belong to resolve_graded_comparison, which every lane shares. variant_meas = { + **r.native_measurement, GRADED_OUTPUT: r.output_throughput, "input_throughput": r.input_throughput, "total_throughput": r.total_token_throughput, @@ -1483,6 +1510,8 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la decision_tput = r.output_throughput tested_update[fp] = { + **r.native_measurement, + **({"materialized_config": r.materialized_config} if r.materialized_config else {}), "fingerprint": fp, "name": gv.name, "extra_server_args": gv.extra_server_args, @@ -1575,7 +1604,23 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la ) if persist_effective_args: effective_control_fields["args_mode"] = "replace" + native_snapshot = None + if r.native_measurement.get("agentx_launch_contract") == 1: + measured_config = yaml.safe_load(Path(r.materialized_config).read_text(encoding="utf-8")) + native_snapshot = measured_config["benchmark"]["agentx"]["launch_overrides"] + next_stack_args = next_effective_args = shlex.join(native_snapshot.get("append_args") or []) + next_envs = dict(native_snapshot.get("env") or {}) + _keep_remove_args = list(native_snapshot.get("remove_args") or []) + _keep_unset_envs = list(native_snapshot.get("unset_env") or []) + effective_control_fields.update( + remove_args=_keep_remove_args, + unset_envs=_keep_unset_envs, + args_mode="replace" if native_snapshot.get("replace_args") else "append", + native_launch_overrides=native_snapshot, + ) keep_entry = { + **variant_meas, + **({"materialized_config": r.materialized_config} if r.materialized_config else {}), "fingerprint": fp, "name": gv.name, "candidate_extra_server_args": gv.extra_server_args, @@ -1621,6 +1666,9 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la stack_remove_args = list(_keep_remove_args) stack_unset_envs = list(_keep_unset_envs) stack_base_args_mode = "replace" if persist_effective_args else "append" + if native_snapshot is not None: + stack_native_snapshot = native_snapshot + stack_base_args_mode = "replace" if native_snapshot.get("replace_args") else "append" if decision_tput and decision_tput > 0: running_base_tput = decision_tput if grade_on_intvty: @@ -1846,6 +1894,10 @@ def _stopped_by_the_run(result: Any, *, variant: GridVariant, idx: int, round_la key=lambda w: float(w.get("gain_pct") or 0.0), default=None, ) + if native_optimizer and winners: + # Each canonical KEEP advances the median objective and launch + # snapshot. Relative gains use different anchors within a round. + best_winner = winners[-1] best_gain_pct = float(best_winner.get("gain_pct") or 0.0) if best_winner else 0.0 # Each KEEP advances ``running_base_tput``, so this is the final stack. diff --git a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py index 629c92af28..3c10109555 100644 --- a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py +++ b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py @@ -56,6 +56,7 @@ from hyperloom.common.github_urls import repo_slug from hyperloom.common.gain_math import gain_pct from hyperloom.common.perf_metric import VERDICT_KEEP +from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session from ...bringup import load_boot_observation, observation_summary, verdict_of, write_boot_observation from ...delivery import file_digest from ...delivery.ledger import append_record, load_prepared_records, load_records, mark_prepared, restore_records @@ -1937,6 +1938,19 @@ async def _stage_resolve( "error_class": "integrate_restore_incomplete", "error": "previous integration still requires recovery", } + if native_agentx_session(shared_state) and not native_agentx_optimization_session(shared_state): + return { + "status": "skipped", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "Native AgentX cannot benchmark source or artifact mutations " + "until the pinned InferenceX launcher exposes a fingerprinted " + "optimizer hook." + ), + "specialist_task_id": str(params.get("specialist_task_id") or "").strip(), + "patches_applied": [], + "patches_reverted": [], + } if shared_state is not None and not params.get("accuracy_baseline"): _base_acc = getattr(shared_state, "baseline_accuracy", 0.0) if isinstance(_base_acc, (int, float)) and _base_acc > 0: @@ -2886,6 +2900,15 @@ async def _stage_gate( # timestamp, so it stays correct as the gate progresses. session_deadline_sec, variant_expected_sec = session_grid_bounds(shared_state) try: + if native_agentx_optimization_session(shared_state): + from ._native_source import applied_source_evidence + + evidence = applied_source_evidence(attempt.framework_root, attempt.applied, attempt.applied_artifacts) + params = { + **params, + "native_source_files": evidence["source_files"], + "native_absent_source_files": evidence["absent_source_files"], + } bench_result, gate_evidence = await self._bench_patch( params=params, output_root=output_root, @@ -2939,6 +2962,9 @@ async def _stage_gate( ) if attempt.dropped_env_overrides: verdict["dropped_env_overrides"] = attempt.dropped_env_overrides + if params.get("overlay_pythonpath") and verdict.get("status") == "kept": + verdict["final_overlay"] = str(params["overlay_pythonpath"]) + verdict["overlay_pythonpath"] = str(params["overlay_pythonpath"]) return verdict async def _gate_enablement( @@ -4522,16 +4548,25 @@ async def _bench_patch( ) override_script = sanitize_script_name(params.get("benchmark_script")) override_result_dir = sanitize_result_dir(params.get("result_dir")) + import yaml + + from ._native_candidate import has_launch_contract + + source_benchmark = (yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}).get("benchmark", {}) + native_optimizer = has_launch_contract(source_benchmark) config_path = materialize_config_with_envs( config_path, output_root, model_path=resolved_model or None, gpu_type=resolved_gpu or None, benchmark_script=override_script, - extra_envs=framework_run_eval_envs(params), - remove_args=params.get("base_remove_args"), - unset_envs=params.get("base_unset_envs"), - args_mode=str(params.get("base_args_mode") or "append"), + extra_envs={"RUN_EVAL": "false"} if native_optimizer else framework_run_eval_envs(params), + remove_args=None if params.get("base_native_launch_overrides") else params.get("base_remove_args"), + unset_envs=None if params.get("base_native_launch_overrides") else params.get("base_unset_envs"), + args_mode="append" + if params.get("base_native_launch_overrides") + else str(params.get("base_args_mode") or "append"), + native_launch_overrides=params.get("base_native_launch_overrides"), out_name="integrate_patch.with_envs.yaml", ) @@ -4550,11 +4585,17 @@ async def _bench_patch( name=f"integrate-patch-{specialist_task_id[:8]}{variant_suffix}", extra_server_args=extra_server_args_applied, extra_envs=variant_envs, - unset_envs=to_str_list(unset_envs), - args_mode=args_mode, + remove_args=to_str_list(params.get("remove_args")), + unset_envs=list(dict.fromkeys(to_str_list(params.get("unset_envs")) + to_str_list(unset_envs))), + args_mode=str(params.get("args_mode") or "append"), note=f"integrate_patch:{specialist_task_id}{variant_suffix}", ) + variant.overlay_pythonpath = str(params.get("overlay_pythonpath") or "") + variant.native_source_files = dict(params.get("native_source_files") or {}) + variant.native_absent_source_files = list(params.get("native_absent_source_files") or []) effective_unset = list(dict.fromkeys(base_unset + list(variant.unset_envs))) + effective_remove = list(dict.fromkeys(base_remove + list(variant.remove_args))) + effective_mode = "replace" if "replace" in (args_mode, variant.args_mode) else "append" _rt = params.get("runtime_override") if isinstance(_rt, dict) and _rt: # Preserve list/dict values; apply_runtime_override expects them. @@ -4607,6 +4648,7 @@ async def _bench_patch( base_extra_envs=base_envs, base_remove_args=base_remove, base_unset_envs=base_unset, + base_native_launch_overrides=params.get("base_native_launch_overrides"), serving_lease=serving_lease, session_deadline_sec=session_deadline_sec, variant_expected_sec=variant_expected_sec, @@ -4642,7 +4684,7 @@ async def _bench_patch( "launch_evidence_path": r.launch_evidence_path or "", "server_log_path": r.server_log_path or "", # Materialized config used for this bench; needed by revalidation. - "materialized_config": str(config_path), + "materialized_config": r.materialized_config or str(config_path), # The composed stack, not the variant alone: revalidation # replays this and would otherwise boot without the base # layer. RUN_EVAL is dropped -- the replay owns its own @@ -4657,15 +4699,22 @@ async def _bench_patch( inherited_args="", base_extra_args=str(params.get("base_extra_args") or "").strip(), variant_extra_args=variant.extra_server_args, - remove_args=base_remove, - args_mode=args_mode, + remove_args=effective_remove, + args_mode=variant.args_mode, ), - "remove_args": list(base_remove), + "remove_args": list(effective_remove), "unset_envs": list(effective_unset), - "args_mode": args_mode, + "args_mode": effective_mode, + **({"final_overlay": variant.overlay_pythonpath} if variant.overlay_pythonpath else {}), }, } ) + if native_optimizer and r.materialized_config: + from hyperloom.orchestrator.measurement.native_launch import native_launch_config + + # This is the verified server snapshot, including inherited + # runtime/source layers and the recipe's replace semantics. + bench["effective_config"].update(native_launch_config(bench)) # Classified here so the gate compares two observations produced by # the same reader. diff --git a/src/hyperloom/orchestrator/actions/executors/profile.py b/src/hyperloom/orchestrator/actions/executors/profile.py index c2c10ab100..9874434920 100644 --- a/src/hyperloom/orchestrator/actions/executors/profile.py +++ b/src/hyperloom/orchestrator/actions/executors/profile.py @@ -5,10 +5,13 @@ from __future__ import annotations +import asyncio import gzip import json import logging import os +import re +import subprocess import tempfile import uuid from pathlib import Path @@ -26,6 +29,11 @@ from hyperloom.common.io import atomic_write_json, safe_mtime from hyperloom.common.profile_args import sanitize_profile_server_args as _sanitize_profile_server_args from hyperloom.common.timeutil import now_iso +from hyperloom.common.agentx_mode import ( + managed_native_agentx_session, + native_agentx_optimization_session, + native_agentx_session, +) from hyperloom.inference_optimizer.session.paths import asset_root, mn_profile_trace_root from hyperloom.inference_optimizer import framework_registry from ._inferencex_patcher import ( @@ -74,7 +82,8 @@ def _trace_contains(path: Path, substring: str, max_bytes: int | None = None) -> carry = "" chunk_size = 4_000_000 try: - with gzip.open(path, "rt", encoding="utf-8", errors="replace") as fh: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8", errors="replace") as fh: while read < max_bytes: chunk = fh.read(chunk_size) if not chunk: @@ -91,9 +100,10 @@ def _trace_contains(path: Path, substring: str, max_bytes: int | None = None) -> def _sample_trace_text(path: Path) -> str | None: - """Read up to ``_TRACE_INSPECT_BYTES`` of decompressed text from a gzipped trace.""" + """Read a bounded sample from a plain or compressed trace.""" try: - with gzip.open(path, "rt", encoding="utf-8", errors="replace") as fh: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8", errors="replace") as fh: return fh.read(_TRACE_INSPECT_BYTES) except (OSError, EOFError, UnicodeDecodeError) as e: # Best-effort: a malformed sample must not fail the profile path. @@ -297,6 +307,7 @@ def _build_trace_validate( "probe_version": certificate.get("probe_version"), "probe_status": probe_status, "probe_error": str(probe_error or ""), + "workload_params": certificate.get("workload_params") or {}, "checked_at": now_iso(timespec="seconds"), "trace_dir": str(trace_dir), "framework": str(framework or ""), @@ -328,21 +339,29 @@ def _write_trace_certificate(trace_dir: Path, validate: dict[str, Any]) -> str: return str(target) -def _certify_trace_dir(trace_dir: Path, framework: str) -> dict[str, Any]: +def _certify_trace_dir( + trace_dir: Path, framework: str, *, workload_params: Mapping[str, Any] | None = None +) -> dict[str, Any]: """Run the capture-time self-certification probe over a profile trace.""" from hyperloom.orchestrator.trace_analysis import trace_selfcert # The workload parameters shape the split forecast, and reading them from the benchmark config keeps the # certificate independent of any analysis having run -- the point of certifying at capture time. - params = trace_selfcert.read_workload_params(trace_dir) - return trace_selfcert.certify_trace_dir( + params = trace_selfcert.read_workload_params(trace_dir) if workload_params is None else workload_params + settings = { + "num_steps": int(params.get("num_steps", trace_selfcert.DEFAULT_NUM_STEPS)), + "conc": int(params["conc"]) if params.get("conc") is not None else None, + "osl": float(params["osl"]) if params.get("osl") is not None else None, + "r": float(params.get("r", trace_selfcert.DEFAULT_R)), + } + resolved_framework = framework or str(params.get("framework") or "") + certificate = trace_selfcert.certify_trace_dir( trace_dir, - framework=framework or str(params.get("framework") or ""), - num_steps=params.get("num_steps", trace_selfcert.DEFAULT_NUM_STEPS), - conc=params.get("conc"), - osl=params.get("osl"), - r=params.get("r", trace_selfcert.DEFAULT_R), + framework=resolved_framework, + **settings, ) + certificate["workload_params"] = {"source": params.get("source"), "framework": resolved_framework, **settings} + return certificate def _validate_trace_structure( @@ -490,7 +509,11 @@ def _note_check( # --- Check 3 (Deval): main trace has user_annotation + execute_* --- execute_* annotations = InferenceX per-step # writes when detailed_annotations is honoured (distinct from check 5). main_traces = sorted( - (p for p in trace_dir.glob("*.trace.json.gz") if p.is_file()), + ( + p + for p in trace_dir.glob("*.trace.json*") + if p.is_file() and p.name.endswith((".trace.json", ".trace.json.gz")) + ), key=lambda p: p.stat().st_size, reverse=True, ) @@ -783,6 +806,11 @@ def _default_profile_config() -> Path: class ProfileExecutor(BenchmarkRunExecutor): """Benchmark round with the torch profiler on; extracts and certifies the trace_dir.""" + # The compatibility client is diagnostic evidence only. Baseline/grid + # materialization keeps the default False and therefore cannot silently + # downgrade a native AgentX measurement into this path. + allow_agentx_profile_compat = True + def __init__( self, *, @@ -814,6 +842,281 @@ def _resolve_default_config(self) -> Path: """Pick the profile yaml for $FRAMEWORK.""" return _default_profile_config() + def _agentx_profile_compatibility_error(self, shared_state: Any) -> str: + """Reject a generic trace whose TP would flatten native PP/PCP ranks.""" + if managed_native_agentx_session(shared_state): + return "" + config_path = str(getattr(shared_state, "baseline_config_path", "") or "").strip() + if not config_path: + return ( + "AgentX compatibility profiling requires an accepted native " + "baseline config with fingerprint-bound topology metadata." + ) + try: + cfg = yaml.safe_load(Path(config_path).read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError) as exc: + return f"Cannot read accepted AgentX baseline config {config_path!r}: {exc}" + bench = cfg.get("benchmark") if isinstance(cfg, dict) else {} + workload = bench.get("workload_spec") if isinstance(bench, dict) else {} + topology = workload.get("resolved_topology") if isinstance(workload, dict) else {} + if not isinstance(topology, dict): + return "Accepted AgentX baseline is missing fingerprint-bound workload_spec.resolved_topology metadata." + required = ("tp", "pp", "pcp_size", "gpu_count") + missing = [name for name in required if name not in topology] + if missing: + return f"Accepted AgentX topology is incomplete for compatibility profiling: missing {', '.join(missing)}." + try: + tp = int(topology["tp"]) + pp = int(topology["pp"]) + pcp = int(topology["pcp_size"]) + gpu_count = int(topology["gpu_count"]) + except (TypeError, ValueError): + return "Accepted AgentX topology contains non-integer rank counts." + if min(tp, pp, pcp, gpu_count) <= 0 or gpu_count != tp * pp * pcp: + return ( + "Accepted AgentX topology is inconsistent: expected " + f"gpu_count=TP*PP*PCP, got {gpu_count} vs {tp}*{pp}*{pcp}." + ) + if pp == 1 and pcp == 1: + return "" + return ( + "AgentX compatibility profiling cannot preserve the accepted " + f"native topology (PP={pp}, PCP={pcp}); a generic TP-only server " + "would produce a non-equivalent trace." + ) + + def _agentx_runtime_checkout( + self, + *, + config_path: Path, + output_dir: Path, + inferencex_path: str, + agentx_session: bool, + ) -> tuple[str, dict[str, Any] | None]: + """Clone a disposable checkout for the AgentX compatibility profile. + + This leg deploys ``aiperf_client.sh`` and applies profiler compatibility + patches. The accepted native baseline pins a clean InferenceX tree by + commit and executable hashes, so mutating that tree would invalidate + every subsequent measurement. A local, detached clone keeps the + diagnostic profile isolated without requiring network access. + """ + if not agentx_session: + return inferencex_path, None + from ._native_profile import managed_profile_benchmark + + configured = (yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}).get("benchmark") or {} + if managed_profile_benchmark(configured): + return str(configured.get("inferencex_path") or inferencex_path), None + try: + source = Path(inferencex_path).expanduser().resolve() + except (OSError, RuntimeError) as exc: + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"cannot resolve pinned AgentX InferenceX checkout: {exc}", + } + if not source.is_dir(): + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"pinned AgentX InferenceX checkout is missing: {source}", + } + + def run_git( + argv: list[str], + *, + timeout: int, + ) -> tuple[subprocess.CompletedProcess[str] | None, dict[str, Any] | None]: + try: + return ( + subprocess.run( + argv, + capture_output=True, + text=True, + timeout=timeout, + check=False, + ), + None, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + return None, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": (f"cannot prepare isolated AgentX profile checkout: {type(exc).__name__}: {exc}"), + } + + head_proc, git_error = run_git( + ["git", "-C", str(source), "rev-parse", "HEAD"], + timeout=30, + ) + if git_error is not None: + return inferencex_path, git_error + assert head_proc is not None + head = (head_proc.stdout or "").strip().lower() + if head_proc.returncode != 0 or not re.fullmatch(r"[0-9a-f]{40}", head): + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"cannot verify pinned AgentX InferenceX checkout at {source}", + } + expected = os.environ.get("INFERENCEX_REF", "").strip().lower() + if re.fullmatch(r"[0-9a-f]{7,40}", expected) and not (head.startswith(expected) or expected.startswith(head)): + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": (f"InferenceX HEAD {head} does not match pinned ref {expected}"), + } + + output_root = output_dir.expanduser().resolve() + isolated = output_root / ".agentx-profile-inferencex" + if isolated.is_symlink(): + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": (f"isolated InferenceX profile checkout must not be a symbolic link: {isolated}"), + } + try: + isolated_resolved = isolated.resolve(strict=False) + isolated_resolved.relative_to(output_root) + except (OSError, RuntimeError, ValueError) as exc: + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": (f"isolated InferenceX profile checkout escapes its output directory: {isolated} ({exc})"), + } + if isolated_resolved == source: + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": (f"isolated InferenceX profile checkout resolves to the pinned native checkout: {source}"), + } + if not isolated.exists(): + clone, git_error = run_git( + [ + "git", + "clone", + "--local", + "--no-hardlinks", + "--no-checkout", + "--no-tags", + str(source), + str(isolated_resolved), + ], + timeout=300, + ) + if git_error is not None: + return inferencex_path, git_error + assert clone is not None + if clone.returncode != 0: + detail = (clone.stderr or clone.stdout or "").strip() + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"cannot clone isolated InferenceX profile tree: {detail}", + } + checkout, git_error = run_git( + ["git", "-C", str(isolated_resolved), "checkout", "--detach", head], + timeout=120, + ) + if git_error is not None: + return inferencex_path, git_error + assert checkout is not None + if checkout.returncode != 0: + detail = (checkout.stderr or checkout.stdout or "").strip() + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"cannot checkout isolated InferenceX profile tree: {detail}", + } + top_level, git_error = run_git( + ["git", "-C", str(isolated_resolved), "rev-parse", "--show-toplevel"], + timeout=30, + ) + if git_error is not None: + return inferencex_path, git_error + assert top_level is not None + try: + isolated_top = Path((top_level.stdout or "").strip()).resolve(strict=True) + except (OSError, RuntimeError): + isolated_top = Path() + if top_level.returncode != 0 or isolated_top != isolated_resolved: + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": ( + "isolated InferenceX profile checkout is not its own git " + f"worktree: path={isolated_resolved}, top={isolated_top}" + ), + } + isolated_head, git_error = run_git( + ["git", "-C", str(isolated_resolved), "rev-parse", "HEAD"], + timeout=30, + ) + if git_error is not None: + return inferencex_path, git_error + assert isolated_head is not None + isolated_sha = (isolated_head.stdout or "").strip().lower() + if ( + isolated_head.returncode != 0 + or isolated_sha != head + or not (isolated_resolved / "benchmarks" / "benchmark_lib.sh").is_file() + ): + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": ( + "isolated InferenceX profile checkout is incomplete or at " + f"the wrong revision: path={isolated_resolved}, head={isolated_sha or ''}, " + f"expected={head}" + ), + } + status, git_error = run_git( + [ + "git", + "-C", + str(isolated_resolved), + "status", + "--porcelain=v1", + "--untracked-files=all", + ], + timeout=30, + ) + if git_error is not None: + return inferencex_path, git_error + assert status is not None + dirty = (status.stdout or "").strip() + if status.returncode != 0 or dirty: + detail = dirty or (status.stderr or "").strip() or "git status failed" + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": ( + "isolated InferenceX profile checkout is not clean; refusing " + f"to reuse stale profiler patches: {detail}" + ), + } + try: + cfg = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + bench = cfg.get("benchmark") if isinstance(cfg, dict) else None + if not isinstance(bench, dict): + raise ValueError("materialized profile config has no benchmark mapping") + bench["inferencex_path"] = str(isolated_resolved) + envs = bench.get("envs") + if isinstance(envs, dict) and "INFERENCEX_PATH" in envs: + envs["INFERENCEX_PATH"] = str(isolated_resolved) + config_path.write_text( + yaml.safe_dump(cfg, sort_keys=False), + encoding="utf-8", + ) + except (OSError, ValueError, yaml.YAMLError) as exc: + return inferencex_path, { + "status": "failed", + "error_class": "agentx_profile_checkout_unavailable", + "error": f"cannot bind profile config to isolated checkout: {exc}", + } + return str(isolated_resolved), None + def _resolve_mn_round_trace_root(self, ctx) -> str: """Return the shared torch-trace base dir for multi-node, or ''.""" from ._multi_node_env import is_multi_node, rayjob_id_from_state @@ -913,6 +1216,19 @@ def _after_materialize_config( output_dir: Path, ) -> dict[str, Any] | None: """Arm the host probe, then patch the InferenceX checkout Magpie will execute.""" + from ._native_profile import managed_profile_benchmark + + configured = (yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}).get("benchmark") or {} + if managed_profile_benchmark(configured): + self._host_probe_dir = "" + self._host_probe_status = "magpie_managed_diagnostic" + self._instrumentation_preflight = _check_row( + CHECK_INSTRUMENTATION_PREFLIGHT, + status="passed", + owner="magpie", + detailed_annotations=configured["profiler"]["torch_profiler"].get("detailed_annotations", False), + ) + return None try: self._host_probe_dir = self._inject_host_probe(config_path, output_dir) self._host_probe_status = "" @@ -1073,6 +1389,10 @@ def _collect_rewrite_evidence(self, result: dict[str, Any]) -> None: async def __call__(self, ctx) -> dict[str, Any]: """Run the profiling action for the given context.""" + extra = getattr(ctx, "extra", None) or {} + state = extra.get("shared_state") or self.shared_state + if managed_native_agentx_session(state): + return await self._managed_profile(ctx, state) # atom: the Magpie atom wrapper bridges PROFILE=1 to atom's --torch-profiler-dir and writes standard # *.pt.trace.json.gz, so the executor falls through to the sglang/vllm path. params = ctx.task.params or {} @@ -1115,6 +1435,28 @@ async def __call__(self, ctx) -> dict[str, Any]: from ._workload_envs import agentx_active agentx_session = agentx_active(shared_state) + if native_agentx_optimization_session(shared_state): + from ._native_profile import project_native_profile + + try: + await asyncio.to_thread(project_native_profile, params, shared_state) + params["extra_server_args"] = _sanitize_profile_server_args(str(params.get("extra_server_args") or "")) + except (OSError, ValueError, subprocess.SubprocessError) as exc: + return { + "status": "failed", + "error_class": "native_profile_launch_unverified", + "error": str(exc), + "trace_input_ready": False, + } + if native_agentx_session(shared_state): + compatibility_error = self._agentx_profile_compatibility_error(shared_state) + if compatibility_error: + return { + "status": "failed", + "error_class": "agentx_profile_topology_incompatible", + "error": compatibility_error, + "trace_input_ready": False, + } if not (params.get("output_dir") or extra.get("workspace")): output_dir = self._resolve_workspace(ctx, "profile") output_dir.mkdir(parents=True, exist_ok=True) @@ -1579,6 +1921,63 @@ def _safe_size(p: Path) -> int: log.warning("profile_executor: failed to write AgentX trace manifest: %s", exc) return result + async def _managed_profile(self, ctx, state: Any) -> dict[str, Any]: + """Keep Magpie's accepted server and validate each independent trace window.""" + from ._native_profile import managed_profile_workload_params, prepare_managed_profile + + params = ctx.task.params + if params is None: + params = ctx.task.params = {} + output_dir = self._resolve_workspace(ctx, "profile") + try: + await asyncio.to_thread(prepare_managed_profile, params, state, output_dir) + except (OSError, ValueError, TypeError, KeyError, yaml.YAMLError, subprocess.SubprocessError) as exc: + return { + "status": "failed", + "error_class": "native_profile_launch_unverified", + "error": str(exc), + "trace_input_ready": False, + } + params["output_dir"] = str(output_dir) + result = await super().__call__(ctx) + if result.get("status") != "succeeded" or not result.get("trace_input_ready"): + result["trace_input_ready"] = False + return result + config = yaml.safe_load(Path(result["materialized_config"]).read_text(encoding="utf-8")) + framework = str(config["benchmark"].get("framework") or "") + for capture in result["profile_rounds"]: + directory = Path(capture["trace_dir"]) + health = _validate_trace_structure(directory, framework) + certificate: dict[str, Any] = {} + probe_error = "" + try: + certificate = _certify_trace_dir( + directory, + framework, + workload_params=managed_profile_workload_params( + config["benchmark"], + config_path=Path(result["materialized_config"]), + capture=capture, + ), + ) + except (OSError, ValueError, TypeError, KeyError, ImportError) as exc: + probe_error = f"{type(exc).__name__}: {exc}" + validation = _build_trace_validate( + health, + trace_dir=directory, + framework=framework, + certificate=certificate, + probe_error=probe_error, + preflight=self._instrumentation_preflight, + ) + capture["trace_health"] = health + capture["trace_validate"] = validation + capture["trace_validate_path"] = _write_trace_certificate(directory, validation) + selected = result["profile_rounds"][-1] + for name in ("trace_health", "trace_validate", "trace_validate_path"): + result[name] = selected[name] + return result + profile_executor = ProfileExecutor() diff --git a/src/hyperloom/orchestrator/actions/executors/report.py b/src/hyperloom/orchestrator/actions/executors/report.py index 8abf3052b1..5c2c789b9a 100644 --- a/src/hyperloom/orchestrator/actions/executors/report.py +++ b/src/hyperloom/orchestrator/actions/executors/report.py @@ -485,6 +485,7 @@ def _build_summary_dict( session_dir: Path | None = None, ) -> dict[str, Any]: """Assemble the machine-readable session summary dict.""" + from hyperloom.common.agentx_mode import native_agentx_session from hyperloom.common.gain_math import gain_pct from ...state.shared_state import resolve_graded_comparison @@ -508,6 +509,7 @@ def _build_summary_dict( # Read back by the graded-axes section: the persisted AgentX marker outlives the shell, so a report rendered # from a resumed session still names the mode the run was graded under. "benchmark_mode": str(getattr(state, "benchmark_mode", "") or ""), + "native_agentx": native_agentx_session(state), "grading": dict(getattr(state, "grading", None) or {}), "baseline_accuracy": state.baseline_accuracy, "current_best": state.current_best, @@ -595,17 +597,14 @@ def _format_md(summary: dict[str, Any]) -> str: lines.append(f"- **Budget**: {summary['max_minutes']} minutes") lines.append(f"- **Generated**: {summary['report_generated_at']}") lines.append("") - # Per-framework primary metric: serving reports throughput (tok/s/GPU), scriptable xDiT reports per-image latency - # (e2el_mean_ms). - from hyperloom.inference_optimizer import framework_registry + from hyperloom.inference_optimizer.performance_display import format_session_metric - _fw = summary.get("framework") lines.append("## Throughput") lines.append("") - lines.append(f"- baseline : `{framework_registry.format_primary_metric(_fw, summary['baseline_tput'])}`") + lines.append(f"- baseline : `{format_session_metric(summary, summary['baseline_tput'])}`") if cb_tput is not None: lines.append( - f"- current_best : `{framework_registry.format_primary_metric(_fw, cb_tput)}` " + f"- current_best : `{format_session_metric(summary, cb_tput, measurement=cb)}` " f"(action=`{cb.get('action', '?')}`)" ) # Printed even when never validated, so the absence is stated, not implied. diff --git a/src/hyperloom/orchestrator/actions/executors/tests/test_baseline_eval_fallback.py b/src/hyperloom/orchestrator/actions/executors/tests/test_baseline_eval_fallback.py index d86636b286..9095802968 100644 --- a/src/hyperloom/orchestrator/actions/executors/tests/test_baseline_eval_fallback.py +++ b/src/hyperloom/orchestrator/actions/executors/tests/test_baseline_eval_fallback.py @@ -1292,6 +1292,7 @@ def test_an_mlperf_baseline_without_a_clean_inline_score_stops_the_session(tmp_p monkeypatch.setenv("HYPERLOOM_AGENTX", "1") monkeypatch.setenv("HYPERLOOM_AGENTIC_BACKEND", "mlperf") + monkeypatch.setattr("hyperloom.inference_optimizer.agentx.runtime.maybe_prepare_agentx", lambda **kwargs: True) base = tmp_path / "base.yaml" _write_yaml(base) @@ -1311,6 +1312,8 @@ def fake_run(cmd, *args, **kwargs): state, ) with patch("hyperloom.orchestrator.actions.executors.baseline.run_with_session_kill", side_effect=fake_run): - _run(executor(ctx)) + result = _run(executor(ctx)) + assert result["accuracy_task"] == "mlperf_agentic_inline" + assert result["accuracy"] == 0.0 assert state.stop_reason == "baseline_accuracy_failed" diff --git a/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_executor.py b/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_executor.py index 6e48eaaf80..99c0e283cd 100644 --- a/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_executor.py +++ b/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_executor.py @@ -482,6 +482,51 @@ async def test_executor_apply_only_succeeds(tmp_path: Path): assert (repo / "src.py").read_text().endswith("return 2\n") +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("recovery_phase", "expected_status", "expected_error"), + [ + (None, "skipped", "unsupported_upstream_launcher_hook"), + ("restored", "skipped", "unsupported_upstream_launcher_hook"), + ("applied", "failed", "integrate_restore_incomplete"), + ], +) +async def test_native_agentx_rejects_integrate_patch_before_mutation( + tmp_path: Path, recovery_phase, expected_status, expected_error +): + from types import SimpleNamespace + + session_dir = tmp_path / "session" + session_dir.mkdir() + repo = tmp_path / "framework" + init_git_repo(repo) + _write_specialist_workspace( + session_dir, + "t-agentx-spec", + patch_contents=[_VALID_PATCH], + ) + ctx = _make_ctx( + "t-agentx-integrate", + { + "specialist_task_id": "t-agentx-spec", + "framework_source_root": str(repo), + "apply_only": True, + }, + ) + ctx.extra["shared_state"] = SimpleNamespace( + benchmark_mode="agentx", + agentx_epoch=2, + pending_integrate={"recovery": {"phase": recovery_phase}} if recovery_phase else {}, + ) + + result = await IntegratePatchExecutor(session_dir=session_dir)(ctx) + + assert result["status"] == expected_status + assert result["error_class"] == expected_error + assert not result.get("patches_applied") + assert (repo / "src.py").read_text().endswith("return 1\n") + + @pytest.mark.asyncio @pytest.mark.parametrize("reuse_context", [False, True], ids=["new-context", "reused-context"]) async def test_same_executor_second_early_return_does_not_reuse_runtime(tmp_path, monkeypatch, reuse_context): diff --git a/src/hyperloom/orchestrator/kernel/conc_sweep.py b/src/hyperloom/orchestrator/kernel/conc_sweep.py index 63aae5238e..09849b84a8 100644 --- a/src/hyperloom/orchestrator/kernel/conc_sweep.py +++ b/src/hyperloom/orchestrator/kernel/conc_sweep.py @@ -174,6 +174,7 @@ def _point_from_variant(v: VariantResult, *, arm: str) -> dict[str, Any]: "output_throughput": v.output_throughput, "request_throughput": v.request_throughput, "total_token_throughput": total, + "agentx_gpu_count": v.agentx_gpu_count, "input_throughput": v.input_throughput, "e2e_norm_intvty_p90": v.intvty_p90, "tpot_p90_ms": v.tpot_p90_ms, @@ -224,6 +225,7 @@ def _write_csv(csv_path: Path, points: list[dict[str, Any]]) -> None: "output_throughput", "request_throughput", "total_token_throughput", + "agentx_gpu_count", "input_throughput", "e2e_norm_intvty_p90", "tpot_p90_ms", diff --git a/src/hyperloom/orchestrator/kernel/conc_sweep_plot.py b/src/hyperloom/orchestrator/kernel/conc_sweep_plot.py index 6037517e31..39020fee8b 100644 --- a/src/hyperloom/orchestrator/kernel/conc_sweep_plot.py +++ b/src/hyperloom/orchestrator/kernel/conc_sweep_plot.py @@ -27,9 +27,10 @@ def _agentx_xy(point: Mapping[str, Any], tp_eff: float) -> tuple[float, float] | """p90 interactivity against token throughput per chip.""" intvty = _positive(point.get("e2e_norm_intvty_p90")) total = _positive(point.get("total_token_throughput")) + recipe_gpus = _positive(point.get("agentx_gpu_count")) or tp_eff if intvty is None or total is None: return None - return intvty, total / tp_eff + return intvty, total / recipe_gpus def _synthetic_xy(point: Mapping[str, Any], tp_eff: float) -> tuple[float, float] | None: @@ -61,7 +62,7 @@ def _axes_for_metric(metric: str, tp_eff: float) -> _Axes: return _Axes( point_xy=_agentx_xy, x_label="P90 Interactivity (tok/s/user)", - y_label=f"Token Throughput per Chip (tok/s/chip, tp={int(tp_eff)})", + y_label="Token Throughput per Chip (tok/s/chip, resolved recipe topology)", ) return _Axes( point_xy=_synthetic_xy, diff --git a/src/hyperloom/orchestrator/kernel/request_handlers.py b/src/hyperloom/orchestrator/kernel/request_handlers.py index 80abec5d47..789e21b2d2 100644 --- a/src/hyperloom/orchestrator/kernel/request_handlers.py +++ b/src/hyperloom/orchestrator/kernel/request_handlers.py @@ -4775,6 +4775,30 @@ async def integrate_handler( # {kernel_id} payload isn't failed with a phantom "missing base_tput". payload = _fill_integrate_defaults_from_state(payload, session_dir=session_dir) + # Native AgentX is measurement-only at this revision. Its pinned + # InferenceX launchers own the complete server command and expose no + # fingerprinted optimizer hook, so applying a kernel/source mutation here + # would either benchmark an unchanged server or fail much later while + # materializing the native recipe. Refuse before touching the framework + # tree. The specialist integrate path enforces the same boundary. + state = SharedState.load_or_init(session_dir) + from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session + + if native_agentx_session(state) and not native_agentx_optimization_session(state): + return { + "status": "skipped", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "Native AgentX cannot benchmark kernel, source, or runtime " + "mutations until the pinned InferenceX launcher exposes a " + "fingerprinted optimizer hook." + ), + "decision": "NEEDS_REVIEW", + "kernel_id": payload.get("kernel_id"), + "patches_applied": [], + "patches_reverted": [], + } + if payload.get("_vendor_playbook_deploy_blocked"): # A vendor-playbook KEEP (e.g. mori dispatch/combine launch-config # tuning) has no deployable artifact: best_artifact_path is a copy of @@ -4833,7 +4857,6 @@ async def integrate_handler( if missing_inputs is not None: return missing_inputs - state = SharedState.load_or_init(session_dir) patch_path = payload.get("patch_path") kernel_id = payload.get("kernel_id") preapplied = payload.get("preapplied_apply_result") @@ -4938,6 +4961,16 @@ async def integrate_handler( "remove_args": to_str_list(payload.get("remove_args")), "unset_envs": to_str_list(payload.get("unset_envs")), "args_mode": str(payload.get("args_mode") or "append"), + **{ + key: payload[key] + for key in ( + "native_launch_overrides", + "base_native_launch_overrides", + "runtime_override", + "overlay_pythonpath", + ) + if payload.get(key) + }, # The only artifact that patches FlyDSL sources, so the only run that # needs the JIT cache key widened. "flydsl_source_dirs": (str(payload.get("artifact_kind") or "") == _FRAMEWORK_APPLYBACK_ARTIFACT_KIND), @@ -5019,6 +5052,12 @@ def _restore_aiter_rebuild_env() -> None: } try: + if native_agentx_optimization_session(state) and mode != "env_only": + from ..actions.executors._native_source import kernel_source_evidence + + evidence = kernel_source_evidence(apply_result) + fake_task.params["native_source_files"] = evidence["source_files"] + fake_task.params["native_absent_source_files"] = evidence["absent_source_files"] bench_result = await _run_integrate_rebaseline_with_lock_retry( baseline_executor, ctx, diff --git a/src/hyperloom/orchestrator/loop/intent_router.py b/src/hyperloom/orchestrator/loop/intent_router.py index 9d4a3e484d..ddc982648e 100644 --- a/src/hyperloom/orchestrator/loop/intent_router.py +++ b/src/hyperloom/orchestrator/loop/intent_router.py @@ -773,6 +773,10 @@ async def _apply_verdict_outcome( sid_candidate: str, ) -> None: """Materialise, deny, or send back a proposal according to its collapsed verdict.""" + from ..phases.geak_native_revalidation import record_review + + if record_review(self, pending, verdict=verdict, reasoning=reasoning): + return # Both `approve` and `advise` mean "dispatch may proceed"; treat them # identically for materialization. if verdict in ("approve", "advise"): diff --git a/src/hyperloom/orchestrator/loop/writeback.py b/src/hyperloom/orchestrator/loop/writeback.py index 6e60c6a71f..1c9bc3beac 100644 --- a/src/hyperloom/orchestrator/loop/writeback.py +++ b/src/hyperloom/orchestrator/loop/writeback.py @@ -287,6 +287,10 @@ def _source_layer_handles(result: Mapping[str, Any]) -> dict[str, Any]: } if "source_snapshot_complete" in result: handles["source_snapshot_complete"] = bool(result["source_snapshot_complete"]) + effective = result.get("effective_config") or {} + overlay = result.get("final_overlay") or result.get("overlay_pythonpath") or effective.get("final_overlay") + if overlay: + handles["final_overlay"] = str(overlay) return handles @@ -843,8 +847,9 @@ async def record_integrate_keep(self, result: dict[str, Any]) -> None: def is_promotable_result(self, task_kind: str, result: dict[str, Any]) -> bool: """Decide whether a settled task result should be promoted. - Per-kind rules: baseline/profile require a valid measurement, sweep - requires ``status == "succeeded"``, ``replay_warm_recipe`` always routes + Per-kind rules: baseline requires a valid measurement, profile also accepts + a successful diagnostic capture, sweep requires ``status == "succeeded"``, + ``replay_warm_recipe`` always routes through promotion (it owns its own failure bookkeeping), and everything else is promotable unless ``status == "failed"``. @@ -866,6 +871,8 @@ def is_promotable_result(self, task_kind: str, result: dict[str, Any]) -> bool: measurement_status = result.get("measurement_status") if measurement_status is not None and str(measurement_status) != "succeeded": return False + if result.get("diagnostic_only") is True: + return result.get("status") == "succeeded" and result.get("trace_input_ready") is True return is_valid_measurement(result) # replay_warm_recipe always routes through promote_warm_replay (owns its own failure bookkeeping). if task_kind == "replay_warm_recipe": @@ -1363,7 +1370,7 @@ def lift_to_current_best( ``True`` when the configuration was accepted, ``False`` when a performance winner was not comparable or did not beat the anchor. """ - from hyperloom.common.perf_metric import VERDICT_KEEP, graded_axes_of + from hyperloom.common.perf_metric import VERDICT_KEEP prebaseline_enablement = ( task_kind == "integrate_patch" @@ -1410,21 +1417,30 @@ def lift_to_current_best( graded.candidate, ) return False + from ..measurement.native_launch import native_launch_config + + native_config = native_launch_config(bv) + if native_config: + bv = {**bv, **native_config} previous = self.shared_state.current_best or {} base_args = "" - if isinstance(previous, dict): + if isinstance(previous, dict) and not native_config: base_args = strip_benchmark_harness_flags(previous.get("extra_server_args")) # An authored-kernel overlay stays active until another KEEP replaces it. _overlay = str((bv.get("final_overlay") if isinstance(bv, dict) else "") or "").strip() if not _overlay and isinstance(previous, dict): _overlay = str(previous.get("final_overlay") or "").strip() candidate_args = "" - if isinstance(bv, dict): + if native_config: + candidate_args = native_config["candidate_extra_server_args"] + elif isinstance(bv, dict): candidate_args = strip_benchmark_harness_flags( bv.get("candidate_extra_server_args") or bv.get("extra_server_args") ) full_args = "" - if isinstance(bv, dict): + if native_config: + full_args = native_config["extra_server_args"] + elif isinstance(bv, dict): full_args = strip_benchmark_harness_flags(bv.get("extra_server_args")) controls_effective = bool( isinstance(bv, dict) @@ -1435,12 +1451,12 @@ def lift_to_current_best( ) ) # Build cumulative launch args without double-stacking; helper dedupes repeated --flag pairs (last wins). - if controls_effective: + if not native_config and controls_effective: # Removal/replace winners publish their effective cumulative config # from ExploreExecutor. Prepending the prior current_best would # reintroduce flags the variant deliberately removed. full_args = dedupe_extra_server_args(full_args) - else: + elif not native_config: full_args = merge_cumulative_extra_server_args( base_args, candidate_args, @@ -1550,9 +1566,13 @@ def lift_to_current_best( if bv.get("task_id"): stack_entry["task_id"] = str(bv.get("task_id")) if bv.get("effective_extra_server_args"): - stack_entry["effective_extra_server_args"] = dedupe_extra_server_args( + stack_entry["effective_extra_server_args"] = native_config.get( + "effective_extra_server_args" + ) or dedupe_extra_server_args( strip_benchmark_harness_flags(bv.get("effective_extra_server_args")) ) + if native_config: + stack_entry["native_launch_overrides"] = native_config["native_launch_overrides"] # Stable filter label for "what kind of optimization" (backend / # param / env), so the stack can be sliced like the timeline. _stack_envs = dict(bv.get("extra_envs") or {}) if isinstance(bv, dict) else {} @@ -1632,6 +1652,8 @@ def lift_to_current_best( for _key in to_str_list(bv.get("unset_envs") if isinstance(bv, dict) else None): _merged_envs.pop(_key, None) _merged_envs.update(_new_envs) + if native_config: + _merged_envs = dict(native_config["extra_envs"]) current_best = { "action": task_kind, "tput": float(best_tput), @@ -1648,15 +1670,17 @@ def lift_to_current_best( # The axes of the measurement this KEEP was graded on. Without them the # next round's anchor has no snapshot, and the session degrades to # output grading permanently after the first KEEP. - current_best.update(graded_axes_of(cand_source)) + current_best.update(integrate_measurement_fields(cand_source)) + if native_config: + current_best["native_launch_overrides"] = native_config["native_launch_overrides"] if isinstance(bv, dict): for _ctrl_key in ("remove_args", "unset_envs", "args_mode"): if bv.get(_ctrl_key): current_best[_ctrl_key] = bv.get(_ctrl_key) if bv.get("effective_extra_server_args"): - current_best["effective_extra_server_args"] = dedupe_extra_server_args( - strip_benchmark_harness_flags(bv.get("effective_extra_server_args")) - ) + current_best["effective_extra_server_args"] = native_config.get( + "effective_extra_server_args" + ) or dedupe_extra_server_args(strip_benchmark_harness_flags(bv.get("effective_extra_server_args"))) if (bv.get("remove_args") or bv.get("unset_envs")) and not current_best.get("args_mode"): current_best["args_mode"] = "replace" self.shared_state.current_best = current_best @@ -1959,6 +1983,7 @@ async def _promote_baseline( "tpot_mean_ms": result.get("tpot_mean_ms"), "input_throughput": result.get("input_throughput"), "total_throughput": result.get("total_token_throughput"), + "agentx_gpu_count": result.get("agentx_gpu_count"), "tpot_p90_ms": result.get("tpot_p90_ms"), "e2e_norm_intvty_p90": result.get("e2e_norm_intvty_p90"), "e2e_norm_intvty_p50": result.get("e2e_norm_intvty_p50"), @@ -1973,6 +1998,7 @@ async def _promote_baseline( current_best["e2e_norm_intvty_p90"] = snap["e2e_norm_intvty_p90"] for _axis in ( "input_throughput", + "agentx_gpu_count", "tpot_p90_ms", "e2e_norm_intvty_p50", "duration_seconds", @@ -3368,6 +3394,7 @@ def _stamp_current_best_measurement(self, evidence: Mapping[str, Any] | None = N launch_evidence = evidence.get("launch_evidence") or {} launch_evidence = dict(launch_evidence) if isinstance(launch_evidence, Mapping) else {} measurement = { + **integrate_measurement_fields(evidence), "schema_version": 2, "tput": float(cb.get("tput") or 0.0), "benchmark_workspace": str( @@ -3429,7 +3456,7 @@ def current_best_launch_config(self) -> dict[str, Any]: args = merge_cumulative_extra_server_args(args, token, "") else: envs[name] = str(value) - return { + config = { "extra_server_args": args, "extra_envs": envs, "remove_args": to_str_list(cb.get("remove_args")), @@ -3437,6 +3464,9 @@ def current_best_launch_config(self) -> dict[str, Any]: "args_mode": "replace" if str(cb.get("args_mode") or "").strip().lower() == "replace" else "append", "final_overlay": str(cb.get("final_overlay") or "").strip(), } + if isinstance(cb.get("native_launch_overrides"), dict): + config["native_launch_overrides"] = cb["native_launch_overrides"] + return config def build_env_spec( self, @@ -4290,6 +4320,19 @@ def geak_rebench_params(self, *, reason: str) -> dict[str, Any]: ``"fallback": "geak_harness"`` when only GEAK's own harness can deploy the candidate. """ + from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session + + if native_agentx_session(self.shared_state) and not native_agentx_optimization_session(self.shared_state): + # The native launcher has no optimizer-argv/overlay hook. A GEAK + # rebench would therefore enqueue a grid candidate that can never + # be represented by the canonical AgentX harness. Refuse here as + # well as at KERNEL dispatch so resumed/stale handbacks cannot + # create an un-runnable explore task. + return { + "skipped": True, + "reason": "unsupported_upstream_launcher_hook", + } + benchmark_script = self.shared_state.accepted_baseline_script() recipe_generation = int(self.shared_state.working_recipe_generation or 0) ps = self.shared_state.geak_result if isinstance(self.shared_state.geak_result, dict) else {} @@ -4325,6 +4368,31 @@ def geak_rebench_params(self, *, reason: str) -> dict[str, Any]: from ..actions.executors._proposal_identity import effective_fingerprint + if native_agentx_optimization_session(self.shared_state): + from ..phases.geak_native_revalidation import prepare_source_revalidation + from ..phases.geak_rebench import geak_candidate_matches + + pending_source = (self.shared_state.geak_pending or {}).get("native_review") or {} + if pending_source.get("params") and geak_candidate_matches(pending_source.get("candidate"), ps): + return dict(pending_source["params"]) + source_params = { + "reason": reason, + "extra_server_args": ps_flags, + "extra_envs": dict(ps_envs), + **ps_controls, + **stack_base_params({**(self.shared_state.current_best or {}), **self.current_best_launch_config()}), + "config_path": self.shared_state.baseline_config_path, + "framework": self.shared_state.framework, + "model_path": self.shared_state.model_path, + } + self._coord.proposals.inject_explore_runtime_params(source_params) + try: + source_params = prepare_source_revalidation(self.session_dir, ps, source_params, overlay=ps_overlay) + except (OSError, ValueError) as exc: + return {"skipped": True, "reason": "geak_native_source_invalid", "error": str(exc)} + if source_params is not None: + return source_params + # An overlay that cannot load installs nothing, so the measured delta would belong to the flags alone; drop it # before the run so the row cannot be read as a kernel win. overlay_requested = bool(ps_overlay) diff --git a/src/hyperloom/orchestrator/measurement/integrate_performance.py b/src/hyperloom/orchestrator/measurement/integrate_performance.py index 5b32437679..8e417587d7 100644 --- a/src/hyperloom/orchestrator/measurement/integrate_performance.py +++ b/src/hyperloom/orchestrator/measurement/integrate_performance.py @@ -105,6 +105,13 @@ def integrate_measurement_fields(measurement: Mapping[str, Any]) -> dict[str, An "launch_evidence", "launch_evidence_path", "server_log_path", + "agentx_server_launch", + "valid_measurement", + "benchmark_valid", + "publishable", + "submission_valid", + "native_agentx_protocol_valid", + "native_agentx_protocol_errors", ) if key in measurement }, diff --git a/src/hyperloom/orchestrator/measurement/native_launch.py b/src/hyperloom/orchestrator/measurement/native_launch.py new file mode 100644 index 0000000000..fb42de9267 --- /dev/null +++ b/src/hyperloom/orchestrator/measurement/native_launch.py @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Recover the exact native launch snapshot that produced an accepted measurement.""" + +from __future__ import annotations + +import copy +import shlex +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import yaml + + +def native_launch_config(measurement: Mapping[str, Any]) -> dict[str, Any]: + """Retain tokenized launch controls; display strings never become replay identity.""" + if measurement.get("agentx_launch_contract") != 1: + return {} + from hyperloom.inference_optimizer.agentx.identity import has_launch_contract, validate_server_launch + + path = str(measurement.get("materialized_config") or "") + if not path: + raise ValueError("Native KEEP requires the actual materialized candidate configuration") + config = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + benchmark = config.get("benchmark") if isinstance(config, Mapping) else None + if not isinstance(benchmark, Mapping) or not has_launch_contract(benchmark): + raise ValueError("Native KEEP configuration lacks the optimization launch contract") + errors = validate_server_launch(benchmark, measurement.get("agentx_server_launch")) + if errors: + raise ValueError(f"Native KEEP configuration does not match its measured launch: {', '.join(errors)}") + snapshot = copy.deepcopy(benchmark["agentx"]["launch_overrides"]) + args = shlex.join(snapshot.get("append_args") or []) + return { + "native_launch_overrides": snapshot, + "extra_server_args": args, + "candidate_extra_server_args": args, + "effective_extra_server_args": args, + "extra_envs": dict(snapshot.get("env") or {}), + "remove_args": list(snapshot.get("remove_args") or []), + "unset_envs": list(snapshot.get("unset_env") or []), + "args_mode": "replace" if snapshot.get("replace_args") else "append", + } diff --git a/src/hyperloom/orchestrator/phases/geak_native_revalidation.py b/src/hyperloom/orchestrator/phases/geak_native_revalidation.py new file mode 100644 index 0000000000..ee1cb2cf1d --- /dev/null +++ b/src/hyperloom/orchestrator/phases/geak_native_revalidation.py @@ -0,0 +1,294 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Review GEAK source products and measure them inside the kernel task's lease.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import time +import uuid +from pathlib import Path +from typing import Any + +from hyperloom.common.io import atomic_write_json +from hyperloom.inference_optimizer.session.session_paths import runs_dir +from hyperloom.orchestrator.bus.message_bus import Message +from hyperloom.orchestrator.kernel.geak_config import ( + _geak_overlay_is_loadable, + geak_spec_is_env, +) +from hyperloom.orchestrator.state.task_registry import Task + +ORIGIN = "geak_native_revalidation" + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def overlay_source_files(overlay: str) -> dict[str, str]: + """Use the native launch's source evidence for every deployable overlay file.""" + from hyperloom.orchestrator.actions.executors._native_source import source_file_hashes + + if not overlay: + return {} + return source_file_hashes( + sorted(p for p in Path(overlay).rglob("*") if p.is_file() and "__pycache__" not in p.parts) + ) + + +def _patch_sources(result: dict[str, Any], *, overlay: str) -> list[Path]: + combined = str(result.get("final_patch") or "").strip() + kernels = [ + spec + for spec in (result.get("accepted_kernels") or []) + (result.get("accepted_heads") or []) + if isinstance(spec, dict) and not geak_spec_is_env(spec) + ] + if ( + not combined + and not overlay + and any(not (spec.get("final_patch") or spec.get("patch_path")) for spec in kernels) + ): + raise ValueError("GEAK accepted kernel has no deployable source patch or overlay") + raw = ( + [combined] + if combined + else [str(spec.get("final_patch") or spec.get("patch_path") or "").strip() for spec in kernels] + ) + root = Path(str(result.get("eval_dir") or "")).expanduser().resolve() + paths = [] + for value in dict.fromkeys(item for item in raw if item): + path = Path(value).expanduser() + path = (path if path.is_absolute() else root / path).resolve() + if not path.is_relative_to(root) or not path.is_file() or not path.read_bytes().strip(): + raise ValueError(f"GEAK source patch is missing, empty, or outside its eval directory: {path}") + paths.append(path) + return paths + + +def prepare_source_revalidation( + session_dir: Path, result: dict[str, Any], params: dict[str, Any], *, overlay: str +) -> dict[str, Any] | None: + """Stage declared source patches; refuse incomplete products instead of dropping a layer.""" + if overlay and not _geak_overlay_is_loadable(overlay): + raise ValueError("GEAK source candidate declares an unloadable overlay") + patches = _patch_sources(result, overlay=overlay) + if not patches: + if not overlay and not any( + params.get(name) for name in ("extra_server_args", "extra_envs", "remove_args", "unset_envs", "args_mode") + ): + raise ValueError("GEAK accepted kernels have no deployable source patch or overlay") + return None + identity = { + "patches": {str(path): _sha256(path) for path in patches}, + "launch": params, + "overlay": overlay, + "overlay_files": overlay_source_files(overlay), + } + digest = hashlib.sha256(json.dumps(identity, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + sid = f"geak-native-{digest[:24]}" + workspace = runs_dir(session_dir, "specialist", sid) + workspace.mkdir(parents=True, exist_ok=True) + staged = [] + for index, path in enumerate(patches): + destination = workspace / f"{index:03d}-{path.name}" + content = path.read_bytes() + if hashlib.sha256(content).hexdigest() != identity["patches"][str(path)]: + raise ValueError(f"GEAK source patch changed while staging: {path}") + destination.write_bytes(content) + staged.append(str(destination)) + params = { + **params, + "origin": ORIGIN, + "specialist_task_id": sid, + "patches": staged, + "source_phase": "KERNEL_AGENT", + "domain": "kernel", + "provenance": ORIGIN, + "overlay_pythonpath": overlay, + "native_geak_artifact_digest": digest, + "native_geak_files": {path: _sha256(Path(path)) for path in staged}, + "native_geak_original_files": identity["patches"], + "native_geak_overlay_files": identity["overlay_files"], + } + atomic_write_json( + workspace / "specialist_done.json", + { + "patches_written": staged, + "source_phase": "KERNEL_AGENT", + "domain": "kernel", + "proposal_set": [ + { + "name": "GEAK source candidate", + "extra_args": params.get("extra_server_args", ""), + "extra_envs": params.get("extra_envs", {}), + } + ], + }, + ) + return params + + +def record_review(coord: Any, pending: Any, *, verdict: str, reasoning: str) -> bool: + """Persist only an actual Critic verdict for the covered GEAK review in flight.""" + params = (pending.payload or {}).get("params") or {} + if params.get("origin") != ORIGIN: + return False + saved = dict((coord.shared_state.geak_pending or {}).get("native_review") or {}) + if saved.get("proposal_msg_id") != pending.proposal_msg_id or saved.get("artifact_digest") != params.get( + "native_geak_artifact_digest" + ): + return True + saved.update(verdict=verdict, reasoning=reasoning) + coord.shared_state.geak_pending = {**coord.shared_state.geak_pending, "native_review": saved} + if verdict in {"approve", "advise"}: + coord.shared_state.record_specialist_patch_verdict(str(params["specialist_task_id"]), verdict) + coord.shared_state.save(coord.session_dir) + return True + + +def _verify_artifacts(params: dict[str, Any]) -> None: + for filename, expected in params["native_geak_files"].items(): + path = Path(filename) + if not path.is_file() or _sha256(path) != expected: + raise ValueError(f"reviewed GEAK source patch changed or disappeared: {path}") + overlay = str(params.get("overlay_pythonpath") or "") + if overlay and overlay_source_files(overlay) != params.get("native_geak_overlay_files"): + raise ValueError("reviewed GEAK overlay changed or disappeared") + + +async def _review(coord: Any, params: dict[str, Any]) -> dict[str, Any]: + from hyperloom.orchestrator.loop.proposals import PendingProposal + + digest = params["native_geak_artifact_digest"] + saved = dict((coord.shared_state.geak_pending or {}).get("native_review") or {}) + if saved.get("artifact_digest") == digest and saved.get("verdict") in {"approve", "advise"}: + return saved + payload = { + "action_name": "integrate", + "predicted_gain_pct": 0.0, + "params": {**params, "covered_executor": "integrate_patch"}, + "rationale": "Review GEAK source patches and their complete serving configuration before canonical native AgentX validation; GEAK proxy scores are not acceptance evidence.", + } + message = Message.new("coordinator", "*", "proposal", {**payload, "needs_review": True}) + saved = { + "artifact_digest": digest, + "proposal_msg_id": message.msg_id, + "verdict": "pending", + "task_id": uuid.uuid4().hex, + "params": params, + "candidate": dict(coord.shared_state.geak_result), + } + coord.shared_state.geak_pending = { + **(coord.shared_state.geak_pending or {}), + "status": "awaiting_source_review", + "native_review": saved, + } + coord.shared_state.save(coord.session_dir) + coord.state.pending_proposals[message.msg_id] = PendingProposal( + message.msg_id, "coordinator", "integrate", 0.0, payload + ) + await coord.bus.append_and_seq(message) + remaining = coord.shared_state.remaining_minutes() + deadline = time.monotonic() + min(300.0, remaining * 60.0 if remaining is not None else 300.0) + while time.monotonic() < deadline: + saved = dict((coord.shared_state.geak_pending or {}).get("native_review") or {}) + if saved.get("verdict") != "pending": + return saved + await asyncio.sleep(0.25) + saved.update(verdict="timeout", reasoning="Critic review did not complete within the revalidation window") + return saved + + +def _finish(coord: Any, review: dict[str, Any], *, status: str, error: str = "", measurement: Any = None) -> None: + coord.shared_state.geak_result = { + **coord.shared_state.geak_result, + "revalidation_status": status, + "revalidation_error": error, + "native_source_revalidation": dict(review, status=status), + "canonical_revalidation": measurement, + } + coord.shared_state.geak_pending = {} + coord.shared_state.save(coord.session_dir) + + +def finish_invalid_source(coord: Any, error: str) -> None: + """Settle an undeployable product without invoking GEAK's proxy harness.""" + _finish(coord, {}, status="fallback_failed", error=error) + + +def _record_canonical_adoption(coord: Any, params: dict[str, Any], measurement: dict[str, Any]) -> None: + from hyperloom.orchestrator.state.shared_state import resolve_graded_comparison + + graded = resolve_graded_comparison(coord.shared_state, measurement, against_baseline=True) + if graded.comparable: + coord.phase_kernel._record_geak_adopted_kernels( + coord.shared_state.geak_result, + measured_tput=graded.candidate, + baseline_tput=graded.reference, + provenance="native_source_canonical_revalidation", + overlay_loaded=bool(params.get("overlay_pythonpath")), + source_applied=True, + ) + + +async def revalidate_source(coord: Any, params: dict[str, Any]) -> None: + """Use real review and the existing transactional integration and promotion paths.""" + try: + _verify_artifacts(params) + review = await _review(coord, params) + if review.get("verdict") not in {"approve", "advise"}: + _finish( + coord, + review, + status="no_promote", + error=str(review.get("reasoning") or "Critic declined source integration"), + ) + return + _verify_artifacts(params) + except (OSError, ValueError) as exc: + _finish(coord, {}, status="fallback_failed", error=str(exc)) + return + task = Task( + task_id=review["task_id"], + kind="integrate_patch", + state="running", + params=params, + idempotency_key=f"geak-source-{params['native_geak_artifact_digest']}", + ) + if any( + row.get("task_id") == task.task_id + for row in (coord.shared_state.optimization_stack or []) + if isinstance(row, dict) + ): + stack = coord.shared_state.optimization_stack + if stack[-1].get("task_id") == task.task_id: + _record_canonical_adoption(coord, params, coord.shared_state.current_best) + _finish(coord, review, status="validated", measurement=coord.shared_state.current_best) + return + coord.shared_state.geak_pending = { + **coord.shared_state.geak_pending, + "status": "source_rebench_running", + "revalidation_task_id": task.task_id, + } + coord.shared_state.save(coord.session_dir) + result = await coord.sub.execute_covered(task) + measurement = result.get("bench_result") or result + stack_before = len(coord.shared_state.optimization_stack or []) + if coord.writeback.is_promotable_result("integrate_patch", result): + await coord.writeback.promote_to_shared_state("integrate_patch", result, task=task) + else: + await coord.writeback.handle_unpromotable_result(task, result) + kept = result.get("status") == "kept" and len(coord.shared_state.optimization_stack or []) > stack_before + if kept: + _record_canonical_adoption(coord, params, measurement) + _finish( + coord, + review, + status="validated" if kept else "no_promote", + error=str(result.get("reason") or ""), + measurement=measurement, + ) diff --git a/src/hyperloom/orchestrator/phases/geak_rebench.py b/src/hyperloom/orchestrator/phases/geak_rebench.py index ef177889b7..6192723d79 100644 --- a/src/hyperloom/orchestrator/phases/geak_rebench.py +++ b/src/hyperloom/orchestrator/phases/geak_rebench.py @@ -18,6 +18,8 @@ "revalidation_error", "revalidation_error_class", "revalidation_blocked_overlay", + "native_source_revalidation", + "canonical_revalidation", } ) @@ -50,6 +52,8 @@ def geak_verdict_is_terminal(persisted: Any) -> bool: """ prev = persisted if isinstance(persisted, dict) else {} status = str(prev.get("revalidation_status") or "") + if status == "validated" and (prev.get("native_source_revalidation") or {}).get("status") == "validated": + return True return status in _TERMINAL_REVALIDATION_STATUSES or ( status == "fallback_failed" and str(prev.get("revalidation_error_class") or "") == INCOMPARABLE_REVALIDATION ) @@ -79,6 +83,29 @@ def geak_candidate_is_adjudicated(persisted: Any, recovered: Any, *, harness_can return str(prev.get("revalidation_status") or "") in _TERMINAL_REVALIDATION_STATUSES if not geak_verdict_is_terminal(prev): return False + source_review = prev.get("native_source_revalidation") or {} + if source_review: + import hashlib + from pathlib import Path + + from .geak_native_revalidation import overlay_source_files + + params = source_review.get("params") or {} + files = params.get("native_geak_original_files") or {} + for filename, expected in files.items(): + try: + actual = hashlib.sha256(Path(filename).read_bytes()).hexdigest() + except OSError: + return False + if actual != expected: + return False + overlay = str(params.get("overlay_pythonpath") or "") + if overlay: + try: + if overlay_source_files(overlay) != params.get("native_geak_overlay_files"): + return False + except (OSError, ValueError): + return False blocked_overlay = str(prev.get("revalidation_blocked_overlay") or "") if blocked_overlay: from ..kernel.geak_config import _geak_overlay_is_loadable, _normalize_geak_overlay_dir diff --git a/src/hyperloom/orchestrator/phases/kernel.py b/src/hyperloom/orchestrator/phases/kernel.py index eb4297985c..7b14337dbb 100644 --- a/src/hyperloom/orchestrator/phases/kernel.py +++ b/src/hyperloom/orchestrator/phases/kernel.py @@ -1093,44 +1093,12 @@ def _observed_replay_shape(self, target_tput: float) -> dict[str, Any]: @staticmethod def _resolve_launch_server_script(bench: Mapping[str, Any]) -> str: - """Name the server-phase script GEAK should launch through Magpie. - - GEAK infers its launcher from the recipe's ``benchmark_script``, which on - every non-AgentX run IS a server launcher. The AgentX switch replaces - that field with the aiperf client -- which boots a server, replays the - corpus for ``AGENTX_DURATION``, then tears the server down in its exit - trap. Run under ``MAGPIE_RUN_PHASE=server`` it therefore returns no pid, - and GEAK's bench aborts before it measures a single repeat. - - Naming the builtin the client itself delegates to keeps GEAK on the - Magpie launch path -- the reason that launcher exists, since the platform - kernel preset, ``--trust-remote-code`` and the gpu-mem-util default are - not flags and so cannot be recovered from the accepted-flags handoff -- - while letting its own bench repeats run again. - - Resolution mirrors ``aiperf_client.sh``: the same ``AGENTX_SERVER_SCRIPT`` - override, the same ``{framework}_{gpu}.sh`` fallback, the same - ``/benchmarks/`` directory and deliberately no recursive - search, so what we advertise is the path that would have booted the - server rather than merely a plausible one. Recipe-recorded values beat - the ambient env because the recipe is the record of what actually ran. - - Returns "" for any non-AgentX recipe -- there GEAK's own derivation names - the script that really launched the baseline, which is strictly better - than anything re-derived here -- and "" whenever the builtin cannot be - confirmed on disk, leaving current behaviour untouched. Never raises. - """ - try: - from hyperloom.common.agentx_workload import is_agentx_client_script + """Resolve the server launcher named by a legacy AgentX client recipe.""" + from hyperloom.common.agentx_workload import is_agentx_client_script - # Only the AgentX client misleads the inference; anything else in - # this field is the launcher GEAK should keep deriving for itself. - if not is_agentx_client_script(str(bench.get("benchmark_script") or "").strip()): - return "" - return resolve_launch_server_script(bench) - except Exception: - log.warning("launch_server_script: could not resolve from the recipe", exc_info=True) + if not is_agentx_client_script(str(bench.get("benchmark_script") or "").strip()): return "" + return resolve_launch_server_script(bench) def _geak_timeouts(self) -> tuple[int, int, bool]: """Resolve the GEAK e2e timeouts from the live run budget.""" @@ -1182,6 +1150,7 @@ async def _run_geak_kernel_phase(self, *, from_phase: str) -> None: """Delegate the KERNEL_AGENT phase to GEAK (one whole-pipeline e2e run).""" state = self.shared_state from hyperloom.common.perf_metric import is_agentx_mode + from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session from ..actions.executors._workload_envs import agentx_enabled benchmark_mode = str(state.benchmark_mode or "").strip() @@ -1222,6 +1191,35 @@ def _finish_skip( ) state.save(self.session_dir) + if native_agentx_session(state) and not native_agentx_optimization_session(state): + # The pinned native launcher exposes neither Hyperloom nor GEAK an + # optimizer-argv hook. Running the proxy here can consume the full + # KERNEL_AGENT budget, but every resulting candidate is + # unpromotable by the canonical AgentX revalidation path. Skip + # before dispatch instead of doing work that can never be kept. + recorder = self.timeline() + if recorder is not None: + recorder.finish_failed( + stage="geak_dispatch", + error_class="unsupported_upstream_launcher_hook", + message=( + "native AgentX kernel optimization is unavailable until " + "InferenceX exposes a fingerprinted optimizer-argv hook" + ), + ) + _finish_skip( + { + "status": "skipped", + "error_class": "unsupported_upstream_launcher_hook", + "error": ( + "native AgentX kernel optimization is unavailable until " + "InferenceX exposes a fingerprinted optimizer-argv hook" + ), + }, + record_delegation=False, + ) + return + cb = state.current_best or {} try: env_spec = self._coord.writeback.build_env_spec() @@ -1428,9 +1426,19 @@ def _finish_skip( if env_spec: handoff["baseline_env_spec"] = env_spec if agentx: - # The saved recipe names aiperf_client.sh, not a server launcher. handoff["bench_launcher"] = "native" log.info("GEAK results remain proposal proxies; canonical AgentX validation remains in Hyperloom.") + if native_agentx_optimization_session(state): + from hyperloom.inference_optimizer.agentx.geak_proxy import seed_native_geak_proxy + + try: + seed_native_geak_proxy(handoff, benchmark=recipe_bench, measurement=measurement) + except ValueError as exc: + _finish_skip( + {"status": "error", "error_class": "invalid_native_launch_evidence", "error": str(exc)}, + record_delegation=False, + ) + return out_dir = self.session_dir / "geak" out_dir.mkdir(parents=True, exist_ok=True) @@ -1561,12 +1569,8 @@ def _promote_recovered_result( # to flush result.json), then SIGKILL, instead of orphaning run_e2e + its servers. term_grace = env_int("GEAK_TERM_GRACE_S", default=180) - # GEAK measures whatever axis Hyperloom grades on. An agentic replay is - # graded on total token throughput, so leaving this pinned to output aims - # GEAK's search at a number the session does not score -- on the AgentX - # corpus the two run ~140x apart, and a kernel that helps the decode-side - # output figure need not help the prefill-dominated total by the same - # margin. Synthetic runs resolve to "output" and are unaffected. + # Persisted grading determines the axis even when the session's metric + # override is absent from this process's environment. _geak_e2e_metric, _ = geak_metric_axis( benchmark_mode=str(state.benchmark_mode or ""), grading=state.grading or None, @@ -1785,6 +1789,14 @@ async def _revalidate_geak_candidate(self, *, reason: str) -> None: """ state = self.shared_state params = self._coord.writeback.geak_rebench_params(reason=reason) + from .geak_native_revalidation import ORIGIN, finish_invalid_source, revalidate_source + + if params.get("origin") == ORIGIN: + await revalidate_source(self._coord, params) + return + if params.get("reason") == "geak_native_source_invalid": + finish_invalid_source(self._coord, str(params.get("error") or "GEAK source artifacts are unavailable")) + return skip_reason = params.get("reason") if params.get("skipped") else None if skip_reason == "geak_invalid_config": return @@ -1859,6 +1871,13 @@ def _record_geak_candidate(self, result: dict[str, Any]) -> bool: if not isinstance(result, dict): return False result.setdefault("kernel_event_id", kernel_event_id(int(self.shared_state.macro_cycle or 0))) + prior_pending = self.shared_state.geak_pending or {} + prior_review = prior_pending.get("native_review") or {} + preserve_review = bool(prior_review) and _geak_rebench.geak_candidate_matches( + prior_review.get("candidate"), result + ) + if prior_review and not preserve_review: + self.shared_state.geak_pending = {} try: accepted_flags, parsed_envs = self._parse_geak_accepted_config(result) except ValueError as exc: @@ -1922,6 +1941,12 @@ def _record_geak_candidate(self, result: dict[str, Any]) -> bool: }, "ts": datetime.now(timezone.utc).isoformat(), } + if preserve_review: + self.shared_state.geak_pending.update( + native_review=prior_review, + status=prior_pending.get("status", "awaiting_rebench"), + revalidation_task_id=prior_pending.get("revalidation_task_id", ""), + ) recorder = self.timeline() if recorder is not None: recorder.record_geak_claim( @@ -2101,6 +2126,9 @@ def promote_geak_from_candidate( "workspace": result.get("eval_dir"), } if isinstance(measurement_provenance, Mapping): + from ..measurement.integrate_performance import integrate_measurement_fields + + promotion_measurement.update(integrate_measurement_fields(measurement_provenance)) for key in ( "extra_server_args", "effective_extra_server_args", @@ -2259,6 +2287,7 @@ def _record_geak_adopted_kernels( baseline_tput: float, provenance: str, overlay_loaded: bool | None, + source_applied: bool = False, ) -> None: """Write one adoption row per accepted GEAK kernel.""" if not isinstance(result, dict): @@ -2279,7 +2308,7 @@ def _record_geak_adopted_kernels( if baseline_tput > 0 and measured_tput > 0: rebench_gain = (measured_tput - baseline_tput) / baseline_tput * 100.0 # One kernel, overlay proven loaded, one measured number: the gain is attributable. - attributable = bool(overlay_loaded) and len(rows) == 1 + attributable = bool(overlay_loaded or source_applied) and len(rows) == 1 am = result.get("alignment_metrics") or {} basis = str(am.get("final_basis") or result.get("final_throughput_basis") or "") alignment_status = str((result.get("baseline_alignment") or {}).get("status") or "") @@ -2326,6 +2355,7 @@ def _record_geak_adopted_kernels( "last_status": attempt_status, "validated": attributable, "overlay_loaded": bool(overlay_loaded), + **({"source_applied": True} if source_applied else {}), "basis": basis, "alignment_status": alignment_status, # GEAK's own same-config A/B, kept beside the orchestrator number so the two are never confused diff --git a/src/hyperloom/orchestrator/phases/prelude.py b/src/hyperloom/orchestrator/phases/prelude.py index 3842a11414..2b08fc3083 100644 --- a/src/hyperloom/orchestrator/phases/prelude.py +++ b/src/hyperloom/orchestrator/phases/prelude.py @@ -2238,6 +2238,14 @@ def _settle_warm_replay( keep_threshold = _phase_state.resolve_keep_threshold(state) graded = resolve_graded_comparison(state, result, keep_threshold_pct=keep_threshold, anchor_tput=baseline_tput) reproduced = graded.verdict == VERDICT_KEEP + from hyperloom.common.agentx_mode import native_agentx_optimization_session + + if native_agentx_optimization_session(state): + from hyperloom.common.gain_math import gain_pct + + outcome["graded_objective"] = graded.objective + outcome["graded_gain_pct"] = gain_pct(graded.candidate, graded.reference) + outcome["graded_rejection"] = graded.degrade_reason outcome["keep_threshold_pct"] = keep_threshold if graded.veto_reason: # The grading applies the session's latency ceiling, so the drift branch below rolls the replay back @@ -2374,6 +2382,17 @@ def _settle_warm_replay( "replay_warm_recipe", float(single_round_tput), { + **{ + key: result[key] + for key in ( + "materialized_config", + "agentx_server_launch", + "final_overlay", + "submission_valid", + "native_agentx_protocol_valid", + ) + if key in result + }, "name": "warm_replay", **graded_axes_of(result), # The latency budget grades on this and fails closed without it. @@ -2405,8 +2424,10 @@ def _settle_warm_replay( ) if promoted_checkout: outcome["active_framework_root"] = promoted_checkout - # Resume re-points $INFERENCEX_PATH at this checkout, and stops the run when it has since vanished. - state.active_inferencex_path = promoted_checkout + from hyperloom.common.agentx_mode import native_agentx_session + + if not native_agentx_session(state): + state.active_inferencex_path = promoted_checkout outcome["status"] = "reproduced" outcome.pop("replayed_patch_refs", None) if replayed_patch_refs: @@ -2463,7 +2484,9 @@ def _settle_warm_replay( task, outcome, recorder, - reason=graded.veto_reason or f"measured {measured_gain:+.2f}% below keep threshold {keep_threshold:+.2f}%", + reason=graded.veto_reason + or graded.degrade_reason + or f"measured {measured_gain:+.2f}% below keep threshold {keep_threshold:+.2f}%", ) def _reject_warm_replay_as_drift( diff --git a/src/hyperloom/orchestrator/state/_shared_state/render.py b/src/hyperloom/orchestrator/state/_shared_state/render.py index 990f7f9e72..d73132e53e 100644 --- a/src/hyperloom/orchestrator/state/_shared_state/render.py +++ b/src/hyperloom/orchestrator/state/_shared_state/render.py @@ -159,10 +159,10 @@ def to_mission_summary(self, *, now: datetime | None = None) -> str: geak_pending_tag = f" ⚠ geak candidate dropped unvalidated (rebench_{geak_revalidation_status})" else: geak_pending_tag = "" - from hyperloom.inference_optimizer import framework_registry + from hyperloom.inference_optimizer.performance_display import format_session_metric lines = [ - f"baseline : {framework_registry.format_primary_metric(self.framework, self.baseline_tput)}", + f"baseline : {format_session_metric(self, self.baseline_tput)}", f"current : {self._format_current_best_for_mission()}", f"gain : validated={self.cumulative_gain_validated:.2f}%{validated_age}", f"stack : {len(self.optimization_stack)} entries " @@ -180,11 +180,11 @@ def _format_current_best_for_mission(self) -> str: """Render the ``current_best`` one-liner for the mission summary.""" if not isinstance(self.current_best, dict) or not self.current_best: return "(none)" - from hyperloom.inference_optimizer import framework_registry + from hyperloom.inference_optimizer.performance_display import format_session_metric cb_tput = self.current_best.get("tput") perf = ( - framework_registry.format_primary_metric(self.framework, cb_tput) + format_session_metric(self, cb_tput, measurement=self.current_best) if isinstance(cb_tput, (int, float)) else "?" ) diff --git a/src/hyperloom/orchestrator/state/shared_state.py b/src/hyperloom/orchestrator/state/shared_state.py index d3f4201afd..efaa537fd3 100644 --- a/src/hyperloom/orchestrator/state/shared_state.py +++ b/src/hyperloom/orchestrator/state/shared_state.py @@ -5,6 +5,7 @@ from __future__ import annotations +import copy import json import logging import os @@ -150,6 +151,19 @@ def resolve_graded_comparison( else: ref_perf, reason = resolve_grading_anchor_perf(state) cand_perf = perf_snapshot_from_mapping(measurement) + if int(getattr(state, "agentx_epoch", 0) or 0) >= 3 and not all( + isinstance(perf, dict) + and perf.get("agentx_launch_contract") == 1 + and perf.get("agentx_workload_fingerprint") + for perf in (ref_perf, cand_perf) + ): + return GradedComparison( + objective=GRADED_INTVTY_P50, + candidate=axis_of(cand_perf, GRADED_INTVTY_P50), + reference=axis_of(ref_perf, GRADED_INTVTY_P50), + verdict=VERDICT_REVERT, + degrade_reason="native_workload_identity_missing", + ) if ref_perf and cand_perf: gain = gain_pct(axis_of(cand_perf, GRADED_INTVTY_P50), axis_of(ref_perf, GRADED_INTVTY_P50)) # The output guard reads the raw aggregate: the chip count divides both sides of the ratio, so the band @@ -218,7 +232,10 @@ def _normalize_envs(value: Any) -> dict[str, str]: def stack_base_params(current_best: Any) -> dict[str, Any]: """``base_*`` params projected from the fields ``current_best`` carries.""" cb = current_best if isinstance(current_best, dict) else {} - return {key: normalize(cb[source]) for key, source, normalize in _STACK_BASE_FIELDS if source in cb} + params = {key: normalize(cb[source]) for key, source, normalize in _STACK_BASE_FIELDS if source in cb} + if isinstance(cb.get("native_launch_overrides"), dict): + params["base_native_launch_overrides"] = copy.deepcopy(cb["native_launch_overrides"]) + return params def inject_stack_base_params( @@ -373,11 +390,16 @@ class SharedState(_RenderMixin, GapsStateMixin, _PhaseStateMixin): benchmark_mode: str = "" # Generation counter for AgentX measurements. agentx_epoch: int = 0 - # Which AgentX client measured this session: "aiperf" or "mlperf". Empty on - # sessions that predate the field; resume treats those as aiperf. Compared - # on resume so the two workloads cannot anchor each other without bumping - # the measurement epoch (which would invalidate unchanged aiperf sessions). + # Backend identity is immutable across resume; epoch 1 remains the legacy client. agentx_backend: str = "" + # Immutable native AgentX identity captured before the first baseline. + # This is the resume fallback when a process dies before + # ``baseline_config_path`` has been accepted and written back. + agentx_runtime_pins: dict[str, str] = field(default_factory=dict) + # Session-owned copy of the operator's source YAML. Used only until an + # accepted materialized ``baseline_config_path`` exists, so a pre-baseline + # crash can resume with the same AgentX selector/mode/custom envs. + benchmark_source_config_path: str = "" # The grading configuration this session was seeded with: {"objective": GRADED_INTVTY|GRADED_OUTPUT, # "noise_pct": float}. Recorded rather than re-derived because the derivation reads HYPERLOOM_PERF_METRIC / # HYPERLOOM_PERF_NOISE_PCT, and a resume is a new process: a shell that lost the variable would flip the axis @@ -1084,6 +1106,13 @@ def from_dict(cls, raw: dict[str, Any]) -> "SharedState": filtered["kernel_opt_task_attempts"] = {} if not isinstance(filtered.get("pending_kernel_integrations"), dict): filtered["pending_kernel_integrations"] = {} + if str(filtered.get("benchmark_mode") or "").strip().lower() == "agentx": + from hyperloom.common.agentx_mode import native_agentx_optimization_session, native_agentx_session + + if not filtered.get("agentx_backend"): + filtered["agentx_backend"] = "native" if native_agentx_session(filtered) else "legacy" + if native_agentx_session(filtered) and not native_agentx_optimization_session(filtered): + filtered["warm_replay_enabled"] = False # Normalize the unified ``explore_search`` ledger at load. filtered["explore_search"] = cls._build_explore_search( existing=filtered.get("explore_search"),