feat(agentx) Use Magpie native AgentX for Hyperloom optimization

docs/compatibility.rst

@@ -93,6 +93,26 @@ The following table lists the validated Hyperloom version and component combinat
 .. _AgentKernelArena 0.2.0: https://rocm.docs.amd.com/projects/agent-kernel-arena/en/docs-0.2.0/
 .. _Magpie 0.2.0: https://rocm.docs.amd.com/projects/magpie/en/docs-0.2.0/
 
+.. note::
+
+   The base benchmark path remains compatible with Magpie 0.2.0. The pinned
+   native AgentX pair is `Magpie v0.3.0
+   <https://github.com/AMD-AGI/Magpie/releases/tag/v0.3.0>`_ plus native launch overrides, custom-model replay, and client/eval
+   compatibility fixes at commit ``d80eb4d3dad7fabe01ce81d049e2983adf2c86dd`` and InferenceX commit
+   ``408c015be4b22d14c69518643609669405507077``. Pass the Magpie YAML with
+   ``--benchmark-config``; its ``benchmark.agentx: enable`` source switch
+   automatically selects Hyperloom's persisted AgentX session and grading mode.
+   Fresh ``HYPERLOOM_AGENTX=1`` launches also select native AgentX and resolve
+   a recipe from normal workload inputs. Saved epoch-1 sessions keep the legacy
+   client; saved epoch-2 native sessions keep their measurement-only contract,
+   and epoch-3 sessions keep their upstream launcher contract.
+   New native sessions use epoch 4 and the Magpie-managed launch-overrides contract. ``benchmark.docker_image`` overrides/pins the effective
+   recipe image and is included in its fingerprint; an existing
+   ``HYPERLOOM_IMAGE`` is only a strict consistency assertion. Neither value
+   starts or attests the outer container.
+   ``install.sh`` upgrades an importable pre-AgentX Magpie
+   build instead of treating import success as sufficient.
+
 .. note::
 
    TraceLens does not have hard requirements for the GPU, ROCm version, or the OS; it has scripts to verify whether a trace is valid/parseable. TraceLens is:

docs/components/magpie.md

@@ -39,12 +39,59 @@ cmd = [
 ```
 
 Each run produces a `benchmark_report.json` that Hyperloom parses to extract
-throughput/measurements and pick winners. Hyperloom pins Magpie to a ref that
-already copies benchmark scripts atomically upstream; `_magpie_patcher.py` applies
-install-time compatibility patches (SGLang trust, eval concurrency, client
-tokenizer hook) to the pinned scripts. See
+throughput/measurements and pick winners. Hyperloom pins the Magpie
+[v0.3.0 release](https://github.com/AMD-AGI/Magpie/releases/tag/v0.3.0) plus native launch overrides, custom-model replay, and the
+generic client/eval compatibility fixes at immutable commit
+`d80eb4d3dad7fabe01ce81d049e2983adf2c86dd`. The fixes keep evaluation scripts
+bound to their intended InferenceX source after directory changes and run the
+packaged benchmark client with Transformers 5 in a separate Python environment.
+The serving framework environment is unchanged. The pinned scripts also support
+Hyperloom's client tokenizer and trust settings directly, so compatibility checks
+leave the audited Magpie package unchanged. These capabilities are pinned to a merged Magpie
+commit beyond v0.3.0. Earlier GPU validation used the v0.3.0 release commit and does not
+validate these extensions; their source and wheel share an audited execution tree.
+The pin includes [Magpie #117](https://github.com/AMD-AGI/Magpie/pull/117): SGLang detailed
+profiling accepts runtimes without shape discovery and enables each shape-discovery
+option only when supported. Graph-mode captures still require graph profiling support
+and retain per-batch graph trace export.
+The installer and CLI apply native capability, commit, and execution-tree checks
+only to native AgentX sessions. Ordinary benchmarks retain the importability
+contract, including compatible custom packages and existing offline installations.
+With `benchmark.agentx: enable`, Magpie resolves the serving specification,
+owns server startup and cleanup, and runs the maintained InferenceX/AIPerf client.
+The integration validates the package, recipe/client sources, and effective
+server launch evidence. Both the repository root and the `inferencex-e2e/`
+project layout are supported; no Slurm launcher extension is required.
+
+Fresh `HYPERLOOM_AGENTX=1` sessions use Magpie native AgentX and retain
+Hyperloom's optimization loop. Magpie resolves the recipe and launcher from
+the model, framework, GPU, precision, and concurrency; ambiguous inputs fail
+with a selector error. New sessions record backend `native` and epoch 4.
+With no registered recipe match, an explicit image, TP, EP, concurrency, and
+canonical model identify a custom SGLang/vLLM workload; Magpie validates its
+model context and resolves a formal generic launcher. Registered ambiguities
+remain errors. Candidate KEEPs retain the exact native launch snapshot, so
+resume and subsequent candidates do not apply the same launch arguments twice.
+Persisted epoch-1 sessions resume the legacy client, and epoch-2 native sessions
+retain their measurement-only contract. Saved epoch-3 sessions retain their
+upstream-launcher contract. No baseline or KEEP record is migrated.
+
+Native profiling derives a diagnostic run from the accepted candidate and uses
+Magpie's phase-gated, step-bounded torch profiler. Magpie owns the first-capture
+delay, repetition interval, count limit, and framework-specific controls.
+Hyperloom selects a complete capture from the manifest for kernel/roofline
+analysis. Successful standalone `profile` tasks also publish their trace to the
+Coordinator for analysis, without requiring benchmark-valid throughput.
+Diagnostic success never makes the run a publishable KEEP result.
+Detailed annotations require a compatible instrumented serving runtime; an
+ordinary GPU trace alone does not establish complete kernel shape metadata. See
+[Run an InferenceX AgentX workload](../how-to/optimize.md#run-an-inferencex-agentx-workload)
+for the example and current optimization limitations, and
 [Hyperloom optimization loop](../conceptual/optimization-loop.md) for more information.
 
 ## Magpie documentation
 
 For detailed documentation on Magpie, see [Magpie on ROCm Docs](https://rocm.docs.amd.com/projects/magpie/en/latest/).
+
+The pinned generic serving path also supplies InferenceX evaluation defaults (`EVAL_ONLY=false`,
+`IS_AGENTIC=0`, and a local placeholder API key), preserving explicitly configured values.

docs/conceptual/optimization-loop.md

@@ -102,8 +102,10 @@ PRELUDE establishes the session baseline:
    `competitor_target.json`. With no external GPU configured, it writes a
    no-target marker and clears the competitor target. Advisory and final-report
    comparisons read the same target file; missing targets remain unavailable.
-   AgentX uses accepted `current_best` metrics, normalized by `state.tp` and
-   matched at `state.conc`, without rereading benchmark artifacts.
+   AgentX uses accepted `current_best` metrics, normalizing total throughput by
+   the trusted, fingerprint-bound recipe's recorded TP×PP×PCP GPU count (with
+   `state.tp` only as a legacy-artifact fallback) and matching at `state.conc`,
+   without rereading benchmark artifacts.
    `--no-target-advisory` disables prompt hints, not the final comparison.
    External references never change Objective, scoring, or KEEP/REVERT.
 2. `baseline` measures the starting throughput and records the benchmark
@@ -112,6 +114,27 @@ PRELUDE establishes the session baseline:
    `roofline` is the preferred composite path when enabled; it wraps
    profiling, trace analysis, and `analysis.md` snapshot publication.
 
+For native AgentX, Hyperloom pre-resolves every recipe point through the same
+interpreter that runs Magpie. It fills an omitted outer `--tp` from resolved
+TP×PP×PCP and treats an explicit value as an exact assertion. Topology-changing
+rounds fail before launch. In fresh epoch-4 sessions, Magpie owns server
+startup and records the effective candidate arguments, environment, and source
+overlays. Hyperloom binds that evidence before accepting a result. Profile
+rounds derive from the accepted configuration and use Magpie's phase-gated
+torch capture. Each capture remains separate and diagnostic; it cannot anchor
+KEEP comparisons. GEAK proposals require review and canonical AgentX
+remeasurement before adoption. Saved epoch-2 sessions retain their earlier
+measurement-only restrictions, and epoch-3 sessions retain the upstream
+launcher and compatibility-profiler contract.
+
+Session mode is selected before PRELUDE: `--benchmark-config <yaml>` reads the
+source config, and `benchmark.agentx: enable` automatically stamps AgentX mode
+for grading and persisted state. Fresh `HYPERLOOM_AGENTX=1` sessions also select
+native AgentX; no additional opt-in is required. Backend identity survives
+resume through the accepted/source
+config and measurement epoch; native-only refusals do not apply to legacy
+AgentX sessions.
+
 `model_class` is supplied by the launcher or derived once from model
 metadata at boot. There is no separate live `classify` action.
 
@@ -320,6 +343,11 @@ gates.
 What the phase actually does depends on the kernel backend, and the branches
 look very different from Orchestration's side:
 
+- **Native AgentX**: optimization sessions dispatch diagnostic proposals and
+  require verified candidate launch evidence plus canonical AgentX
+  remeasurement before KEEP. Source changes also require the Critic review
+  and transactional integration. Only persisted epoch-2 measurement-only
+  sessions retain `unsupported_upstream_launcher_hook` and skip this phase.
 - **Default (`geak`)**: entering the phase enqueues one Coordinator-owned
   `kernel_agent` task, which holds `server_lifecycle`, `workspace_mutation` and
   `benchmark_lane` for the whole pipeline. Under GEAK it runs a single
@@ -360,18 +388,21 @@ be attempted or explicitly rejected before report can close the run.
 
 ## SWEEP
 
-SWEEP measures the optimized stack against the baseline across a
-concurrency ladder, one arm each, and produces the throughput-vs-
-interactivity curve. The ladder is sized for the workload: powers of two
-down from 256 for a synthetic run, `1,4,8,10,14,20,28` for an agentic one,
-where a request carries orders of magnitude more prompt and the same card
-saturates far lower. Override either with `--conc-sweep-concs`. Under
-AgentX the sweep is off unless `--enable-conc-sweep` is passed, since each
-rung is a 3600 s window and the session grades at a fixed CONC; SWEEP then
-records a disabled skip.
+SWEEP measures the optimized stack against the baseline across a concurrency
+ladder and produces a throughput-vs-interactivity curve. Synthetic workloads
+default to powers of two down from 256; legacy AgentX and the MLPerf backend
+use `1,4,8,10,14,20,28`. Override the ladder with `--conc-sweep-concs`.
+All AgentX sessions default the sweep off. Legacy AgentX and MLPerf may opt in
+with `--enable-conc-sweep`.
+
+Native AgentX measures the fixed recipe concurrency supplied by `--conc`.
+Changing it would change the accepted workload identity, so explicit
+`--enable-conc-sweep` is rejected during preflight; `--conc-sweep-concs` does
+not enable it.
 
 Results update `last_conc_sweep` and feed the final report and breakdown.
-The phase exits on `sweep_done` (or `sweep_failed`).
+When enabled, the phase exits on `sweep_done` (or `sweep_failed`); a disabled
+sweep records its terminal skip and proceeds to CLOSE.
 
 ## CLOSE