Skip to content

Native AgentX installs AIPerf twice before the benchmark starts. #1919

Open
Open
Native AgentX installs AIPerf twice before the benchmark starts.#1919
Assignees
haofrank
Labels
domain:inferenceRelated to inference optimizationtype:bugSomething is broken or not working as expected

Description

@spandoesai
spandoesai
opened on Oct 15, 2026

Target release

v0.3

How were you running Hyperloom?

Local

Optimization domain

Inference

Issue type

Environment / setup

Which phase failed?

Setup / baseline

Short Summary

A native AgentX run installs AIPerf twice, into two virtualenvs and two uv caches, so the benchmark waits on a second full download.

Job ID (hosted UI runs only)

No response

Command / prompt submitted

python3 -m hyperloom optimize \
  --benchmark-config session/agentx-dsv41flash.yaml \
  --tp 4 \
  --enable-roofline --target-roofline 0 \
  --no-framework-agent --no-kernel --no-research-scout \
  --enablement off \
  --max-hours 18

Branch feature/magpie-0.3.0-agentx at 40a143aa. The recipe launcher is InferenceX benchmarks/srt_agentic.sh (pin 408c015). Magpie is local 10c9fdd.

Expected behavior

install.sh already installs the pinned client into /root/.hyperloom/aiperf-venv. The native AgentX benchmark should reuse that environment, or at least its uv cache, instead of downloading and unpacking the client again.

Actual behavior

The same container installs AIPerf twice before the measurement starts.

install.sh _install_aiperf installs aiperf @ git+https://github.com/SemiAnalysisAI/aiperf.git@754356e9 into /root/.hyperloom/aiperf-venv with Python 3.12 and cache /root/.hyperloom/aiperf-cache. On this run, resolving packages took 14m 17s and preparing them took 19m 04s. The uv wheel download ran at about 165 KB/s. optimize did not start until 59 minutes after the container started.

Magpie then runs srt_agentic.sh. install_agentic_deps in InferenceX benchmark_lib.sh deletes $AIPERF_RUNTIME_DIR/venv and installs again. Magpie sets AIPERF_RUNTIME_DIR to the benchmark workspace .agentx-runtime and drops inherited AIPERF_* variables, so AIPERF_BIN never reaches the recipe. The second install uses a new uv binary, a new cache, and a downloaded Python 3.11. It finished about 33 minutes later. The 3600-second profiling window started at 18:55 UTC.

The package names overlap, but the wheels do not. The first install downloaded numpy 16.0 MiB and pandas 11.8 MiB for Python 3.12. The second downloaded numpy 16.1 MiB and pandas 12.2 MiB for Python 3.11. uv only reuses wheels from the UV_CACHE_DIR it was given.

Steps to reproduce

  1. Launch a native AgentX optimize on a fresh container, so /root/.hyperloom is empty.
  2. Use an AgentX source YAML whose launcher is benchmarks/srt_agentic.sh.
  3. Watch install.sh install AIPerf into /root/.hyperloom/aiperf-venv.
  4. After the server is healthy, watch srt_agentic.sh run uv pip install -e utils/aiperf into /.agentx-runtime/venv.

Logs

[inference-optimizer] installing aiperf (AgentX) into /root/.hyperloom/aiperf-venv: aiperf @ git+https://github.com/SemiAnalysisAI/aiperf.git@754356e9a39acc6cc6afb242d123bb57c3fb6f75
Using Python 3.12.3 environment at: /root/.hyperloom/aiperf-venv
Resolved 130 packages in 14m 17s
Prepared 130 packages in 19m 04s
[inference-optimizer] aiperf installed OK: /root/.hyperloom/aiperf-venv/bin/aiperf
optimize start 2026-10-15T18:05:44+00:00

Using CPython 3.11.17
uv pip install --python .../.agentx-runtime/venv/bin/python -e .../utils/aiperf numpy pandas ...
Installed 131 packages in 232ms
18:55:22 NOTICE Phase profiling started | target: 3600.0s duration

Additional artifacts / links

No response

Data handling

  • I confirmed the attached logs/artifacts contain no sensitive information

Activity

  1. spandoesai
    added
    type:bugSomething is broken or not working as expected
    on Oct 15, 2026
  2. github-actions
    added
    domain:inferenceRelated to inference optimization
    on Oct 15, 2026

  3. chaojhou commented on Oct 17, 2026

    @chaojhou
    chaojhou
    on Oct 17, 2026
    Collaborator

    Thanks for the detailed report and the timings.

    Status on main (7e57b1e, after #1922): confirmed by reading the code at the pinned commits; not reproduced, and the time per run is not measured.

    • First install: gone. Use Magpie's native AgentX contract for optimization #1922 (75df551) removed the Hyperloom AIPerf install; install.sh and src/ no longer install AIPerf.
    • Second install: still there, now on main, and on every run. main uses Magpie's native AgentX path for every benchmark (baseline round, EXPLORE candidate, grid variant, rebench, one run per concurrency). For each run:
      • Magpie@6a1ee26 creates a new timestamped workspace (Magpie/modes/benchmark/benchmarker.py:292, workspace.py:67-73), strips inherited AIPERF_* and sets AIPERF_RUNTIME_DIR=<workspace>/.agentx-runtime (agentx_runtime.py:430-435, :459).
      • Native runs use Magpie's default InferenceX@408c015 (Magpie/modes/benchmark/inferencex.py:22), not the INFERENCEX_REF in Hyperloom's install.sh. Its install_agentic_deps keeps the uv cache inside that runtime dir with no override, deletes the venv and rebuilds a Python 3.11 venv with an editable utils/aiperf (inferencex-e2e/benchmarks/benchmark_lib.sh:3086-3089, :3133, :3140-3142).
      • So the uv cache is always cold: every run downloads the full client again, not only the first run of a session.
    • Why it matters beyond time: the client, and therefore the install, starts after the server is healthy (agentx_runtime.py:618-637), so the download runs while the server holds its GPUs, and it counts against the client timeout (agentx_runtime.py:105-115). A slow or restricted network can fail a candidate for reasons unrelated to the change under test. Measured results are not affected.

    Where to fix (upstream; Hyperloom cannot route around it, since Magpie strips AIPERF_* and sets the runtime dir last):

    • InferenceX: honour a caller-supplied uv cache directory and reuse a matching venv instead of deleting it. InferenceX@3d55815 already makes AIPERF_UV_CACHE_DIR overridable (benchmarks/benchmark_lib.sh:1787-1790), but still deletes the venv (:1841).
    • Magpie: pass a host-stable uv cache (for example under ~/.cache/magpie) to the client and move its default InferenceX to a commit that honours it.
    • Hyperloom: bump MAGPIE_REF once both land.

    Keeping this open to track the upstream fixes; we will link the Magpie and InferenceX issues here and close this after the pin bump.

  4. github-actions
    added
    awaiting-author-responseWaiting for issue author to reply (auto-managed)
    on Oct 17, 2026

  5. chaojhou commented on Oct 17, 2026

    @chaojhou
    chaojhou
    on Oct 17, 2026
    Collaborator

    Upstream tracking: AMD-AGI/Magpie#128 (assigned to @haofrank).

  6. chaojhou commented on Oct 17, 2026

    @chaojhou
    chaojhou
    on Oct 17, 2026
    Collaborator

    InferenceX side: SemiAnalysisAI/InferenceX#4234.

  7. chaojhou commented on Oct 17, 2026

    @chaojhou
    chaojhou
    on Oct 17, 2026
    Collaborator

    Triage against main 7e57b1e8f: partly true, and the remaining part is worse than reported.

    The Hyperloom-side install is gone: #1922 deleted aiperf_client.sh and Hyperloom's own AIPerf install. What remains is upstream, and it happens on every run (baseline, each candidate, each grid variant, rebench, each concurrency point), not once per session. Magpie's native AgentX path gives each benchmark a fresh workspace with AIPERF_RUNTIME_DIR inside it, and InferenceX's install_agentic_deps keeps the uv cache under that runtime dir and unconditionally deletes the venv. The install runs after the server is healthy, so it holds the GPUs and counts against the client timeout. On a slow network that can fail a candidate.

    A Hyperloom-side workaround isn't available, and wouldn't be right anyway. Magpie's default InferenceX 408c015 doesn't support an AIPERF_UV_CACHE_DIR override, and Hyperloom deliberately rejects AIPERF_* / AGENTX_* in benchmark configs (cli/benchmark_config.py:116-122, :140-142). The fix is Magpie#128: pass a stable host-level uv cache and move the default InferenceX to a ref that honours AIPERF_UV_CACHE_DIR and reuses a matching venv. InferenceX should also stop deleting the venv unconditionally. After that, bump MAGPIE_REF (assets/install.sh:182, preflight.py:2169), ideally in the same bump as Magpie#127 for #1930, and close this.

    Removing awaiting-author-response: nothing more is needed from the author. Priority: P2 (P1 on restricted networks). Assigning @haofrank, who owns Magpie#128.

  8. chaojhou
    removed
    awaiting-author-responseWaiting for issue author to reply (auto-managed)
    on Oct 17, 2026
  9. chaojhou
    assigned
    haofrank
    on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

domain:inferenceRelated to inference optimizationtype:bugSomething is broken or not working as expected

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions