Repository navigation
feat(agentx): adopt harness v1.0.6 scenario defaults - #3646
Merged
Merged
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
cquil11
force-pushed
the
cquil11/agentx-v1.0.6-defaults
branch
from
October 1, 2026 22:18
1e08f06 to
46678ab
Compare
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it bumps a vendored harness submodule and changes shared AgentX CLI defaults across all recipes (dataset timeouts, GPU telemetry, dataset size, TCP timeout), a human look would still be worthwhile to confirm the new --scenario agentx preset truly preserves the removed explicit values (the harness submodule isn't checked out in this environment, so its preset contents can't be verified directly here).
What was reviewed: the build_replay_cmd() simplification in benchmark_lib.sh (removed flags/exports vs. check_env_vars), the submodule pointer bump, and doc/changelog sync across English and Chinese pages. Checked the four candidate concerns (lost dataset-entry cap, lost H200 timeout exports, lost --no-gpu-telemetry, standalone doc losing explicit recipe knobs) against the diff and changelog — each is explained as now delegated to the new agentx preset, consistent with the PR description, though not independently verifiable without the submodule checked out.
Extended reasoning...
The diff bumps the vendored AgentX Harness submodule and simplifies benchmark_lib.sh's build_replay_cmd() to rely on a new --scenario agentx preset instead of explicit flags/env exports (dataset timeouts, GPU telemetry, dataset entry count, TCP user timeout), with matching doc and changelog updates in English and Chinese. No auth, crypto, or data-exposure surface is touched; this is CI/benchmark tooling configuration. The bug hunter ran multiple rounds and found no bugs, ruling out the four plausible regressions as covered by the new preset per the PR description and changelog entry, but the actual preset contents live in the submodule, which is not checked out in this shallow clone, so that claim could not be independently verified from this session.
This review covers commit 1e08f06, which is no longer the latest commit on this pull request; later commits are not covered by it.
4 of 10 tasks
7 of 11 tasks
adibarra
added a commit
that referenced
this pull request
Oct 5, 2026
- uv everywhere: shared infx/bench/uv.py; lm-eval, vendor evals, BFCL and the AIPerf venv install with uv (BFCL passes --excludes to reuse the image stack). - fixed-seq: drop the pip3 runtime install and SWEEP_FLAGS so single- and multi-node clients send the same policy. - AgentX: drop inputs the agentx scenario (#3646) owns or nobody varies; keep only the two 062126 trace loaders; move venv deps to requirements.txt. - Tests audited for regression value; perf-changelog entry for the loader pins.
adibarra
added a commit
that referenced
this pull request
Oct 6, 2026
…#3610) * chore(bench): remove the llm-d lane, SWE-bench Lite and native power collector Split out of the benchmark_lib.sh port (#3610) so the port diff is only the port. None of these paths is reachable from a checked-in config: - llm-d: Dockerfile, binaries, recipes and the multi-node job/submit/server scripts. - SWE-bench Lite and Modal: run_swebench_eval and its agentic generation, Modal credential handling, scorer, task YAML, runtime patches, workflow inputs, secrets, uploads, runtime settings and the threshold. - native_power_collect.sh and native_power_lifecycle.sh, used only by the retired AMD lanes. The adapter that reads LOGS/native_power stays. * chore(bench): drop the vLLM chat-template-kwargs and TRT-LLM perf-metrics patches - SPEED-Bench: vLLM ships --chat-template-kwargs for client-rendered datasets since v0.23.0 (vllm-project/vllm#44244). Default collection to vllm/vllm-openai:v0.30.0, note the minimum on the workflow input, and remove apply_chat_template_kwargs_shim and the DSpark preflight. - MiniMax-M3 TRT-LLM AgentX: stop patching PerfMetricsManager off. return_perf_metrics, which these recipes need for trtllm-serve's Prometheus metrics, also enables per-step timing until TensorRT-LLM decouples the two upstream. Both recipes now share minimaxm3-trtllm-agentx.sh, which only applies the store=false patch. * chore(bench): remove benchmark_lib.sh helpers nothing calls and llm-d/SWE-bench leftovers Remove 18 benchmark_lib.sh functions with no callers (_background_process_descendants, _background_process_groups_alive, _background_process_is_running, agentic_kv_offload_enabled, agentic_pip_install, append_command, exit_after_background_process_cleanup, require_agentic_kv_offload_backend, require_agentic_kv_offload_none, rewrite_lm_eval_meta_env, run_amd_multinode_after_preflight, select_available_server_port, setup_eval_context, stop_background_process_groups, stop_background_process_tree, validate_required_agentic_server_metrics, write_agentic_result_json, write_command) along with the one test that exercised select_available_server_port, the llmd-vllm case in multi_node/runtime_settings.sh, and SWE-bench's "resolved" metric family in eval result collection. * chore(agentx): stop generating post-run plots Nothing reads metrics_plots.png or workload_distribution_plots.png. Delete generate_aiperf_plots.py, drop the histogram from analyze_benchmark_distributions (the ISL/OSL summary stays), and drop matplotlib from the AgentX client venv. * refactor(bench): replace benchmark_lib.sh with the infx.bench package Delete benchmarks/benchmark_lib.sh and move its behavior into the container-side Python package infx/bench (stdlib only, Python 3.10): wait, fixed-seq, agentic and eval commands behind one `python3 -m infx.bench` entrypoint, with shared process, readiness and input helpers. srt entry scripts become thin shims; recipe paths are unchanged. benchmarks/check_env.sh keeps check_env_vars for workflows and recipe setup scripts. AgentX single-concurrency validation moves from the workflow into LaunchRequest. Also drops code only the retired AMD/TileRT lanes used (server_watch, agentic --server-pid, the monitor command, fixed-seq point mode), the PROFILE trace relay, and dead topology aliases from the meta_env.json mapping. * chore(agentx): drop the post-run plot step from infx.bench.agentic Follows the plot removal in the base PR: no generate_aiperf_plots call and no matplotlib in the AIPerf client venv. * refactor(bench): address review on infx.bench - uv everywhere: shared infx/bench/uv.py; lm-eval, vendor evals, BFCL and the AIPerf venv install with uv (BFCL passes --excludes to reuse the image stack). - fixed-seq: drop the pip3 runtime install and SWEEP_FLAGS so single- and multi-node clients send the same policy. - AgentX: drop inputs the agentx scenario (#3646) owns or nobody varies; keep only the two 062126 trace loaders; move venv deps to requirements.txt. - Tests audited for regression value; perf-changelog entry for the loader pins. * chore(llm-d): stop forwarding SWE-bench/Modal env SWE-bench is gone; submit.sh required SWEBENCH_USE_MODAL, which nothing sets. * chore(power): remove non-srt-slurm in-tree power collection Delete the client GPU sampler (start/stop_gpu_monitor), the single-node CSV and InferenceX-native power adapters, their workflow uploads and env, and single-node require-power. Multi-node srt-slurm DCGM power is unchanged. Single-node and AMD results publish no power until srt-slurm native telemetry covers them. * fix(agentx): restore the result writer and server-metric check 35813cc removed write_agentic_result_json and validate_required_agentic_server_metrics as uncalled, but run_agentic_replay_and_write_outputs still calls both, so every AgentX point exited 127. Restore them without the retired plot step. * fix(agentx): require KV_OFFLOADING only for matrix-launched points srt-slurm's benchmarks/agentx.sh runs srt_agentic.sh without IS_AGENTIC or KV_OFFLOADING. benchmark_lib.sh checked the KV-offload declaration only for agentic matrix jobs; restore that gate so the standalone entrypoint works. * chore: drop the perf-changelog entry * chore: drop the perf-changelog entry --------- Co-authored-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
AgentX commands repeat defaults that now live in the harness's
agentxpreset.Changes
89b218678), including feat(agentx): add concise agentx scenario defaults agentx-harness#47.--scenario agentxand remove repeated flags and environment exports. Keep recipe overrides, fast mode, routing, and dataset selection.