Merge branch 'amd/dsv41flash-mi355x-vllm-attention-config-indexer' into amd/dsv41flash-mi355x-vllm-58671-mxfp4-indexer

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml
#	inferencex-e2e/configs/amd-master.yaml

Author: Fangzhou-Ai

README.md

@@ -21,6 +21,7 @@ Trusted by Operators of Trillion Dollar Token Factories such as OpenAI, Meta, Mi
 | [`InferenceX-e2e/`](inferencex-e2e/) | 🚀 End-to-end Inference Serving Benchmarks |
 | [`CollectiveX/`](collectivex/) | 🌐 Networking & Collective Communication Benchmarks (Experimental Beta) |
 | [`OperatorX/`](operatorx/) | ⚙️ Operator & Kernel Level Benchmarks (Experimental Beta) |
+| [`power_model/`](power_model/) | ⚡ OSS System Level Power Modelling |
 | [`shared/`](shared/) | 🧩 Home for shared components |
 | [`experimental/`](experimental/) | 🧪 Remaining experiments |
 
@@ -92,6 +93,10 @@ This pace of software advancement creates a challenge: benchmarks conducted at a
 PRs are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for more details on the PR review flow, the [PR Review Checklist](inferencex-e2e/docs/PR_REVIEW_CHECKLIST.md), and the merge process.
 For the maintainer and agent documentation map, start with [`docs/index.md`](inferencex-e2e/docs/index.md). It links the architecture, configuration, workflow, eval, runner, and troubleshooting references.
 
+To benchmark an existing server without CI or Slurm, see
+[Run AgentX-Harness Standalone](inferencex-e2e/docs/agentx-standalone.md)
+for client installation and a direct `aiperf profile` command.
+
 ## Acknowledgements & Supporters
 Thank you to Lisa Su and Anush Elangovan for providing the MI355X and CDNA3 GPUs for this free and open-source project. We want to recognize the many AMD contributors for their responsiveness and for debugging, optimizing, and validating performance across AMD GPUs. 
 We’re also grateful to Jensen Huang and Ian Buck for supporting this open source with access to a GB200 NVL72 rack (through OCI) and B200 GPUs. Thank you to the many NVIDIA contributors from the NVIDIA inference team, NVIDIA Dynamo team.

README_zh.md

@@ -21,6 +21,7 @@
 | [`InferenceX-e2e/`](inferencex-e2e/) | 🚀 端到端推理服务基准测试 |
 | [`CollectiveX/`](collectivex/) | 🌐 网络与集合通信基准测试（实验性 Beta 版本） |
 | [`OperatorX/`](operatorx/) | ⚙️ 算子与内核级基准测试（实验性 Beta 版本） |
+| [`power_model/`](power_model/) | ⚡ 开源系统级功耗建模 |
 | [`shared/`](shared/) | 🧩 共享组件目录 |
 | [`experimental/`](experimental/) | 🧪 其余实验 |
 
@@ -91,6 +92,10 @@ SGLang、vLLM、TensorRT-LLM、CUDA、ROCm 等 AI 软件通过核函式優化、
 欢迎提交 PR！有关 PR 审阅流程、[PR 审阅清单](inferencex-e2e/docs/PR_REVIEW_CHECKLIST_zh.md)与合并流程的更多详情，请参阅 [CONTRIBUTING_zh.md](CONTRIBUTING_zh.md)。
 维护者与 Agent 的文档导航请从 [`docs/index_zh.md`](inferencex-e2e/docs/index_zh.md) 开始，其中集中链接架构、配置、Workflow、评估、Runner 与故障排查参考。
 
+无需 CI 或 Slurm 即可对已有服务进行基准测试，请参阅
+[独立运行 AgentX-Harness](inferencex-e2e/docs/agentx-standalone_zh.md)，
+其中包含客户端安装步骤和直接执行的 `aiperf profile` 命令。
+
 ## 致谢与支持者
 感谢 Lisa Su 与 Anush Elangovan 为这一免费开源项目提供 MI355X 与 CDNA3 GPU。我们也要感谢众多 AMD 贡献者的积极响应，以及他们在各类 AMD GPU 上进行调试、优化与性能验证所付出的努力。
 我们同样感谢 Jensen Huang 与 Ian Buck 通过提供 GB200 NVL72 机架（经由 OCI）与 B200 GPU 来支持本开源项目。感谢来自 NVIDIA 推理团队与 NVIDIA Dynamo 团队的众多 NVIDIA 贡献者。

inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv4/sglang/b300-fp4-eagle/agentic.yaml

@@ -0,0 +1,335 @@
+# DeepSeek-V4-Pro AgentX on B300 with SGLang EAGLE (3 steps, 4 draft tokens).
+# TP4 and TP8 run the flashinfer MXFP4 MoE with GPU-resident KV; DEP8
+# (attention DP + Mega-MoE + FP4 indexer) runs behind the SGLang router with a
+# HiCache DRAM tier.
+base:
+  schema: 2
+  name: dsv4-fp4-b300-sglang-agentic-eagle
+  model:
+    path: hf:deepseek-ai/DeepSeek-V4-Pro
+    container: lmsysorg/sglang:dev-cu13-nightly-09242@sha256:c47bbe7448050608b4e08669c742df4835a45a95bf6b19f2c4e51a7f6609e9d5
+    precision: fp4
+  resources:
+    gpu_type: b300
+    gpus_per_node: 8
+  frontend:
+    type: sglang
+    enable_multiple_frontends: false
+  observability:
+    enabled: false
+    tachometer:
+      enabled: false
+  engine: sglang
+  roles:
+    agg:
+      nodes: 1
+      workers: 1
+      gpus: 8
+      args:
+        served-model-name: deepseek-ai/DeepSeek-V4-Pro
+        trust-remote-code: true
+        tensor-parallel-size: 8
+        moe-runner-backend: flashinfer_mxfp4
+        disable-flashinfer-autotune: true
+        mem-fraction-static: 0.88
+        swa-full-tokens-ratio: 0.1
+        allow-auto-truncate: true
+        chunked-prefill-size: 8192
+        tool-call-parser: deepseekv4
+        reasoning-parser: deepseek-v4
+        chat-template: /infmax-workspace/benchmarks/single_node/chat_templates/deepseek_v4_thinking.jinja
+        watchdog-timeout: 1800
+        speculative-algorithm: EAGLE
+        speculative-num-steps: 3
+        speculative-eagle-topk: 1
+        speculative-num-draft-tokens: 4
+        attention-backend: dsv4
+        page-size: 256
+        disable-shared-experts-fusion: true
+        enable-metrics: true
+        enable-cache-report: true
+      env:
+        SGLANG_ENABLE_UNIFIED_RADIX_TREE: '1'
+        SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: '1'
+        PYTHONNOUSERSITE: '1'
+        TORCH_CUDA_ARCH_LIST: '10.0'
+        # Outlast AIPerf's pooled connections past Uvicorn's keep-alive.
+        SGLANG_TIMEOUT_KEEP_ALIVE: '900'
+        SGLANG_JIT_DEEPGEMM_FAST_WARMUP: '1'
+        SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: '1'
+        SGLANG_OPT_USE_JIT_NORM: '1'
+        SGLANG_OPT_USE_JIT_INDEXER_METADATA: '1'
+        SGLANG_OPT_USE_TOPK_V2: '1'
+        SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: '1'
+        # Triton compiles with the image's CUDA ptxas.
+        TRITON_PTXAS_PATH: /usr/local/cuda/bin/ptxas
+  benchmark:
+    type: custom
+    command: bash /infmax-workspace/benchmarks/srt_agentic.sh
+    env:
+      MODEL: deepseek-ai/DeepSeek-V4-Pro
+      # Warmup can leave bytes unacknowledged past AIPerf's 30 s default.
+      AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
+      AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:'
+
+# One variant per point. TP4/TP8: admission is 2x CONC and the decode graph
+# batch 4x CONC, capped at 64.
+override_tp4_c1:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 2
+        cuda-graph-max-bs-decode: 4
+  benchmark:
+    env:
+      CONC: '1'
+      KV_OFFLOADING: none
+
+override_tp4_c2:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 4
+        cuda-graph-max-bs-decode: 8
+  benchmark:
+    env:
+      CONC: '2'
+      KV_OFFLOADING: none
+
+override_tp4_c4:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 8
+        cuda-graph-max-bs-decode: 16
+  benchmark:
+    env:
+      CONC: '4'
+      KV_OFFLOADING: none
+
+override_tp4_c8:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 16
+        cuda-graph-max-bs-decode: 32
+  benchmark:
+    env:
+      CONC: '8'
+      KV_OFFLOADING: none
+
+override_tp4_c16:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 32
+        cuda-graph-max-bs-decode: 64
+  benchmark:
+    env:
+      CONC: '16'
+      KV_OFFLOADING: none
+
+override_tp4_c32:
+  roles:
+    agg:
+      gpus: 4
+      args:
+        tensor-parallel-size: 4
+        max-running-requests: 64
+        cuda-graph-max-bs-decode: 64
+  benchmark:
+    env:
+      CONC: '32'
+      KV_OFFLOADING: none
+
+override_tp8_c1:
+  roles:
+    agg:
+      args:
+        max-running-requests: 2
+        cuda-graph-max-bs-decode: 4
+  benchmark:
+    env:
+      CONC: '1'
+      KV_OFFLOADING: none
+
+override_tp8_c2:
+  roles:
+    agg:
+      args:
+        max-running-requests: 4
+        cuda-graph-max-bs-decode: 8
+  benchmark:
+    env:
+      CONC: '2'
+      KV_OFFLOADING: none
+
+override_tp8_c4:
+  roles:
+    agg:
+      args:
+        max-running-requests: 8
+        cuda-graph-max-bs-decode: 16
+  benchmark:
+    env:
+      CONC: '4'
+      KV_OFFLOADING: none
+
+override_tp8_c8:
+  roles:
+    agg:
+      args:
+        max-running-requests: 16
+        cuda-graph-max-bs-decode: 32
+  benchmark:
+    env:
+      CONC: '8'
+      KV_OFFLOADING: none
+
+override_tp8_c16:
+  roles:
+    agg:
+      args:
+        max-running-requests: 32
+        cuda-graph-max-bs-decode: 64
+  benchmark:
+    env:
+      CONC: '16'
+      KV_OFFLOADING: none
+
+override_tp8_c32:
+  roles:
+    agg:
+      args:
+        max-running-requests: 64
+        cuda-graph-max-bs-decode: 64
+  benchmark:
+    env:
+      CONC: '32'
+      KV_OFFLOADING: none
+
+# DEP8 replaces the TP8 MoE path: the router keeps each session on the DP rank
+# holding its radix prefix, the global prefill chunk is 8192 per rank, decode
+# graphs cover the padded speculative batch across ranks, and AIPerf owns the
+# warmup. DEP8 mixes prefill into decode batches, schedules shortest prefill
+# first, coordinates speculative prefill across DP ranks, and keeps the router
+# circuit breaker off. HiCache capacity is a host/device ratio of 2.
+override_dep8_c64:
+  frontend: &dep8_frontend
+    type: sglang-router
+    args:
+      policy: consistent_hashing
+      request-id-headers: x-correlation-id
+      dp-aware: true
+      connect-timeout-secs: 900
+      request-timeout-secs: 14400
+      disable-health-check: true
+      disable-circuit-breaker: true
+      # A transient router-to-engine send failure would otherwise abort the run.
+      retry-max-retries: 8
+      retry-initial-backoff-ms: 500
+      retry-max-backoff-ms: 10000
+      retry-backoff-multiplier: 2
+  roles:
+    agg:
+      args:
+        <<: &dep8_args
+          moe-runner-backend: null
+          data-parallel-size: 8
+          tokenizer-worker-num: 8
+          enable-prefill-delayer: true
+          prefill-decode-interval: 32
+          enable-dp-attention: true
+          enable-dp-attention-local-control-broadcast: true
+          incremental-streaming-output: true
+          stream-interval: 20
+          expert-parallel-size: 8
+          moe-a2a-backend: megamoe
+          enable-w4a4-mxfp4-megamoe: true
+          enable-deepseek-v4-fp4-indexer: true
+          enable-mixed-chunk: true
+          schedule-policy: shortest-prefill-first
+          mem-fraction-static: 0.84
+          chunked-prefill-size: 65536
+          cuda-graph-max-bs-decode: 544
+          swa-full-tokens-ratio: 0.075
+          enable-hierarchical-cache: true
+          hicache-ratio: 2
+          hicache-write-policy: write_back
+          hicache-io-backend: direct
+          hicache-mem-layout: page_first_direct
+          skip-server-warmup: true
+        max-running-requests: 128
+      env: &dep8_env
+        SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION: '1'
+        # Covers the 8192-token per-rank prefill budget.
+        SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '8320'
+  benchmark:
+    env:
+      <<: &dep8_client
+        AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID: 'true'
+        KV_OFFLOADING: dram
+        TOTAL_CPU_DRAM_GB: '2849'
+      CONC: '64'
+
+override_dep8_c128:
+  frontend: *dep8_frontend
+  roles:
+    agg:
+      args:
+        <<: *dep8_args
+        max-running-requests: 256
+      env: *dep8_env
+  benchmark:
+    env:
+      <<: *dep8_client
+      CONC: '128'
+
+override_dep8_c256:
+  frontend: *dep8_frontend
+  roles:
+    agg:
+      args:
+        <<: *dep8_args
+        max-running-requests: 512
+      env: *dep8_env
+  benchmark:
+    env:
+      <<: *dep8_client
+      CONC: '256'
+
+override_dep8_c384:
+  frontend: *dep8_frontend
+  roles:
+    agg:
+      args:
+        <<: *dep8_args
+        max-running-requests: 768
+      env: *dep8_env
+  benchmark:
+    env:
+      <<: *dep8_client
+      CONC: '384'
+
+override_dep8_c512:
+  frontend: *dep8_frontend
+  roles:
+    agg:
+      args:
+        <<: *dep8_args
+        max-running-requests: 1024
+      env: *dep8_env
+  benchmark:
+    env:
+      <<: *dep8_client
+      CONC: '512'