[Bug]: Agentic 10 h max_issue_duration_ms cannot complete the required full pass on bandwidth-limited systems for Atlas Inference, longest turn exceeds Qwen3.6's 262,144 context with max_new_tokens 8192 #534

Bug Description

1. The 10 h issue cap conflicts with the full-pass requirement.
qwen_agentic_benchmark.yaml sets max_issue_duration_ms: 36000000 # 10-hour cap on the performance phase. Endpoints Rules §6.4 requires every point to issue a positive integer multiple of the dataset size. When the cap is reached, issuing stops as a normal phase end (session.py), so a run that can't finish a pass in 10 h ends partway through it.

A cap is also effectively mandatory: rules.py asserts that max_issue_duration_ms is not None, while the default ruleset value is None. A submitter therefore has to pick a value, and the rules don't say what an acceptable value is.

On a single-accelerator, bandwidth-limited system (DGX Spark / GB10, 273 GB/s), long-context decode is bound by reading the KV cache. With within-conversation and system-prompt prefix caching, I see one full pass at roughly 15–40 h per point, even at C=4 (details below). Every point on a 7-point curve would hit the 10 h cap.

2. The longest turn cannot fit in Qwen3.6's context with max_new_tokens 8192.
Rendered with the Qwen3.6 chat template, the longest client turn in the official dataset is about 254,329 input tokens (before the salt). With max_new_tokens: 8192 that comes to about 262,521 tokens, roughly 377 over the model's 262,144 limit. Depending on the server, that request is rejected, has its output silently clamped, or fails mid-trajectory. Rules line 631 only says a submitter's truncation must match the reference implementation's. The reference doesn't appear to define what happens in this case.

cc @arekay-nv

Is raising max_issue_duration_ms above 10 h allowed for official agentic runs? If so, is there an upper bound? If yes and none, could the example comment say so, or could the cap default to unlimited for agentic so the full-pass requirement decides the run length?

Steps to Reproduce

1. Download mlperf_agentic_inference_dataset.jsonl (SHA-256 1beb24c8…14ae).
2. Build each client turn's messages as agentic_inference_dataset.py does, then render them with the Qwen/Qwen3.6-35B-A3B chat template (tools + system + history) and tokenize.
3. Take the maximum ISL and add max_new_tokens (8192).
4. For (1): sum the per-turn new-prefill tokens (with prefix caching) and the decode tokens. Then estimate the pass time from the system's prefill rate and KV-bandwidth-bound decode rate.

Environment

• Upstream commit: d3c57f2
• Dataset: 613 trajectories / 20,403 client turns, SHA-256 1beb24c882122df96571cf11b390acbea388944038bc55c78b891475459014ae
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4 (approved checkpoint), fp8 KV
• Target system: NVIDIA DGX Spark (GB10, 128 GB unified memory)
• Analysis was offline: tokenizer, chat template and an event-driven slot simulation. No endpoint was used.

Relevant Logs

Workload per full pass (template-rendered token counts):

| Quantity | Value |
| --- | --- |
| Input tokens, no caching | ~1.68 B |
| New prefill with within-conversation prefix caching | ~36.8 M |
| … plus Workato system-prompt sharing across conversations (§2.9.5) | ~21 M |
| Recorded output tokens (mean 514 per turn) | ~10.5 M |
| Injected tool delay | ~8.9 h |
| ISL p50 / mean / p90 / max | 70k / 82k / 164k / 254,329 |

Estimated time for one full pass on GB10 (optimistic / mid / pessimistic):

| C | Hours |
| --- | --- |
| 4 | 18.4 / 26.4 / 40.7 |
| 16 | 16.6 / 23.9 / 36.9 |
| 48 | 15.2 / 21.9 / 33.8 |

System TPS stays roughly flat across concurrency (~105–140 tok/s), because each decode step reads every active conversation's KV cache. Adding concurrency doesn't shorten the pass.

Before submitting

• I searched existing issues and found no duplicates