[Bug]: Agentic performance cutoff biases measured input lengths at high concurrency #528

Activity

1. tianmu-li

   added

   type: bugSomething isn't workingSomething isn't working

   status: needs-triageNew issue, awaiting reviewNew issue, awaiting review

   on Oct 5, 2026
2. tianmu-li commented on Oct 5, 2026

   

   tianmu-li

   on Oct 5, 2026

   CollaboratorAuthor

   More actions

   Follow-up: ISL fidelity of the agentic performance window (proposal for #528)

   The current performance-run cutoff disproportionately benefits high concurrency: a high-concurrency run is measured on a much lighter workload than a low-concurrency run of the same dataset. The window under-represents late, long-context turns, so measured input sequence length (ISL) falls by about 3x between a full pass and c=300. Shorter contexts mean less KV cache, faster prefill and faster decode, so throughput and latency at high concurrency look better than the dataset's workload would deliver, and the advantage grows with concurrency. Results at different concurrencies are therefore not comparable. The proposal below keeps measured ISL within about 7-13% of the full-pass value, independent of concurrency, without lengthening the run.

   Measured ISL and OSL in the performance window

   A unit-time scheduler simulation (FIFO completions, encounter-order starts, salting and tool delays off, 613 trajectories, stop_issuing_on_first_user_complete: false) reproduces the coverage and position numbers in the issue:

   Concurrency	Coverage (sim / issue)	Mean tracked turn position (sim / issue)	Mean ISL	Mean OSL
   8	96.1% / 96.1%	65.0 / 65.4	~87.9k	~561
   100	63.5% / -	33.5 / -	~60.6k	~573
   300	22.2% / 23.4%	5.2 / 5.3	~29.8k	~241
   Full pass	100%	67.6	~90.4k	~559

   • Mean ISL is 67% lower at c=300 than for the full pass (29.8k vs 90.4k). Mean OSL also drops, to ~241 from ~559.
   • A measured run shows the same pattern (80.6k at c=8, 60.9k at c=100, 32.5k at c=300). The simulated c=100 value is within 0.5% of the measured one; c=8 is about 9% higher and c=300 about 8% lower, consistent with real completion order and coverage differences.
   • The full-population mean position is 67.6 (20,297 client turns counted as assistant rows; the issue counts 20,403, so the counting rule differs slightly).

   ISL and OSL are approximate, computed from the dataset text with the Qwen/Qwen3.6-35B-A3B tokenizer. ISL for a turn is the token count of the system prompt and tool definitions plus every earlier row in the trajectory (user text, assistant reasoning, content and tool calls, tool results), with a fixed 6 tokens of per-message template overhead. Chat-template rendering is not applied. OSL is the token count of the turn's reasoning, content and serialized tool calls as recorded in the dataset, not what a model would generate.

   A measured run is consistent with this. Between c=100 and c=300 the throughput ratio (TPS at c=300 / TPS at c=100 = 1089.58 / 647.29 = 1.683) is close to the inverse ratio of mean ISL (60,862 / 32,479 = 1.874). This is a correlation between two measured ratios, not a controlled attribution: some throughput gain at higher concurrency is real, and the split between the two effects is not quantified here.

   Why the window is ISL-biased

   ISL grows with turn position because agentic context is cumulative. The dataset is two very different populations:

   Subset	Trajectories	Client turns	Turns per trajectory (min / median / mean / max)
   Workflow (Workato)	500	4,316	4 / 8 / 8.6 / 16
   Coding	113	15,981	37 / 139 / 141 / 304

   Coding trajectories are 18% of trajectories but 79% of turns, and they carry the long contexts. Every concurrency slot starts at turn 1 at the same time, and _finish_conversation stops tracking at the first completion after the last start. With a 613-trajectory budget at c=300 that happens after roughly 313 completions, while the long trajectories are still in their early (short-context) turns. Higher concurrency means fewer completions before the budget is exhausted, so a larger share of the measured turns are early ones.

   The run is also tail-bound: the longest trajectory has 304 turns, so a full pass takes at least 304 rounds, and at c=300 average concurrency over a full pass is about 22% of the target. Draining the tail (stop_issuing_on_first_user_complete: false) is therefore slow and mostly runs below the requested concurrency.

   What does not fix it

   • Requiring N >= 3x or 5x concurrency. With the cutoff and every slot starting at turn 1, mean tracked position is 7.9 at N/c = 3 and 24 at N/c = 5 (full = 67.6).
   • Stratified or representative sampling of trajectories. Useful for partial budgets (a prefix of the file over- or under-represents the long trajectories), but the bias remains with the full 613 because it comes from the time cutoff, not from which trajectories are chosen.
   • Starting slots mid-trajectory. It removes the position bias in simulation, but the first request per slot has a long history and a cold prefix cache, so it is not representative without extra warm-up handling.
   • Tracking every turn of every started trajectory (drain the tail). Correct, but runtime becomes tail-bound and most of it runs below the target concurrency.

   Proposal

   1. Assert a large N/c. Reject runs below N/c = 10 at config validation and warn below 20.
   2. Set stop_issuing_on_first_user_complete: true for agentic performance runs. The run ends at the cutoff with concurrency still full, with no tail drain.
   3. Drop the inline accuracy scorer (agentic_inference_inline) for this workload. Use standalone SWE-bench for accuracy, and use the performance run's mean OSL as a sanity check only.

   Simulation of the proposed setup (stop at cutoff, all slots start at turn 1, trajectories resampled from the dataset's turn-count distribution; full pass = 90.4k ISL, 559 OSL, position 67.6):

   c	N/c	Mean position	Mean ISL	Mean OSL	Rounds (unit-time turns)
   100	10	53.2	~78.9k	~578	~270
   100	20	61.2	~85.3k	~566	~600
   300	10	53.8	~78.2k	~551	~270
   300	20	60.3	~83.6k	~553	~600

   Position by N/c (independent of c; 12 seeds):

   N/c	Mean position	Rounds
   3	7.9	~31
   5	24	~91
   10	54	~270
   15	59	~435
   20	61	~600
   30	64	~935

   • Mean ISL is about 13% below the full pass at 10x and about 7% below at 20x, against 67% below at c=300 today. ISL at c=100 and c=300 agrees to within about 2%, so a fixed N/c makes runs comparable across concurrencies.
   • 10x is a compromise. It is comparable across concurrencies but still understates ISL by about 13%. 20x roughly halves that gap at about twice the run time.
   • Mean OSL stays within about 4% of the full pass at both settings, so an OSL sanity check is usable at either.

   Approximate tracked volume (c=200 is scaled linearly from the c=100 and c=300 simulations):

   c	N/c	Trajectories started	Tracked turns	Rounds
   100	10	1,000	~27k	~270
   100	20	2,000	~60k	~600
   200	10	2,000	~54k	~270
   200	20	4,000	~120k	~600
   300	10	3,000	~80k	~270
   300	20	6,000	~179k	~600

   Sample counts are not the limiting factor; bias is. Run length in rounds depends only on N/c. Real wall-clock depends on the endpoint, since context grows with position.

   Required supporting changes

   • Inline accuracy and stop_issuing_on_first_user_complete. The inline score is computed over issued scorable turns, so stopping early does not create "missing" turns, but the score covers a biased subset (early turns, short workflow trajectories) and is not comparable to a full-pass score. Rejecting the combination at config validation (or dropping the inline scorer) avoids a misleading number.
   • Run validity without scores.json. check_perf_validity reads the turns block that only the inline scorer writes, and check_submission silently skips it when the file is absent. Replace it with a dropped-turn check sourced from the metrics report.
   • Workflow accuracy. The Workato intent-match score goes away with the inline scorer. If the workflow domain needs an accuracy gate, it needs a separate full-pass run or scorer.
   • Report fields. Report the achieved N/c and the window's mean turn position and mean ISL next to results, and document that results are comparable only between runs with the same N/c and subset.
   • Mean OSL. Treat it as a sanity check for empty or truncated outputs, not as an accuracy proxy. It depends on window composition in the same way as ISL, so compare only against a run with the same N/c.

   Optional follow-ups

   • Subset selector (coding vs workflow). Workflow alone has negligible ramp bias at N/c >= 5 and short runs; coding alone needs N/c of about 3-4 to get 83% coverage (at the cost of long runs). Splitting makes each number interpretable but loses mixed-load interference, so it is best as a diagnostic.
   • Warm-up discard. Skip tracking until about k x c trajectories have completed. This would let a lower N/c (for example 10) behave more like 20 without starting mid-trajectory.
   • Stratified start order for partial budgets. Seeded, stratified by turn count, so small budgets reflect the dataset's short/long mix.

   Caveats

   • All numbers are from a unit-time simulation (one turn per round for every active trajectory), not from a real endpoint. Real turn times grow with context length, which weights wall-clock time toward late positions. Validate on a real endpoint at c=300 before fixing thresholds.
   • ISL and OSL are token-count approximations from the dataset text, not from the serving stack's rendered prompts.
3. tianmu-li commented on Oct 5, 2026

   

   tianmu-li

   on Oct 5, 2026

   CollaboratorAuthor

   More actions

   @arekay-nv @hvagadia @leopck Appreciate feedback

   👀React with 👀1Reacted by Stanley Phoong
4. hvagadia commented on Oct 5, 2026

   

   hvagadia

   on Oct 5, 2026

   Collaborator

   More actions

   Thanks for the investigation. Can you explain how does this help?

   Set stop_issuing_on_first_user_complete: true for agentic performance runs. The run ends at the cutoff with concurrency still full, with no tail drain.
5. tianmu-li commented on Oct 5, 2026

   

   tianmu-li

   on Oct 5, 2026

   CollaboratorAuthor

   More actions

   Thanks for the investigation. Can you explain how does this help?

   Set stop_issuing_on_first_user_complete: true for agentic performance runs. The run ends at the cutoff with concurrency still full, with no tail drain.

   Issue with doing tail drain but with N/c very large is that the tail drain becomes long while not contributing to measured performance.
6. hvagadia commented on Oct 5, 2026

   

   hvagadia

   on Oct 5, 2026

   Collaborator

   More actions

   We do have a minimum runtime constraint right? Does that implicitly encourage higher N/c ratios?
7. tianmu-li commented on Oct 5, 2026

   

   tianmu-li

   on Oct 5, 2026

   CollaboratorAuthor

   More actions

   The duration floor alone doesn't fix ISL fidelity and rewards the smallest trajectory count that satisfies it. E.g: if concurrency=300, num_trajectories_to_issue=613, and the run takes over 10 minutes, ISL is undercount by >3x and there is no incentive to increase N/c further.
8. hvagadia commented on Oct 6, 2026

   

   hvagadia

   on Oct 6, 2026

   Collaborator

   More actions

   I am hesitant to completely remove inline accuracy and mean osl. How about we enforce a min runtime on the performance phase (not the whole run)? And set it to greater than 10 mins? Basically we want to ensure we cycle through the full dataset once in the performance phase.
9. tianmu-li commented on Oct 6, 2026

   

   tianmu-li

   on Oct 6, 2026

   CollaboratorAuthor

   More actions

   1. A minimum runtime alone doesn't enforce fair ISL. It bounds how long the window lasts, not which turns are in it. At c=300 with N=613 (N/c ≈ 2), mean window ISL is about 3x below the full pass (29.8k vs 90.4k, simulated), and a run that just clears a 10-minute floor would still be valid. Counting turns doesn't fix it: at N/c = 5 the window has over one pass of turns and ISL is still 1.9x under.
   2. Fair ISL needs N/c of about 10 or more. Mean window ISL is within about 13% of the full pass at N/c = 10 and about 10% at N/c ≈ 15, independent of concurrency.
   3. Mean OSL and inline accuracy can stay as sanity checks if the tail drain is removed, but not as gates. They would be computed on the performance window, not the full dataset, so keep gating accuracy on SWE-bench.
   4. An ISL check could enforce fairness directly: require the window's mean ISL to be within a tolerance of the dataset's full-pass mean ISL, and report both.
10. tianmu-li commented on Oct 6, 2026

    

    tianmu-li

    on Oct 6, 2026

    CollaboratorAuthor

    More actions

    One concrete example: a system reaches 1089 tokens/s at concurrency 300 using 613 trajectories but only 657 tokens/s using 6133 trajectories at the same 300 concurrency. Both pass the min duration requirement. The 6133 result is probably still an overestimate of the performance BTW.
11. hvagadia commented on Oct 7, 2026

    

    hvagadia

    on Oct 7, 2026

    Last edited by hvagadia

    Collaborator

    More actions

    Hi Tianmu, we evaluated this issue and agree with your assessment. I think we need to think of a short-term solution that needs to be in effect on Oct 12th, and a longer term solution to resolve this.

    Here are my thoughts on short term solutions:

    1. We cannot really cut off the tail because it will require re-working and aligning accuracy with everyone.
    2. Including the tail, the best solution is to set a threshold on N/c and/or min-runtime (for the perf phase).
    3. We are still evaluating the right ratios that work for us, but N/c >= 10 will likely not work due to long runtime. Somewhere around 5 would be appropriate.

    For longer term solution:

    1. We should rework our perf window to start somewhere in the middle of trajectories.
    2. We should reduce the dependency of accuracy on full dataset and come up with a method that works on partial trajectories.

    I want to note that this is an issue that also affects AgentX and AA-Agentperf, there is some amount of approximation that we would need to live with.

    So my proposal would be to add a rule and enforce a min value of N/c and/or perf-window-runtime. We can get back with the thresholds soon.