Skip to content

patch the dataset sample progress bar - #516

Merged
zihaok merged 4 commits into
mlcommons:mainfrom
zihaok:zihaok_fix_dataset_count_patch
Sep 23, 2026
Merged

zihaok merged 4 commits into
mlcommons:mainfrom
zihaok:zihaok_fix_dataset_count_patch

Conversation

@zihaok

@zihaok zihaok commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

this PR fixes the progress bar display bug when issued trajectory is over the agentic performance dataset total trajectory count.
Without the fix, after 20403 turns, the display will output malformed line.

2026-09-22 10:13:07,922 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True):  99%|█████████▉| 20230/20403 [30:25<00:15, 11.08it/s]
2026-09-22 10:13:17,956 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True): 100%|█████████▉| 20359/20403 [30:35<00:03, 11.09it/s]
2026-09-22 10:13:29,327 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True): 20501it [30:45, 11.11it/s]                           
2026-09-22 10:13:40,357 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True): 20643it [30:57, 11.11it/s]
2026-09-22 10:13:50,596 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True): 20768it [31:08, 11.12it/s]
2026-09-22 10:14:01,009 - sflow.task.endpoint_client - INFO - 0: deepseek-ai/DeepSeek-V4.1-Flash (Streaming: True): 

However, this fix is not a long term elegant fix because the existing flag num_trajectories_to_issue: 613 should logically belong to

settings:
  load_pattern:
    type: agentic_inference
    target_concurrency: ${CONCURRENCY}
    num_trajectories_to_issue: 613

instead of under

datasets:
  - name: mlperf_agentic_inference_dataset
    type: performance
    path: ${DATA_PATH}/perf/mlperf_agentic_inference_dataset.jsonl
    accuracy_config:
      eval_method: agentic_inference_inline
    agentic_inference:
      ......
      num_trajectories_to_issue: ${NUM_TRAJECTORIES}

In order to not break everyone's client.yaml during loading, I decided to implement this temp patch to make the API not changing. however, after this round of submission. This needs a broder refactor.

Filed an issue to track later

#517

Type of change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor/cleanup

Related issues

Testing

  • Tests added/updated
  • All tests pass locally
  • Manual testing completed

Checklist

  • Code follows project style
  • Pre-commit hooks pass
  • Documentation updated (if needed)

@zihaok
zihaok requested a review from a team September 22, 2026 21:07
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Sep 22, 2026
@codecov-commenter

codecov-commenter commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@cec2c28). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #516   +/-   ##
=======================================
  Coverage        ?   80.92%           
=======================================
  Files           ?      154           
  Lines           ?    21247           
  Branches        ?        0           
=======================================
  Hits            ?    17194           
  Misses          ?     4053           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zihaok
zihaok requested a review from arekay-nv September 23, 2026 21:40

@arekay-nv arekay-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread src/inference_endpoint/load_generator/agentic_inference_strategy.py
@zihaok
zihaok merged commit e71b928 into mlcommons:main Sep 23, 2026
9 checks passed
@zihaok
zihaok deleted the zihaok_fix_dataset_count_patch branch September 23, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants