Skip to content

fix(eval,cookbooks,ci): harden evaluation streaming, multi-node checks, and preview generation - #333

Open
umran666 wants to merge 1 commit into
NVIDIA:mainfrom
umran666:fix/pipeline-robustness-and-evaluation
Open

fix(eval,cookbooks,ci): harden evaluation streaming, multi-node checks, and preview generation#333
umran666 wants to merge 1 commit into
NVIDIA:mainfrom
umran666:fix/pipeline-robustness-and-evaluation

Conversation

@umran666

Copy link
Copy Markdown

While running through the evaluation benchmarks and fine-tuning cookbooks, I ran into a few edge cases and stability issues where transient network hiccups, gateway idiosyncrasies, or missing env vars caused jobs to fail or abort before writing aggregate results.

  • UniGenBench Gateway Streaming & Teardown:
    • Skips empty choices chunks emitted by some OpenAI-compatible VLM gateway endpoints (e.g., terminal usage-only frames) to avoid raising IndexError mid-stream.
    • Adds exponential backoff between retries so rate-limited gateways aren't hammered.
    • Explicitly closes the underlying async HTTP client on vlm.close() to prevent lingering connection warnings upon interpreter exit.
  • UniGenBench Judge Tuning & Metrics:
    • Exposes CLI args --vlm_timeout (default 300s, up from hard-coded 100s) and --vlm_max_retry (default 3) so slower judge completions aren't prematurely dropped and re-prompted.
    • ugb_scorer.py now defensively parses both native python lists and legacy stringified list representations.
    • Samples whose retries are exhausted are recorded under a "failed_samples" key in the result JSON (alongside success_count) instead of being silently omitted.
  • PAIBench-G Retry & Partial Results:
    • Added per-video retry (VIDEO_RETRIES = 1) so transient AMT scoring failures get a second attempt before failing the run.
    • If a worker or video scoring fails, the script now writes whatever was cleanly scored and records the remaining entries under "missing_videos" in the result JSON, rather than aborting prior to the merge step.
  • PAIBench-C GPU Clamping & Docs:
    • Parses CUDA_VISIBLE_DEVICES to clamp evaluation GPU workers to the actual number of visible devices, preventing oversubscription when COSMOS3_NUM_GPUS exceeds the physical count.
    • Fixed contradictory quick-start comments in run_paibench_c.sh and added an explicit 1-sample smoke-test example.
  • SFT Launch Script Guards:
    • Added fail-fast parameter validation in cookbooks/cosmos3/generator/action/finetune/*.sh when NNODES, NODE_RANK, or MASTER_ADDR are partially set, surfacing a clear error before torchrun's rendezvous layer fails.
  • Atomic Video Previews:
    • preview_helpers.py now encodes to a sibling .partial.mp4 first and swaps atomically with os.replace, preventing corrupt/incomplete preview files if an interactive cell is interrupted.
  • CI Workflow Guard:
    • Added an empty-array guard in .github/workflows/validate-notebooks.yml so ruff exits cleanly when no notebooks match the glob.
  • Self-Checks:
    • Added evaluation/cosmos3/generator/unigenbench/query_core_selfcheck.py and evaluation/cosmos3/generator/paibench_g/merge_result_selfcheck.py (stdlib-only assert tests, requiring no GPU or network) to verify streaming chunk parsing and merge aggregation edge cases.

…s, and preview generation

## Summary
- evaluation/unigenbench: handle empty streaming chunks, add exponential backoff on retries, clean up async client on close, record failed samples.
- evaluation/paibench_g: support partial result persistence on worker failure so completed metrics are not discarded.
- evaluation/paibench_c: clamp evaluation GPUs against CUDA_VISIBLE_DEVICES to prevent oversubscribing workers.
- cookbooks/action: fail fast with actionable error when multi-node env vars are partially set.
- cookbooks/transfer: make preview mp4 generation atomic to prevent partial file writes.
- ci: guard notebook validation against empty file sets.

Signed-off-by: umran666 <shaikumran666@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant