fix(eval,cookbooks,ci): harden evaluation streaming, multi-node checks, and preview generation - #333
Open
umran666 wants to merge 1 commit into
Open
Conversation
…s, and preview generation ## Summary - evaluation/unigenbench: handle empty streaming chunks, add exponential backoff on retries, clean up async client on close, record failed samples. - evaluation/paibench_g: support partial result persistence on worker failure so completed metrics are not discarded. - evaluation/paibench_c: clamp evaluation GPUs against CUDA_VISIBLE_DEVICES to prevent oversubscribing workers. - cookbooks/action: fail fast with actionable error when multi-node env vars are partially set. - cookbooks/transfer: make preview mp4 generation atomic to prevent partial file writes. - ci: guard notebook validation against empty file sets. Signed-off-by: umran666 <shaikumran666@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While running through the evaluation benchmarks and fine-tuning cookbooks, I ran into a few edge cases and stability issues where transient network hiccups, gateway idiosyncrasies, or missing env vars caused jobs to fail or abort before writing aggregate results.
choiceschunks emitted by some OpenAI-compatible VLM gateway endpoints (e.g., terminal usage-only frames) to avoid raisingIndexErrormid-stream.vlm.close()to prevent lingering connection warnings upon interpreter exit.--vlm_timeout(default 300s, up from hard-coded 100s) and--vlm_max_retry(default 3) so slower judge completions aren't prematurely dropped and re-prompted.ugb_scorer.pynow defensively parses both native python lists and legacy stringified list representations."failed_samples"key in the result JSON (alongsidesuccess_count) instead of being silently omitted.VIDEO_RETRIES = 1) so transient AMT scoring failures get a second attempt before failing the run."missing_videos"in the result JSON, rather than aborting prior to the merge step.CUDA_VISIBLE_DEVICESto clamp evaluation GPU workers to the actual number of visible devices, preventing oversubscription whenCOSMOS3_NUM_GPUSexceeds the physical count.run_paibench_c.shand added an explicit 1-sample smoke-test example.cookbooks/cosmos3/generator/action/finetune/*.shwhenNNODES,NODE_RANK, orMASTER_ADDRare partially set, surfacing a clear error before torchrun's rendezvous layer fails.preview_helpers.pynow encodes to a sibling.partial.mp4first and swaps atomically withos.replace, preventing corrupt/incomplete preview files if an interactive cell is interrupted..github/workflows/validate-notebooks.ymlsoruffexits cleanly when no notebooks match the glob.evaluation/cosmos3/generator/unigenbench/query_core_selfcheck.pyandevaluation/cosmos3/generator/paibench_g/merge_result_selfcheck.py(stdlib-only assert tests, requiring no GPU or network) to verify streaming chunk parsing and merge aggregation edge cases.