feat(benchmark): add HLE eval script for NeMo Skills

[AlphaBladez]

Summary

srt-slurm has no way to run HLE (Humanity's Last Exam). The structured eval runners cannot provide it: sglang.test.run_eval has no HLE task in any release, and sgl-eval does not ship one either. NeMo Skills does, including the LLM judge HLE needs, because many of its answers are free-form and cannot be scored with a regex.

This PR adds HLE the way AIME already runs (#91): a script under configs/hle/ that recipes invoke with benchmark.type: custom in the NeMo Skills container. There are no changes under src/, no new benchmark type and no new schema fields.

benchmark:
  type: custom
  container_image: nemo-skills    # alias from srtslurm.yaml `containers:`
  env:
    MODEL: "Qwen/Qwen3-0.6B"      # the served model name
    NUM_EXAMPLES: "50"            # optional; omit for the whole split
  command: bash /configs/hle/run.sh

Changes

• configs/hle/run.sh: ns prepare_data hle, then ns eval --benchmarks=hle:<REPEAT> --split=text against the in-job endpoint (localhost:8000/v1, as in AIME).
  • Answers are graded by NeMo Skills' HLE judge. Its default is o3-mini-2025-01-31 on api.openai.com, the judge of the official leaderboard, which needs OPENAI_API_KEY. JUDGE_MODEL, JUDGE_SERVER_ADDRESS and JUDGE_SERVER_TYPE point it at another OpenAI-compatible server instead.
  • Knobs are env vars: MODEL (required), ENDPOINT, SPLIT, NUM_EXAMPLES, REPEAT, MAX_TOKENS, NUM_THREADS, TEMPERATURE, TOP_P, SEED, OUTPUT_DIR. Sampling and timeout defaults match configs/aime/run.sh. REPEAT defaults to 1 because every repeat is another judge call per question.
  • SEED defaults to 0. NeMo Skills 26.03's judge step reads output-rs0.jsonl, so a non-zero starting seed fails the judge after generation has finished.
  • The script fails before generation when MODEL is unset, or when the default judge is used without OPENAI_API_KEY. It exits non-zero when eval-results/hle/metrics.json is missing at the end, because ns can exit 0 after a failed prepare_data or judge step.
• Example: examples/features/hle.yaml, Qwen3-0.6B on two aggregated SGLang workers behind sglang-router, evaluated on 50 questions.
• Docs: an HLE section in docs/accuracy.md covering prerequisites (an OpenAI key for the official judge; a Hugging Face token whose account has accepted the terms of the gated cais/hle dataset), the recipe shape, credential handling and outputs. Also the nemo-skills alias and features-table row in examples/README.md, and a comment in srtslurm.yaml.example.
• Tests: tests/test_hle_script.py runs run.sh against a stub ns and checks the ns arguments, the judge override, both early failures and the missing-metrics failure. It also loads the example recipe.

Credentials are exported in the submitting shell and inherited by the job. They are never written into the recipe: benchmark.env values are passed literally, so ${HF_TOKEN} there would not expand.

Launch snapshot diff

One new file, tests/snapshots/launch/features__hle.txt, and no existing snapshot changes. The benchmark step runs bash -lc 'bash /configs/hle/run.sh' in the nemo-skills image.

Validation

• tests/test_hle_script.py and tests/test_launch_snapshots.py pass; Ruff and srtctl schema-docs --check are clean.
• A cluster run of the example with lmsysorg/sglang:v0.5.21 and nvcr.io/nvidia/eval-factory/nemo-skills:26.03 led to the second commit. It removes disable-piecewise-cuda-graph, which v0.5.21 rejects, and adds the SEED default and missing-metrics check described above.
• Upstream behavior is cited from NeMo Skills at commit 5a17f307. That covers the hle dataset's text split and the judge defaults in nemo_skills/dataset/hle/__init__.py.
• Not done: a full-split run graded by the official o3-mini judge.