Notable changes to NeMo Eval Author are recorded here. Eval Author is a research preview: workflows, skills, and saved artifacts may change between releases.
Initial research-preview release, tested with Harbor 0.20.0 and NeMo Gym v0.6.0.
To install exactly this version, clone the v0.1.0 tag and follow the
manual installation steps.
- Eleven skills that install together, with
eval-authoras the entry point that selects a workflow and saves its output under.eval-author/. See Get started and the skill reference. - Ethos authoring with the bundled
ethosskill, which records your agent's purpose, boundaries, and success criteria in a localETHOS.mdthat first evaluations and audits build on. - First evaluations for NeMo Gym or Harbor. Agree on a reviewed plan, build a starter suite in a proven environment, and validate each task before evaluating your agent. See Build your first evaluations.
- Discovery and readiness checks. Inventory the evaluations in a repository and check whether its Gym and Harbor suites are ready to run. See Find existing evaluations and Check readiness.
- Guided coverage audits. Define what your evaluations should cover, measure coverage from ATIF traces, and review gaps in a readable Audit coverage report. See Audit coverage.
- Proposals and task creation. Turn audit findings into ranked recommendations, and create a Gym or Harbor task for an eligible, measured tool-coverage gap. See Propose new evaluations and the Gym task-authoring guide.
- Reviewable, rerunnable suites. Each generated suite gets a
README.mdwith a case inventory and rerun commands, and Gym records and ATIF traces open in an offline HTML reader. See Review generated cases and Open readable cases and traces. - Trace tools. Explain a NeMo Intake trace and convert MLflow and Gym traces to ATIF. See Inspect an Intake trace and Convert local traces.
- Experimental trace-to-environment workflow, which derives a private Harbor or native Gym task from trace evidence. See Build an evaluation task.