Skip to content

Export model predictions from benchmark runs - #42

Draft
adrian-prior wants to merge 1 commit into
adrian/prediction-artifactsfrom
adrian/prediction-artifact-runner
Draft

Export model predictions from benchmark runs#42
adrian-prior wants to merge 1 commit into
adrian/prediction-artifactsfrom
adrian/prediction-artifact-runner

Conversation

@adrian-prior

Copy link
Copy Markdown
Collaborator

Generated by Codex

Export model validation and test predictions with --predictions-dir so results can be evaluated offline. Add --refit-all-configs to generate test artifacts for successful configs beyond the selected/default configs. Keep extra fit/predict timings in artifact metadata, separate from benchmark result timings.

Behavior and validation
  • Normal export mode reuses tuning predictions and selected/default test predictions without adding fits.
  • Separate benchmark and extra-refit loops make the timing policy explicit.
  • Shared validation labels follow the evaluated inner rows; test labels are exported after prediction.
  • Extra timings cover fit and predict calls, not scoring or artifact I/O.
  • Tests verify classification and regression round trips, metric reproduction from saved labels, row/class/split mismatch rejection, label corruption detection, fit counts and timing separation.
  • Full stack: 440 tests pass, 11 skip. All pre-commit hooks, pinned Ruff checks, distribution build and required-package-data verification pass on Python 3.11.

@adrian-prior
adrian-prior added this pull request to stack #43 September 11, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant