Skip to content

Add record_datasets_as_runs setting to trainer - #996

Merged
MaxFeucht merged 10 commits into
mainfrom
995-evals-where-multiple-prediction-datasets-are-not-supported
Apr 22, 2026
Merged

Add record_datasets_as_runs setting to trainer#996
MaxFeucht merged 10 commits into
mainfrom
995-evals-where-multiple-prediction-datasets-are-not-supported

Conversation

@nkaenzig

@nkaenzig nkaenzig commented Mar 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a record_datasets_as_runs parameter to Trainer for evaluation sessions with multiple val and/or test datasets.
  • When enabled, each dataset output is recorded as a separate run in the session summary instead of being kept as separate datasets within a single run.
  • Passes the setting through the Interface CLI entry points and run_evaluation_session.

How to use

Set record_datasets_as_runs: true in the trainer config, and provide a list of datasets in the val or test section of the data config. Each dataset entry can point to a different data path, for example predictions from different offline runs:

trainer:
  class_path: eva.Trainer
  init_args:
    record_datasets_as_runs: true
    # ... other trainer args

data:
  class_path: eva.DataModule
  init_args:
    datasets:
      val:
        - class_path: eva.language.datasets.TextPredictionDataset
          init_args:
            path: ./predictions/run_1/manifest.jsonl
            split: val
        - class_path: eva.language.datasets.TextPredictionDataset
          init_args:
            path: ./predictions/run_2/manifest.jsonl
            split: val
        - class_path: eva.language.datasets.TextPredictionDataset
          init_args:
            path: ./predictions/run_3/manifest.jsonl
            split: val

In the generated results.json, metrics are reported as one dataset entry whose values list contains the per-dataset run values together with the aggregated mean and stdev:

{
  "metrics": {
    "val": [
      {
        "val/AverageLoss": {
          "mean": 1.58,
          "stdev": 0.0,
          "values": [1.60, 1.55, 1.59]
        },
        "val/BinaryBalancedAccuracy": {
          "mean": 0.50,
          "stdev": 0.01,
          "values": [0.49, 0.51, 0.50]
        }
      }
    ]
  }
}

@nkaenzig nkaenzig linked an issue Mar 11, 2026 that may be closed by this pull request
@nkaenzig
nkaenzig marked this pull request as draft March 11, 2026 16:36
@nkaenzig
nkaenzig marked this pull request as ready for review March 11, 2026 17:05
@nkaenzig
nkaenzig requested a review from MaxFeucht March 11, 2026 17:05
@nkaenzig nkaenzig changed the title Add combine_dataloader_results setting to trainer Add record_datasets_as_runs setting to trainer Mar 12, 2026
@nkaenzig nkaenzig self-assigned this Mar 24, 2026
@DriessenA

DriessenA commented Apr 14, 2026

Copy link
Copy Markdown

@nkaenzig Any chance we can merge this PR? I'd be happy to help if needed :)

@MaxFeucht MaxFeucht left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MaxFeucht
MaxFeucht merged commit 0707872 into main Apr 22, 2026
8 checks passed
@MaxFeucht
MaxFeucht deleted the 995-evals-where-multiple-prediction-datasets-are-not-supported branch April 22, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals where multiple prediction datasets are not supported

3 participants