Skip to content

Provide scripts for running model evals #592

Description

@nkaenzig

Currently, to reproduce the leaderboard, we provide this doc page: https://kaiko-ai.github.io/eva/latest/user-guide/advanced/replicate_evaluations/

With the benchmark datasets & models growing, generating the full leaderboard this way becomes challenging.
We should provide scripts to easily run different models with the right configs.

An initial idea:

#!/bin/bash    

export MODEL_NAME=kaiko_vitb8
export NORMALIZE_MEAN=[0.5,0.5,0.5]
export NORMALIZE_STD=[0.5,0.5,0.5]

source "$(dirname "$0")/../_run.sh"
run_evals

To facilitate this, we need to first finalize the new models API: #575

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions