Currently, to reproduce the leaderboard, we provide this doc page: https://kaiko-ai.github.io/eva/latest/user-guide/advanced/replicate_evaluations/
With the benchmark datasets & models growing, generating the full leaderboard this way becomes challenging.
We should provide scripts to easily run different models with the right configs.
An initial idea:
#!/bin/bash
export MODEL_NAME=kaiko_vitb8
export NORMALIZE_MEAN=[0.5,0.5,0.5]
export NORMALIZE_STD=[0.5,0.5,0.5]
source "$(dirname "$0")/../_run.sh"
run_evals
To facilitate this, we need to first finalize the new models API: #575
Currently, to reproduce the leaderboard, we provide this doc page: https://kaiko-ai.github.io/eva/latest/user-guide/advanced/replicate_evaluations/
With the benchmark datasets & models growing, generating the full leaderboard this way becomes challenging.
We should provide scripts to easily run different models with the right configs.
An initial idea:
To facilitate this, we need to first finalize the new models API: #575