This directory contains the evals for the A*PA paper.

• experiments/ contains .yaml files with experiment configurations, roughly corresponding to the plots in the paper. (Some .yaml files are the source of multiple plots.)
• Use make download_results to download and extract the results.zip from the astarpa-evals release on github.
• Alternatively, make run reruns all experiments locally, but be aware that this takes a long time (1-2 days). You could make things faster by running more in parallel (add e.g. -j 5 to the run rule in makefile), or by lowering the number of sequences tested (e.g. lower total_size in experiments/*.yaml).
• evals.ipynb contains all plots and other statistics presented in the paper from the results/*.json files generated by one of the two steps above. The version in the repo should correspond to the latest executed version and include all plots.

note: the git history of the evals.ipynb file has been squashed to reduce the size of the repo.