Evaluation, Reproducibility, Benchmarks Meeting 35

Minutes of Meeting 35

Date: 28th May, 2025

Present

• Carole
• Annika
• Anne
• Olivier
• Nick
• Michela
• Nicola

Metrics Reloaded Implementations

• Michela and Olivier working together on this
• Goal is to run metrics implementations on all cases from the decathlon
• This large scale test has allowed them to learn a lot
  • Relatively slow, especially when running it on a lot of data
    • Parallelization is key -- lots of data usually implies more compute will be available
  • Found three specific errors
• There was some discussion about possible optimization strategies, what the code currently does to improve speed, and what to do about these few errors that are being worked through
• Regarding publication -- how do we pitch this?
  • Useful to show that metric distributions are not Gaussian
  • Could show implications of the true distributions on CIs, mean vs. median dilemma
  • Overall, we have much more evidence now than we had for the MICCAI paper, but we still need a consensus process if we're going to pitch this as guidelines
    • Should we start this now?
• Is dependent data part of the scope?
  • Could be challenging
  • We don't have to decide this now

Next Steps

• Olivier will prepare a 1-pager to circulate within the working group
• Each of us suggests some number of experts to use for the consensus process
• Each suggested expert would be given the opportunity to suggest additional experts

Follow-up On Data Licenses Project

• Annika is about 70% of the way through resolving conflicts between raters

Follow-up From Last Time

• MICCAI paper is in rebuttal
• WG website swapped links -- this appears to be fixed
• WG website suggestion box?
  • It sounds like they would like us to fork the repo and PR any changes in
• Next generation of the BIAS initiative?
  • Talk about this next time
  • Annika has looked into other similar guidelines, borrowing items that we seem to be missing

Upcoming Meetings

• Summer break is coming up
• Maybe we should re-arrange some of these monthly meetings
  • Keep 16th July
  • Move the August meeting earlier
  • Annika won't be available next month