Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

232 Open 180 Closed
232 Open 180 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Fix broken workflows chore Thing that needs doing
#892 opened Mar 10, 2025 by rogthefrog
@rogthefrog
1
pick chinese SUTs
#890 opened Mar 7, 2025 by wpietri
create chinese practice set
#889 opened Mar 7, 2025 by wpietri
evaluator proxy dynamically starts and stops instances
#888 opened Mar 7, 2025 by wpietri
benchmark Deepseek R2
#887 opened Mar 4, 2025 by wpietri
Chinese prototype milestone A goal that other cards add up to
#886 opened Mar 4, 2025 by wpietri
Hindi prototype milestone A goal that other cards add up to
#885 opened Mar 4, 2025 by wpietri
Change % safe to % violating throughout chore Thing that needs doing
#884 opened Mar 4, 2025 by wpietri
Add param to run-csv-items to specific output file path
#878 opened Feb 24, 2025 by bkorycki
Search for config directory up the tree
#877 opened Feb 21, 2025 by wpietri
Only 1 SUT completion per prompt chore Thing that needs doing
#876 opened Feb 20, 2025 by bkorycki
Simplify annotation object hierarchy in modelgauge chore Thing that needs doing
#874 opened Feb 20, 2025 by bkorycki
Run James Ezick's #1: Variation testing
#870 opened Feb 18, 2025 by wpietri
automatic evalutator spin-up and spin-down CLI
#866 opened Feb 14, 2025 by rogthefrog
Fix auth requirements for prompt sets
#860 opened Feb 12, 2025 by rogthefrog
@rogthefrog
1
Delete old tests chore Thing that needs doing
#855 opened Feb 7, 2025 by bkorycki
SUT Options are defined at test-level chore Thing that needs doing
#854 opened Feb 7, 2025 by bkorycki
Test Item only has 1 prompt chore Thing that needs doing
#853 opened Feb 7, 2025 by bkorycki
play with MLFlow
#852 opened Feb 7, 2025 by wpietri
play with DVC
#851 opened Feb 7, 2025 by wpietri
GOAL: some analytics tools milestone A goal that other cards add up to
#847 opened Feb 6, 2025 by wpietri
Murder static site generator
#844 opened Feb 5, 2025 by rogthefrog
1
run consistency checker on FR results
#843 opened Feb 5, 2025 by wpietri
1
Dynamic SUTs
#842 opened Feb 4, 2025 by wpietri
Make the Mistral SUT run reliably
#841 opened Feb 4, 2025 by wpietri
ProTip! Add no:assignee to see everything that’s not assigned.