Ability to simultaneously evaluate LLM responses against multiple criteria

[jweisz]

I'd like to run a batch job in the UI where I upload a list of questions + LLM responses (generated elsewhere), then define a set of criteria to judge them against. Right now it seems that I can only evaluate against a single criteria at a time.

[martinscooper]

@jweisz we've discussed this feature in the past. As other users have been asking for it too I am starting its implementation on the backend side. However, surfacing different criteria results on the current UI is still unclear and needs design planning.

As for the first implementation of multi-criteria, I am going for the simplest approach which IMO is:

• Specify a list of criteria instead of just a single one
• For each instance, evaluate it based on the different criteria. There will be no optimization regarding running multiple criteria in the same prompt.
• Aggregate the results of the evaluation. This requires all criteria having specified a numeric score mapping from the criterion name.

[martinscooper]

@jweisz I added support for multi-criteria in direct assessment evaluation. Docs are here. Let me know what you think.

[wernergeyer]

Any thoughts @jweisz