You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a relevance engineer, I would like to evaluate my binary judgment metrics, such as precision or reciprocal rank, on cases that have multi-valued judgments.
Describe the solution you'd like
I would like to specify a separate parameter in the settings menu that passes the mapping choices (max irrelevant value for scales, or a map) in the same fashion as qoption is passed in for individual queries to the scorer.
This would allow implementing a single scorer for a metric, eg:
presumes #1259
RR@k
const k = get_k()
let rank = 0;
const max_irrel = get_max_irrel()
eachDoc(function(doc, i) {
if (rank === 0 && hasDocRating(i) && (docRating(i)) > max_irrel) {
rank = i+1; // remember the rank of the first relevant document
}
}, k);
const score = rank > 0 ? 1.0 / rank : 0.0;
setScore(score);
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
As a relevance engineer, I would like to evaluate my binary judgment metrics, such as precision or reciprocal rank, on cases that have multi-valued judgments.
Describe the solution you'd like
I would like to specify a separate parameter in the settings menu that passes the mapping choices (max irrelevant value for scales, or a map) in the same fashion as qoption is passed in for individual queries to the scorer.
This would allow implementing a single scorer for a metric, eg:
presumes #1259
RR@k
The text was updated successfully, but these errors were encountered: