Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide scorers that use binary judgments the mapping of values to relevant or irrelevant #1260

Open
david-fisher opened this issue Mar 3, 2025 · 0 comments

Comments

@david-fisher
Copy link
Contributor

Is your feature request related to a problem? Please describe.
As a relevance engineer, I would like to evaluate my binary judgment metrics, such as precision or reciprocal rank, on cases that have multi-valued judgments.

Describe the solution you'd like
I would like to specify a separate parameter in the settings menu that passes the mapping choices (max irrelevant value for scales, or a map) in the same fashion as qoption is passed in for individual queries to the scorer.

This would allow implementing a single scorer for a metric, eg:
presumes #1259

RR@k

const k = get_k() 
let rank = 0;
const max_irrel = get_max_irrel()
eachDoc(function(doc, i) {
    if (rank === 0 && hasDocRating(i) && (docRating(i)) > max_irrel) {
        rank = i+1; // remember the rank of the first relevant document
     }
}, k);
const score = rank > 0 ? 1.0 / rank : 0.0;
setScore(score);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant