[FEA] Model precision with precision_score

[beckernick]

I'd like to be able to use cuML to calculate the precision score of an estimator in Python. I didn't see a C++/CUDA implementation in the metrics/ header files, so this may require both C++/CUDA and Python/Cython bindings eventually. Didn't see an existing issue but will close if I find one.

Currently, this requires going to the CPU if we want to use an existing API (relying on NEP-18 for dispatch doesn't work here). With about three million rows, sklearn's precision_score can take about 1.5 seconds. A stopgap GPU implementation for binary precision using CuPy can be 500x+ faster.

def cupy_precision_score(y, y_pred):
    
    pos_pred_ix = cupy.where(y_pred == 1)
    tp_sum = (y_pred[pos_pred_ix] == y[pos_pred_ix]).sum()
    fp_sum = (y_pred[pos_pred_ix] != y[pos_pred_ix]).sum()
​
    return (tp_sum / (tp_sum + fp_sum)).item()
​
%time print(cupy_precision_score(y, yp.values))
%time print(precision_score(y_host, y_pred))
0.9998934583422118
CPU times: user 2.59 ms, sys: 0 ns, total: 2.59 ms
Wall time: 2.05 ms
0.9998934583422118
CPU times: user 1.47 s, sys: 59.5 ms, total: 1.53 s
Wall time: 1.5 s

Cross-listing to the tracking issue #608

[dantegd]

@beckernick starting with a CuPy implementation in cuML is totally fine. Particularly if submitted by the awesome person that is requesting it ;)

[beckernick]

@cjnolet This is the n-class version of the loose code above. It doesn't handle all of what sklearn does, of course. But it gets the core averages (binary, none, and macro). 100-300x speedup on low millions of rows.

Someone could perhaps run with this for an initial CuPy version as a good first issue.

import cupy as cp
import numpy as np

def cupy_precision_score(y, y_pred, average='binary'):
    """
    TODO: Handle the following
        - average=micro (slightly more annoying)
        - average=weighted (slightly more annoying)
    """
    nclasses = len(cp.unique(y))
    
    if average == 'binary' and nclasses > 2:
        raise ValueError
        
    if nclasses < 2:
        raise ValueError("Single class precision is not yet supported")
        
    res = cp.zeros(nclasses)
    
    for i in range(nclasses):
        pos_pred_ix = cp.where(y_pred == i)[0]
        
        # short circuit
        if len(pos_pred_ix) == 0:
            res[i] = 0
            break
            
        tp_sum = (y_pred[pos_pred_ix] == y[pos_pred_ix]).sum()
        fp_sum = (y_pred[pos_pred_ix] != y[pos_pred_ix]).sum()
        res[i] = (tp_sum / (tp_sum + fp_sum)).item()
        
    if not average:
        return res.get()
    elif average == 'binary':
        return res[nclasses-1].item()
    elif average == 'macro':
        return res.mean().item()
    return res.get()

[beckernick]

Related to #242 . May want to separate the scorers into individual issues and keep using #242 as a tracker.

[ptartan21]

Hi, I'd like to pick this issue up if there's no one else working on it (I'm currently an intern working at NVIDIA).

[beckernick]

cc @JohnZed

[Nzouh]

Hey! I'm a first-time contributor and would love to pick this up using the approach suggested by @beckernick. Is the issue still open for contribution?

[VaggelisGian]

Hi, I'd like to pick this up if it is still free. Linked PR #3184 has been dormant since late 2021 and the last activity here was months ago, so checking first that nobody is actively working on it.

Plan, following @dantegd's CuPy guidance above and @beckernick's sketch:

• Pure-CuPy precision_score in cuml.metrics, implemented next to accuracy_score using the current input-validation helpers
• sklearn-compatible signature: average in {None, 'binary', 'macro', 'micro', 'weighted'} plus labels, pos_label and zero_division, computed from the GPU confusion matrix rather than a per-class loop
• Tests against sklearn across numpy/cupy/cudf/pandas inputs, following the conventions in tests/test_metrics.py

If @ptartan21 or @Nzouh are still on this, happy to stand down.