LaSCal: Label-Shift Calibration without target labels

Official code for "LaSCal: Label-Shift Calibration without target labels", NeurIPS 2024.

LaSCal is a label-free, consistent calibration error estimator for label shift, paired with a post-hoc calibration strategy for unsupervised calibration.

Installation

pip install -e .
pip install torch numpy scipy scikit-learn joblib pyyaml tqdm abstention
pip install matplotlib seaborn netcal tikzplotlib   # only for the plotting script

The package declares no pinned dependencies, so the runtime requirements are installed separately.

Data layout

Each experiment is a directory under a common root:

<experiments_path>/
└── <experiment_name>/            # e.g. cifar10_long_tail_resnet32_sgd_if_0.1
    ├── config.yaml               # must contain a SEED entry
    ├── source_agg.json           # {"y_logits": [[...]], "y_true": [...]}
    ├── target_agg.json
    ├── train_agg.json            # required by head_to_tail
    ├── source_features.pt        # required by head_to_tail, cpcs, transcal
    ├── target_features.pt
    └── train_features.pt

y_logits is N x num_classes pre-softmax scores; y_true is N integer labels.

if_0.1 is the paper's IF = 10: the least frequent class holds 10% of the samples of the most frequent one.

Running the code

Post-hoc calibration

Calibrates with every method in lascal.utils.constants.CAL_METHODS and reports the resulting CE per model, plus a macro average, as LaTeX table rows.

python scripts/post_hoc_calibration.py \
    --experiments_path experiments \
    --which_datasets cifar \
    --reported sum

--reported sum sums the per-class CE, matching the paper; the default mean divides by the number of classes. temp_scale_target and vector_scale_target use target labels and are oracle baselines.

Flag	Effect
--which_datasets	Subset of cifar / wilds
--exclude_methods	Method names to skip, space separated
--calibrate_top_label / --report_top_label	Top-label instead of classwise CE
--criterion	cross_entropy (default) or ece
--num_bootstrap_samples	Bootstrap samples for the std estimate (default 100)
--verbose	Log the temperature found by each method

Calibration error estimation under label shift

Reports source/target accuracy, CE on source and target, then CE under label shift with the true weights w* and with weights estimated by RLLS, ELSA, EM-BCTS and BBSE. This script covers CIFAR-10/100 only.

python scripts/ece_label_shift_estimation.py \
    --experiments_path experiments \
    --imbalance_factor 0.1 \
    --reported sum

Reliability diagrams

Writes one .tex (pgfplots) and one .png per method into figs/, for the uncalibrated model, source temperature scaling, HeadToTail and LaSCal.

python scripts/plot_reliablity_diagram.py \
    --experiment_path experiments/cifar10_long_tail_resnet32_sgd_if_0.1

Using LaSCal as a library

import json

import torch

from lascal import Calibrator, EceLabelShift, get_importance_weights

experiment_path = "experiments/cifar10_long_tail_resnet32_sgd_if_0.1"
source = json.load(open(f"{experiment_path}/source_agg.json"))
target = json.load(open(f"{experiment_path}/target_agg.json"))
source_logits, source_labels = torch.tensor(source["y_logits"]), torch.tensor(source["y_true"])
target_logits, target_labels = torch.tensor(target["y_logits"]), torch.tensor(target["y_true"])

source_agg = {"y_logits": source_logits, "y_true": source_labels}
target_agg = {"y_logits": target_logits, "y_true": target_labels}

# Label-free calibration of the target: no target labels are used
calibrator = Calibrator(experiment_path=None, verbose=True)
calibrated = calibrator.lascal(source_agg=source_agg, target_agg=target_agg)
calibrated_target_logits = calibrated["target"]["y_logits"]

# Label-free calibration error estimate on the target
weights = get_importance_weights(
    valid_preds=source_logits.softmax(-1).numpy(),
    valid_labels=torch.nn.functional.one_hot(source_labels).numpy().astype(float),
    shifted_test_preds=target_logits.softmax(-1).numpy(),
    method="rlls-hard",
)["weights"]

ce_estimate = EceLabelShift(p=2, n_bins=15, classwise=True)(
    logits_source=source_logits,
    labels_source=source_labels,
    logits=calibrated_target_logits,
    weights=torch.tensor(weights),
).nanmean()

Other calibration methods run through calibrator.calibrate(method_name=..., source_agg=..., target_agg=..., train_agg=...). Weight estimators: em, em-bcts, rlls-hard, rlls-soft, bbse-hard, bbse-soft, elsa.