This is the code for paper "Decision-Alignment Evaluation of Uncertainty Quantification". In this README, we give instructions on how to run the experiments and refer to the paper for a detailed explanation of the setup.
Install uv (if required) and simply run
uv sync
The code for the experiments on benchmark datasets can be found in experiments/benchmark_datasets.
There are four separate folders for the classification (binary and multiclass) and regression (univariate and multivariate) experiments.
We now explain how to run the experiments for binary classification; the others follow analogously.
All required files can be found in the folder experiments/benchmark_datasets/binary_classification.
To fetch and preprocess the datasets, run 01_preprocess_datasets.py.
A new folder appears at the root of the project containing the preprocessed data with all 100 different 5-fold splits.
To train the models for one specific dataset, repeat, and fold, run 02_train_models.py with the respective arguments for dataset, repeat, and fold, for example, after navigating in the binary_classification folder, run
python 02_train_models.py --dataset bank --repeat 0 --fold 1
The results will be stored in the new folder predictions.
To train the models for several datasets over several repeats and folds, you can use the orchestration script 02_train_models.sh.
For the evaluation for one dataset, run 03_evaluate_models.py with the respective argument for dataset.
This can be orchestrated with 03_evaluate_models.sh.
The results will be stored in the new folder results.
Running the sensitivity analysis for prior misspecification (available for binary classification and univariate regression) works just as regular evaluation, but using the Python script 04_sensitivity_analysis.py (and the corresponding .sh file for orchestration).
The code for this case study can be found in experiments/case_studies/electricity_market.
To be able to preprocess the datasets, we first need to access the raw data.
The Belgian balancing price data is from Open Data Elia, license: CC BY 4.0.
Via this link, the raw data can be downloaded.
The dataset needs to be stored as balancing_prices.csv in data/case_studies/electricity_market/raw.
The remaining two raw datasets are fetched automatically when running the script 01_preprocess_datasets.py. However, for this to work, one needs an API key for the ENTSO-E transparency platform.
One can generate a key in the account settings, after creating a (free) account.
Please store the key at the root of the project as entsoe_key.txt.
Now you can run 01_preprocess_datasets.py.
The processed data will be stored in data/case_studies/electricity_market/preprocessed.
For model training and prediction run 02_train_models.py.
Results will be stored in the predictions folder and model checkpoints for the GP in models.
For evaluation, run 03_evaluate_models.py.
The results will be stored in results.
The code for this case study can be found in experiments/case_studies/credit_approval.
To be able to preprocess the datasets, we first need to access the raw data.
The Kaggle dataset can be downloaded here and needs to be stored in data/case_studies/credit_approval/raw/kaggle.
The PAKDD dataset is fetched automatically when running the script 01_preprocess_datasets.py.
For model training and prediction run 02_train_models.py for a specific dataset (either kaggle or pakdd), repeat, and fold, or orchestrate the runs via 02_train_models.sh.
Results will be stored in the predictions folder.
For evaluation, run 03_evaluate_models.py for the kaggle or pakdd dataset separately.
The results will be stored in results.
The code for this case study can be found in experiments/case_studies/p2p_lending.
To be able to preprocess the datasets, we first need to access the raw data.
The dataset can be downloaded here and needs to be stored in data/case_studies/p2p_lending/raw.
We only need the dataset of accepted loans.
Now you can run 01_preprocess_datasets.py.
The processed data will be stored in data/case_studies/p2p_lending/preprocessed.
For model training and prediction run 02_train_models.py for a repeat and fold, or orchestrate the runs via 02_train_models.sh.
Results will be stored in the predictions folder.
For evaluation, run 03_evaluate_models.py.
The results will be stored in results.
All scripts to create figures can be found in the folder visualization.
case_study_figures.py, priors.py, pathological_priors.py, and prior_misspecification.py can simply be executed and the resulting figure will be stored in a new folder figures.
The result boxplots can be created with the script boxplots.py, specifying the experiment and alignment measure.
The sensitivity analysis results can be visualized with the script sensitivity_analysis, specifying the task.