This project provides a collection of scripts to easily download various open-source histopathology foundation models and their associated codebases. It is designed to be modular, fault-tolerant, and resumable.
- Modular: Each model has its own download script, making it easy to add or modify models.
- Fault-Tolerant: If a model download fails, the script will log the error and continue to the next model without halting the entire process.
- Resumable: The script tracks the download status of each model in a
download_status.csvfile. When you rerun the script, it will automatically skip any models that have already been successfully downloaded.
Before you begin, ensure you have the following installed on your system:
- Git: Required for cloning the model repositories. You can download it from git-scm.com.
- Python: The scripts are written in Python 3.8+.
- uv: This project uses
uvfor fast Python environment management and package installation. You can install it by following the official instructions on its GitHub page.
-
Clone the repository (if you haven't already):
git clone <repository_url> cd histopathology-model-downloader
-
Create the virtual environment and install dependencies: This project uses
uvto manage its environment and dependencies, which are listed in thepyproject.tomlfile. Run the following command to create a virtual environment in a.venvdirectory and install the required packages:uv venv && uv pip install -e .
-
Activate the virtual environment: Before running the download scripts, activate the environment:
source .venv/bin/activate
Several models are hosted on the Hugging Face Hub and require you to accept their terms of use on the model page before downloading. You also need to authenticate your machine. You can do this in one of two ways:
-
CLI Login (Recommended for local use): Run the following command in your terminal and follow the prompts to log in with your Hugging Face account token:
huggingface-cli login
-
Environment Variable (Recommended for non-interactive environments): Set the
HUGGING_FACE_TOKENenvironment variable to your Hugging Face access token. The scripts will automatically detect and use it for authentication.export HUGGING_FACE_TOKEN="your_token_here"
To download all the supported foundation models, simply execute the main script from the root of the project:
bash download_models.shThe script will create a histopathology_models directory in your project root, which will contain:
- The downloaded model repositories and weights.
- A
download_status.csvfile to track progress.
If the script is interrupted or a download fails, you can simply run it again. It will pick up where it left off, skipping any completed downloads.
The download_models.sh script includes the necessary #SBATCH directives to run as a job on a SLURM-managed cluster like the UCalgary ARC.
Before submitting: Ensure the line source .venv/bin/activate is uncommented in the script if you are using the uv virtual environment.
To submit the job, run:
sbatch download_models.shThe scheduler will allocate the requested resources and run the script. The output, including any print statements or errors, will be saved to a file named hist-downloader-JOB_ID.out. You can monitor the download_status.csv file to check the progress of the job.
The project is organized to be clean and modular:
download_models.sh: The main orchestrator script. It handles status tracking, fault tolerance, and calls the individual helper scripts.pyproject.toml: Defines the project's Python dependencies for theuvenvironment.download_helpers/: This directory contains all the logic for downloading the models.download_*.sh: Individual shell scripts, each responsible for downloading a specific model or its repository. These scripts are designed to exit with an error code if they fail, which the main script will catch.download_from_hf.py: A generic, reusable Python script that handles all downloads from the Hugging Face Hub. It is called by the shell scripts with model-specific arguments.
histopathology_models/download_status.csv: This file is created automatically to track the status (completedorfailed) of each model download.
The modular structure makes it easy to add new models. Here's how:
Create a new shell script in the download_helpers/ directory (e.g., download_new_model.sh).
Inside your new script, add the commands needed to clone the repository and download the weights.
Example: Adding a model from Hugging Face
If the model can be downloaded using timm or huggingface-hub, you can reuse the generic download_from_hf.py script. Important: Ensure your script will exit with a non-zero status code on failure so the main script can log it correctly (set -e at the top of the script helps with this).
#!/bin/bash
set -e
echo "--- 5. Downloading New Model ---"
# Clone the repository if there is one
# git clone ...
# Download the weights using the generic helper
python3 ../download_helpers/download_from_hf.py \
--model-name "New Model" \
--repo-id "user/new-model" \
--downloader "timm" \
--output-file "new_model_weights.pth"
echo ""Example: Adding a model from another source (e.g., Google Drive)
Ensure your script handles errors correctly. The gdown command will exit with an error if the download fails, which will be caught by the main script.
#!/bin/bash
set -e
echo "--- 5. Downloading New Model ---"
# Clone repo
git clone https://github.com/user/new-model.git
# Download weights
gdown --id "your_file_id" -O "new-model/weights.zip"
unzip "new-model/weights.zip" -d "new-model/"
rm "new-model/weights.zip"
echo ""Open the main download_models.sh script and add a call to your new helper script at the end of the "Run individual download scripts" section.
# ... existing download calls ...
if [ -f "$HELPERS_DIR/download_new_model.sh" ]; then
bash "$HELPERS_DIR/download_new_model.sh"
else
echo "⚠️ Could not find download_new_model.sh in $HELPERS_DIR"
fiFinally, add your new model to the list of "Shortlisted FMs" in this README.md and in the header of download_models.sh to keep the documentation current.