Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Histopathology Foundation Model Downloader

This project provides a collection of scripts to easily download various open-source histopathology foundation models and their associated codebases. It is designed to be modular, fault-tolerant, and resumable.

Key Features

  • Modular: Each model has its own download script, making it easy to add or modify models.
  • Fault-Tolerant: If a model download fails, the script will log the error and continue to the next model without halting the entire process.
  • Resumable: The script tracks the download status of each model in a download_status.csv file. When you rerun the script, it will automatically skip any models that have already been successfully downloaded.

Prerequisites

Before you begin, ensure you have the following installed on your system:

  1. Git: Required for cloning the model repositories. You can download it from git-scm.com.
  2. Python: The scripts are written in Python 3.8+.
  3. uv: This project uses uv for fast Python environment management and package installation. You can install it by following the official instructions on its GitHub page.

Setup

  1. Clone the repository (if you haven't already):

    git clone <repository_url>
    cd histopathology-model-downloader
  2. Create the virtual environment and install dependencies: This project uses uv to manage its environment and dependencies, which are listed in the pyproject.toml file. Run the following command to create a virtual environment in a .venv directory and install the required packages:

    uv venv && uv pip install -e .
  3. Activate the virtual environment: Before running the download scripts, activate the environment:

    source .venv/bin/activate

Authentication with Hugging Face

Several models are hosted on the Hugging Face Hub and require you to accept their terms of use on the model page before downloading. You also need to authenticate your machine. You can do this in one of two ways:

  1. CLI Login (Recommended for local use): Run the following command in your terminal and follow the prompts to log in with your Hugging Face account token:

    huggingface-cli login
  2. Environment Variable (Recommended for non-interactive environments): Set the HUGGING_FACE_TOKEN environment variable to your Hugging Face access token. The scripts will automatically detect and use it for authentication.

    export HUGGING_FACE_TOKEN="your_token_here"

Usage

To download all the supported foundation models, simply execute the main script from the root of the project:

bash download_models.sh

The script will create a histopathology_models directory in your project root, which will contain:

  • The downloaded model repositories and weights.
  • A download_status.csv file to track progress.

If the script is interrupted or a download fails, you can simply run it again. It will pick up where it left off, skipping any completed downloads.

Submitting to a SLURM Scheduler (UCalgary ARC)

The download_models.sh script includes the necessary #SBATCH directives to run as a job on a SLURM-managed cluster like the UCalgary ARC.

Before submitting: Ensure the line source .venv/bin/activate is uncommented in the script if you are using the uv virtual environment.

To submit the job, run:

sbatch download_models.sh

The scheduler will allocate the requested resources and run the script. The output, including any print statements or errors, will be saved to a file named hist-downloader-JOB_ID.out. You can monitor the download_status.csv file to check the progress of the job.

Project Structure

The project is organized to be clean and modular:

  • download_models.sh: The main orchestrator script. It handles status tracking, fault tolerance, and calls the individual helper scripts.
  • pyproject.toml: Defines the project's Python dependencies for the uv environment.
  • download_helpers/: This directory contains all the logic for downloading the models.
    • download_*.sh: Individual shell scripts, each responsible for downloading a specific model or its repository. These scripts are designed to exit with an error code if they fail, which the main script will catch.
    • download_from_hf.py: A generic, reusable Python script that handles all downloads from the Hugging Face Hub. It is called by the shell scripts with model-specific arguments.
  • histopathology_models/download_status.csv: This file is created automatically to track the status (completed or failed) of each model download.

How to Add a New Model

The modular structure makes it easy to add new models. Here's how:

1. Create a New Helper Script

Create a new shell script in the download_helpers/ directory (e.g., download_new_model.sh).

2. Implement the Download Logic

Inside your new script, add the commands needed to clone the repository and download the weights.

Example: Adding a model from Hugging Face

If the model can be downloaded using timm or huggingface-hub, you can reuse the generic download_from_hf.py script. Important: Ensure your script will exit with a non-zero status code on failure so the main script can log it correctly (set -e at the top of the script helps with this).

#!/bin/bash
set -e

echo "--- 5. Downloading New Model ---"

# Clone the repository if there is one
# git clone ...

# Download the weights using the generic helper
python3 ../download_helpers/download_from_hf.py \
    --model-name "New Model" \
    --repo-id "user/new-model" \
    --downloader "timm" \
    --output-file "new_model_weights.pth"

echo ""

Example: Adding a model from another source (e.g., Google Drive)

Ensure your script handles errors correctly. The gdown command will exit with an error if the download fails, which will be caught by the main script.

#!/bin/bash
set -e

echo "--- 5. Downloading New Model ---"

# Clone repo
git clone https://github.com/user/new-model.git

# Download weights
gdown --id "your_file_id" -O "new-model/weights.zip"
unzip "new-model/weights.zip" -d "new-model/"
rm "new-model/weights.zip"

echo ""

3. Update the Main Orchestrator

Open the main download_models.sh script and add a call to your new helper script at the end of the "Run individual download scripts" section.

# ... existing download calls ...

if [ -f "$HELPERS_DIR/download_new_model.sh" ]; then
    bash "$HELPERS_DIR/download_new_model.sh"
else
    echo "⚠️  Could not find download_new_model.sh in $HELPERS_DIR"
fi

4. Update the Documentation

Finally, add your new model to the list of "Shortlisted FMs" in this README.md and in the header of download_models.sh to keep the documentation current.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages