Snakemake is a powerful and versatile workflow manager for all kinds of data processing tasks. However, over the past years a considerable backlog of open but potentially outdated, duplicated, or highly connected issues have accumulated.
The purpose of this project is to implement a triage workflow that watches the public Snakemake issues, groups issues by topic, and ranks the resulting clusters by relevance.
The goal is also to flag issues/meta-issues that are potentially outdated and superseded by the current code-base, and suggest those for closing or quick fixes.
The tool can be run locally or as a Github action. Briefly, it consists of the following steps:
- Fetch open issues from the Snakemake GitHub repository.
- Embed issue text using a sentence-transformer model.
- Cluster issues by similarity into meta-issues, identify overarching topics
- Post-processing: Rank meta-issues by relevance (code impact, criticality, age)
- Render results as a static web page, served on GitHub.io
src/snaketriage/cli.py— Main entrypoint forsnaketriagesrc/snaketriage/github_client.py— Fetches open issues from GitHubsrc/snaketriage/embedding.py— Creates text embeddings for downloaded issuessrc/snaketriage/clustering.py— Clusters issue embeddings and generates visualizationsdata/raw/— raw issue dumpsdata/models/— downloaded sentence-transformer modelsdata/embeddings/— serialized embeddings and metadatadata/clusters/— generated output, including cluster figures and tables
This project is managed by pixi for dependency management and tasks. The tool can be run and tested using pixi tasks, it can be installed via PyPi, or in the most usual case, run via GitHub Actions which will render and deploy a static website (planned, not yet implemented).
# fetch latest version of the repository from GitHub
git clone https://github.com/MPUSP/snaketriage.git
cd snaketriage
# create and activate virtual environment for pixi
pixi install
pixi shell
# run dedicated tasks
pixi run test-small-repo- TODO: Installation instructions for PyPi to be added once it is released
Test fetching and embedding issues using pytest:
pixi run pytest-github-client
pixi run pytest-embed
pixi run pytest-clusterCode formatting and linting is performed using Ruff:
pixi run ruff check .
pixi run ruff format .For authenticated reads against the GitHub API (higher rate limit), export your GitHub token as an environment variable:
export GITHUB_TOKEN=<your_token_here>- Dr. Michael Jahn
- Affiliation: Max-Planck-Unit for the Science of Pathogens (MPUSP), Berlin, Germany
- ORCID profile: https://orcid.org/0000-0002-3913-153X
- github page: https://github.com/m-jahn
This project is licensed under the MIT License. See the LICENSE file for details.