Adopt pyproject.toml

[stefnotch]

This project currently uses a requirements.txt file for specifying dependencies, and politely tells us students to create a virtual environment.
That is absolutely the correct way of doing it today. 🎉

However, the requirements.txt file comes with a number of shortcomings (main one being that there's no separation between "dependencies that I'm using" and "dependencies of dependencies")¹.

The Python world is currently fixing this, by very recently having adopted

• pyproject.toml: Direct dependencies and Python version go in here.
• PEP 751, a standardized lockfile: All dependencies, with their exact versions go in here. This was adopted less than a month ago!

Those files are currently being implemented in the various Python package managers. Notably

• UV implements them today Add support for PEP 751 lockfiles astral-sh/uv#12584
• Pip is going to implement them Implement installation from PEP 751 aka standardized lockfiles pypa/pip#13334

With that in mind, I would love it if eventually, we would be using a pyproject.toml in this repository!

Big Limitation

One of the dependencies is no longer in active development, and does not work with the new pyproject.toml standard. It seems like it won't ever be updated.
rusty1s/pytorch_cluster#185

Tangential Note

On that note, if you haven't tried out uv for Python, I can wholeheartedly recommend it. It's super fast, implements official Python standards, and automatically creates virtual environments. ✨

With that, the whole step instructions in the Readme can be shortened down to:

2. Install UV
3. Install all necessary dependencies with uv sync
4. Run the code with uv run make_dataset.py (or select the interpreter in Pycharm, as per usual)

Footnotes

1. If that dependency management stuff doesn't immediately make sense, I'm happy to properly explain it. ↩

[stefnotch]

By the way, here's the pyproject.toml that I am using for the first two parts of the exercise. The final part currently does not work. :(

[project]
dependencies = [
    "Levenshtein==0.27.1",
    "jsonargparse[signatures]>=4.27.7",
    "matplotlib==3.10.1",
    "numpy==2.2.3",
    "openpyxl==3.0.9",
    "overrides==7.7.0",
    "pandas==2.2.3",
    "pykdtree==1.4.1",
    "pysdf==0.1.9",
    "pytorch-lightning==2.4.0",
    "rtree>=1.4.0",
    "scipy==1.15.2",
    "tensorboard==2.19.0",
    "torch==2.4.1",
    "torchvision==0.19.1",
    "tqdm==4.67.1",
    "trimesh==4.6.4",
    "xmlschema==3.4.3",
]
name = "ppsurf-modeling"
requires-python = "==3.12.*"
version = "0.1.0"
[[tool.uv.index]]
explicit = true
name = "pytorch-cu124"
url = "https://download.pytorch.org/whl/cu124"

[tool.uv.sources]
torch = [
    { index = "pytorch-cu124", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]
torchvision = [
    { index = "pytorch-cu124", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]

Pytorch is sadly a bit of a special case, and needs some bonus configuring. Currently I did the UV-specific way of configuring it, and didn't look into how Poetry, Pip, etc would configure it.

And since I am on Windows, I had to install PySDF by editing it and then doing uv pip install pysdf-0.1.9.tar.gz

[stefnotch]

I noticed that the requirements.txt of this project only pins the direct dependencies. That means it's prone to breaking at any time. Breaking changes in a minor release are not too uncommon in the Python ecosystem. So an indirect dependency could break at any time.

[ErlerPhilipp]

@stefnotch thanks for the insights. I didn't know that there is so much development going on.

I recently learned that Pytorch dropped Conda support and Huggingface dropped Docker support. So I had to re-create my whole setup from Conda, tried some stuff (Pipenv), failed, and ended up with good-old Pip and venv. I'm still fed up and don't want to spend a lot of time on these things.

About pinning only the direct dependencies: I had some bad experience with repos that pin everything. I think one was due to Windows-Linux differences, but that's maybe only relevant for Conda. The other is with hosted systems like Huggingface and Google Colab that enforce certain (Numpy/Pandas/Pytorch) versions. Pinning those just creates errors on my side.

That aside, I have some questions about your suggested setup:

1. What advantages has UV over Pip? Can I just wait until Pip implements the lockfile?
2. Is UV a full replacement if it requires this special treatment?
3. Is there a way to pin a Python version like in Conda?

[stefnotch]

I can absolutely relate to the struggles with Python setups. I've had so many bad experiences with it. That's also why I'm so excited about Python finally improving!

Yes, pinning everything without accounting for cross platform differences is one of those things that many Python package managers handle incorrectly. The new Python lockfile standard should handle it correctly, I believe.

Being explicit about the supported Python versions and environments for the file overall is also unique to this PEP. This is to alleviate the issue of not knowing when a requirements file targets a specific platform.
-- says the PEP

1. UV is much faster than pip ever will be. And on Windows, it saves me the trouble of installing Python without accidentally bricking another Python installation. This is a very real pain point. Finally, it automatically creates virtual environments. Maybe pip will also do that last point in the future?
2. Not at the moment for this particular project. While it handles typical Pytorch projects really well, we sadly have one dependency that hasn't adopted the latest standard yet. Though, upon further inspection, maybe they already have fix(packaging): pyproject.toml rusty1s/pytorch_cluster#208 , and maybe they just haven't released that.
3. Yes, there is! The pyproject.toml file has this line requires-python = "==3.12.*". And all compliant tools will at least check this line, while the better tools will go out of their way to automatically use that Python version.

So, to summarize, my recommendation is:

• Whenever you have a small Python project, do try out uv. It's very convincing.
• For when the computer modeling course is to be held again next year, it may be worth looking at whether the Python standardization has gotten far enough. With any luck, you'll be able to use a pyproject.toml and things will actually work with pip (and uv, and poetry, and ...). Might require a bit of extra fiddling for Tensorflow and Pytorch though.

[ErlerPhilipp]

Thanks a lot! I'll keep this issue open as a reminder.