Installing & using PDM on macOS - some issues and observations

[justasojourner]

Introduction

Firstly, and most importantly, a big thanks to you @frostming for building PDM. I have seen your input supporting Pipenv over some time, so I was interested when I saw that you had created PDM.

I have been checking in on PDM over a period of months, and, as I was starting a new, fairly major, project I thought it would be a good time to install it and have a look with a view to using it for the new project.

Some background: I develop company devops solutions using Python (>=3.8) with Pipenv for virtual environments. My IDE is PyCharm Professional. I develop on macOS and usually deploy on Linux (was CentOS, but now Rocky Linux).

I usually code in one major version of Python 3.8 / 3.9 and install Python into the macOS standard location in...

[...]/Frameworks/Python.framework/Versions/3.8 and then use the symbolic links in /usr/local/bin to the various Python files.

I tend to steer away from Python version changing tools - I try to keep it simple as we don't need to change around Python versions. I use Pipenv to create a virtual environment for each project, use the Python in /usr/local/bin as the Python interpreter and code in PyCharm.

PDM Install - Some Problems

So - to PDM. I had some problems, and found the install and remove scripts did not work as expected. First problem was the curl install script referred to python and I had hit the button before I realised that it would need python3 because macOS still has Python 2 as the system Python. Also the 'copy to clipboard' button for the curl script copied the command prompt - which obviously caused a problem (I didn't notice) running the command, and also, using zsh not bash, my command prompt is '%'.

I saw that PDM was installed in /Users/<user>/Library/Python/3.9/bin which, of course, is understandable for a macOS user specific install but I like to keep things simpler, so I didn't want to add that to the PATH, so I opted to install it in /usr/local/bin.

I then started having some issues, I can't remember whether running the CURL script install worked OK for the 'remove' operation but when I tried to specify the path using -p /usr/local/bin CURL came back with errors. In the end I download the script and ran it locally and it was OK (I did see later that the original install directory was cleared out but the empty directory was not removed - I deleted it manually).

However I then saw that PDM was installed in /usr/local/bin/bin - the file was a symbolic link to /usr/local/bin/venv/bin/pdm so I moved pdm to /usr/local/bin - the goal obviously being to not have to add to the path. It seems to run OK from there, it should not cause any problems?

Post Install - Some Problems

I then made a new test project https://pdm.fming.dev/usage/dependency/#initialize-a-project and added packages which worked fine. When I setup a PyCharm project it complained about the directory not being empty, I guess it would be OK to create the project in PyCharm first and then run pdm init to initialise PDM.

Anyway, installed packages, setup __pypackages__/<major.minor>/lib as sources root in PyCharm, copied some code from another project, ran it and all was good.

Then I tried out the pdm sub-commands, when I ran pdm config it showed the current config setting and said

Home configuration (/Users/<user_name>/.pdm/config.toml)

but there was no directory .pdm and no config.toml obviously. I couldn't find the file anywhere and did not know the syntax so I created it and added the config options one-by-one by running pdm config [option] [setting].

Understanding PDM Architecture

I was wanting to get a 'high level' understanding of the PDM architecture, I couldn't see anything regarding that in the documentation https://pdm.fming.dev

I assume because it implements as a PEP 582 package management platform that the package management part can be considered to be separate from the project once the __pypackages__ folder contains all the appropriate versions of required packages. What I mean is for deployment (onto another computer) the project folder could - conceiveably - simply be copied across without PDM and the Python app run. PDM would not need to be installed on the target computer.

Of course I would not work that way in a deployed production system - the project would be cloned from the project on our Git server (which would not include the __pypackages__ directory of course) and the packages installed from the PDM lock file. But I am curious as to the architecture.

One other reason for the question above is because, as a test, I created a subfolder under the project root so /project_root/test/ copied a Python script there and ran it and it worked. Packages are in /project_root/__pypackages__ as standard. I thought that apps had to be invoked from the project root folder which has __pypackages__ (and the PDM config files?).

In the app I added print(sys.path) and in both cases (from project root and subfolder) I saw [...]/pdm-test/__pypackages__/3.9/lib in the Python system path.

I setup this option - https://pdm.fming.dev/#enable-pep-582-globally - is that enabling this?

I have this PYTHONPATH=/usr/local/bin/venv/lib/python3.9/site-packages/pdm/pep582 in the environment.

Documentation

Do you have people running PDM on macOS? If you would like some Mac (and PyCharm on Mac) focused documentation would be happy to help. I have some experience with MkDocs via Rocky Linux documentation.

[frostming]

Thank you for such long feedback which is very valuable and helpful.

0. I am a Mac user myself, but only on my home computer while using Windows at work.

1. The install script's -p/--path is to modify the location of the library files(site-packages and bin directories), if you don't bother to add to PATH, you can also create a symlink of the executable file. Note if you don't give --path option, the resulting executable can be symlink itself, so you would need to find the link origin.

2. For the prompt issue, I put it there for better readability(noting this is run under the shell not anywhere else, like python REPL).

3. I tried to specify the path using -p /usr/local/bin CURL came back with errors

   How do you add that arguments? It should be done like this:

   curl -sSL <URL> | python - -p /usr/local/bin
   

4. I guess it would be OK to create the project in PyCharm first and then run PDM init to initialise PDM.

   Yes

5. If the home configuration doesn't exist, the command just shows the default config values, you don't need to create the file yourself, it will be created when you actually change some configurations.

6. For deployment, it is a bad practice to checkout __pypackages__ in VCS so they are gitignored by intent. Yes, the folder can be simply copied to the deployment machine and run without PDM, only add a PYTHONPATH pointing to __pypackages__/X.Y/lib. Otherwise, it is always recommended to run pdm sync from the pdm.lock to reproduce the environment.

7. After you enabled PEP 582 loading globally as instructed, you can run with vanilla Python directly: python /path/to/my_project/app.py as long as /path/to/my_project/__pypackages__/ exists, it isn't necessary to chdir to the project root.

8. Documentation contribution is welcome!

[justasojourner]

1. I got it working installed in /usr/local/bin, the pdm file there with the venv folder under it. It seems to be working OK.

[...]

3. Ahhh, in the docs it says:

usage: install-pdm.py [-h] [-v VERSION] [--prerelease] [--remove] [-p PATH] [-d DEP]

which gives one the impression that it should be:

curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py -p /usr/local/bin | python -

[...]

5. I gather the default config settings are encoded in PDM somewhere, I couldn't find them (yet). It's just when one sees this

Home configuration (/Users/<username>/.pdm/config.toml):

when running pdm config it seems that the file would be there - as it is so specifically referred to.

6. For deployment, it is a bad practice to checkout __pypackages__ in VCS

No arguments from me there for that, we always use VCS without packages. I'm just working on getting a handle on how PDM is setup. I guess another way of saying it is that (and this is good) PDM's role is a nice 'wrapper' around PEP 582 (which, hopefully, will become the standard for Python so we can stop with all this virtual environment mess) so a system could, conceivably, work following a standard PEP 582 layout - regardless of which tool created the PEP 582 layout. So that is a plus for PDM as you can say to people "if you use PDM to create your PEP 582 setup Python projects then you can be comfortable knowing that the application plus dependancies can work following the standard regardless of PDM". You know how developers love having to fix apps ;-)

7. Great
8. I will have a look. PDM is a good (and necessary) solution for the Python community. As I said I use Pipenv, I really don't want to have to get into learning Poetry (another Pipenv), pip + venv don't provide full functionality and PDM follows the PEP which we hope will be ratified.

[frostming]

work following a standard PEP 582 layout - regardless of which tool created the PEP 582 layout

It is a standard PEP 582 layout. Hopefully python interpreter can recognize __pypackages__ and load packages from it. But since the PEP is not accepted yet so PDM fills this gap by pdm --pep582 and PDM must be present. Alternatively, manipulating PYTHONPATH env should work most of the time, without the presence of PDM.