Difference between dev-dependencies and optional-dependencies?

[pawamoy]

Hello everyone, and hi @frostming, thank you for building this great tool 🙂

I'm super excited by these optional dependencies thing, but I'm not sure I understand how I should use them.
What I understandd from reading the docs is:

• dev-dependencies are for development only, they will not appear in the sdist/wheel as extras
• optional-dependencies will appear in the sdist/wheel, and users will be able to select them with extras, like pip install thing[group1,group2]

Did I get it right?

[frostming]

Yes, you are correct. dev-dependencies work like the same field in Pipfile or Poetry, it is an addition to the PEP 621 and won't appear in the converted metadata shipped with distributions. While optional-dependencies is a standard PEP 621 field and acts like extras_require in setup.py.

If you have some general dev-only libraries, put them in dev-dependencies, otherwise if would like some tools and libraries to be installed only under certain scenarios, optional-dependencies is a good place to go.

[pawamoy]

Thank you!

I was hoping we could have groups for dev-dependencies... maybe another time 🙂

[frostming]

Just put them in optional-dependencies, it is harmless to have them in package metadata. :)

[pawamoy]

Sure, but then, when developing, I have to specify these groups manually, as pdm install will not install these groups by default. It's really a nitpick though, not really important :)

Unless there's a dynamic dev group that would install every optional dependencies?

[pohlt]

My understanding of extra packages always was that the package I'm installing could do additional (extra) things if the extra packages are installed (e.g. support a numpy interface if numpy is installed).

On the other hand, development packages do not enable additional functionality for the user of a package, but are required for development (testing, deploying, etc.) on the package.

For the potential user of a package (who cares about upstream dependencies) this is a fundamental difference and might influence her/his decision to install and use a certain package. Libraries.io currently shows all these development packages as extras which they are not according to my definition.

IMHO, it would make a lot of sense to not have development packages in the "project.optional-dependencies". Is it possible to have sections (which make a lot of sense) in the dev-dependencies like we have it now in the "project.optional-dependencies"?

[frostming]

An interesting fact is PDM 0.x had a meta field extras which is a list of group names of optional dependencies that should be included in the project's Extras Require metadata. But in the refactor to 1.x release I dropped this field because it isn't specified in PEP-621.

According to the specification, fields whose actual values are to be decided by the build backend should appear in the dynamic field. So if we are to introduce extras back, we should put optional-dependencies in dynamic when extras is not empty.

[pawamoy]

Very well formulated @pohlt!

This is what I actually meant with "I was hoping we could have groups for dev-dependencies".

It's exactly what I need: dev-dependencies sections.
It would tremendously help both locally and in CI, as I wouldn't have to install every dev-dep in every job in CI anymore, but just the right sections, and without these dev-deps appearing as extras in the final wheel/sdist (I don't want them to).

I would still love being able to install all dev-deps at once though, with for example a dynamic dev section that magically contains all other dev sections, or by altering the behavior of a default pdm install to install everything, optional or not, so contributors and myself could do pdm install -s dev or pdm install respectively upon cloning the repo.

I actually opened the "Support dependencies groups" feature request in Poetry back at the time, which is currently the 9th most upvoted issue on the repo, and is included in the roadmap for version 1.2.

To copy and adapt the examples back here:

# in CI
pdm install -s ci

# or in specific CI jobs
pdm install -s /bandit  # bandit reads the code, we don't need anything else
pdm install --with /safety    # safety needs production dependencies to be installed

# locally, all three lines are equivalent
pdm install  # installs everything, as usual
pdm install --with local,ci,docs
pdm install -s default,local,ci,docs

# on readthedocs
poetry install --with docs  # sphinx-autodoc needs production dependencies to be installed

I'm totally open to other specification suggestions of course, to better match PEPs and PDM's own spec and vision 🙂

[frostming]

This proposed approach is different from PDM's current behavior. Default dependencies will be always installed unless explicitly excluded by --no-default flag, so:

pdm install   # install default
pdm install -d  # install default+dev, for somebody who are to develop this project
pdm install -s doc   # for building doc, as production deps need to be installed
pdm install -d -s tests -s docs  # default+dev+tests+docs, -s can be given multiple times instead of comma-separated value.
pdm install -s bandit --no-default   # bandit only

The principle is the shorter the command is, the more common the scenario should be. Dependencies given by -s are required under the very special cases and are not needed most time.

[pohlt]

Personally, I would always use the -s notation to split up the different development stages (tests, docs, lint, deploy, ...) and never use -d because I don't see a canonical definition for "the default development packages".

This might have been different when "development" mainly consisted of pytest and a few of its plugins, but nowadays where "development" is just a different word for CI/CD, there are different stages and tools for each stage during development and using sections to structure this (and save installation time in the pipeline) makes a lot of sense to me.

[frostming]

Hey folks, I have made a PR #351 to support dev-dependencies groups. I have tried to make the change backward-compatible, but still needs more testing. I appreciate if you would like to help with this. @pawamoy @pohlt

[pawamoy]

Hmmm it fact my VSCode setup wasn't right. It used the user installed version of pytest, which explains why it found jupyter-server and stuff.

[pawamoy]

I was finally able to configure VSCode properly.

In the end, since I had pytest installed in user-mode using my system Python, and PDM used that same interpreter for version 3.9, when I was running pdm run pytest, it was in fact using the user-installed pytest and not the one in __pypackages__/3.9.

Is PDM incompatible with user-installed libs, or is there something I missed?

[frostming]

when I was running pdm run pytest, it was in fact using the user-installed pytest and not the one in pypackages/3.9.

Did you mean the binary used is form user site? But it isn't on my side:

$ pdm run pytest -VV
This is pytest version 6.2.2, imported from /Users/fming/wkspace/github/test-pdm/__pypackages__/3.9/lib/pytest/__init__.py
setuptools registered plugins:
  pytest-cov-2.11.1 at /Users/fming/.local/lib/python3.9/site-packages/pytest_cov/plugin.py

As you see the pytest-cov plugin is loaded from the user site, this is the designed behavior, but the binary is from __pypackages__.

[pawamoy]

Ah, you're right:

% pdm run pytest -VV           
This is pytest version 6.2.2, imported from /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/pytest/__init__.py
setuptools registered plugins:
  pytest-xdist-2.2.1 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/xdist/plugin.py
  pytest-xdist-2.2.1 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/xdist/looponfail.py
  pytest-cov-2.11.1 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/pytest_cov/plugin.py
  pytest-randomly-3.6.0 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/pytest_randomly.py
  pytest-sugar-0.9.4 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/pytest_sugar.py
  pytest-forked-1.3.0 at /media/data/dev/pawamoy/copier-templates-extensions/__pypackages__/3.9/lib/pytest_forked/__init__.py
  anyio-2.2.0 at /home/pawamoy/.local/lib/python3.9/site-packages/anyio/pytest_plugin.py

[pawamoy]

Well, I guess something was wrong on my system, causing pytest to fail while trying to load pytest_tornasync.