feature: Only lock selected groups into the lockfile

[frostming]

Is your feature request related to a problem? Please describe.

This could partially solve #564. The biggest barrier to solving this is that we have to resolve ALL dependencies as a whole, to form a viable lock file, even if you only install one group.

Describe the solution you'd like

In #1692 we start to write the groups of requirements that are locked in the lock file. For compatibility, it contains all groups in the pyproject.toml by default. We can extend this to only selected groups, details are as below.

• Change pdm lock to recognize the dependency selection options(--group, --production, --no-default), so that:
  • Only selected dependency groups participate in the resolution and the group names will be stored in metadata.groups field in the lock file.
  • This introduces a breaking change: by default, only default+dev groups are locked, compared to all groups as the current behavior.
• When running pdm sync and pdm install, PDM will check the selected groups passed from the option against the ones encoded in the lock file, if the former contain groups that don't exist in the lock file, the installation will abort with an error.
• Similar to the above, when running pdm update or pdm remove to change a dependency that doesn't exist in the groups in the lock file, an error occurs.
• When running pdm add <package> with a target group, another option will be added to add the group to the lockfile, in this case <package> parameter can be omitted, meaning to add the selected group only.

Advantages

With this change, and that we already support specifying the target lock file via PDM_LOCKFILE or --lock/-L, we have improved the multi-lockfile workflow. Imagine you have a default lock file pdm.lock and many job-specific lockfiles in pdm-locks/ so that:

• pdm.lock contains the default(production) group, for deployment
• pdm-locks/pdm.lint.lock contains locked versions for lint dependencies
• pdm-locks/pdm.doc.lock contains locked versions for default+doc dependencies

As a result, you can have different versions of the same packages pinned in lint and doc dependencies, as a solution to #564. Using tox + tox-pdm, you can achieve this:

[tox]
envlist = py3{7,8,9,10}, lint
passenv = LD_PRELOAD
isolated_build = True

[testenv]
commands =
    test {posargs}

[testenv:lint]
set_env:
    PDM_LOCKFILE=pdm-locks/pdm.lint.lock
skip_install = true
commands = lint

[testenv:doc]
set_env:
    PDM_LOCKFILE=pdm-locks/pdm.doc.lock
skip_install = true
commands = doc

[sigma67]

Is it still possible to use the old behavior with a single lockfile after this change?

Edit: It seems pdm.lock -d will result in a lock of default+dev dependencies.

[frostming]

pdm lock -G:all

[sanmai-NL]

@frostming

I don't think your solution addresses my need to have a single lock file with the union of all possible groups/modes.

pdm lock --production --group ':all'

produces a different lock file than

pdm lock --group ':all'

And

pdm lock --dev --production --group ':all'

is rejected by PDM.

[frostming]

@sanmai-NL I don't know what you mean.

What I said is that pdm lock -G:all can do the trick. While you say pdm lock --production --group ':all' works differently from pdm lock --group :all. I don't understand why this can refute my argument. I didn't give other commands, just run pdm lock -G:all and all groups are locked.

P.S. --production of course makes a difference because it is excluding dev dependencies: --production == no dev

[sanmai-NL]

Sorry, I meant --dev instead of --production.

The problem is that I can't lock all of development, production and groups. I don't know if the first two are groups as well, in your solution vocabulary. If they are, then the docs and behavior of pdm lock confuses me since --dev and --production are listed explicitly and behave as mutually exclusive without mentioning that --groups potentially subsume them. If development and production are not subsumed by groups, then see my point as originally worded.

Thanks for your attention and careful development of this great product.

[frostming]

• pdm lock: lock default + dev dependencies
• pdm lock -G:all lock default + dev + optional dependencies(this is what you want)
• pdm lock -d -G:all lock default + dev dependencies(same as the first)

[sanmai-NL]

@frostming Can you explain a bit how to understand that pdm lock --group ':all' includes optional dependencies and pdm lock --dev --group ':all' does not? From the parameters this isn't intuitively clear to me, and from the docs I did not get this either.

-d, --dev Select dev dependencies
-G GROUP, --group GROUP
Select group of optional-dependencies separated by comma or dev-dependencies (with -d). Can be supplied multiple times, use ":all" to
include all groups under the same species.

1. Where is the definition of default dependencies documented? I searched the docs and the PEPs that are linked in there.
2. Where is the definition of species documented? (In English, species is a biological term I've never seen it used in IT outside PDM.)

[frostming]

1. Where is the definition of default dependencies documented? I searched the docs and the PEPs that are linked in there.

It's dependencies field in PEP 621 metadata, I add "default" distinguish it from other type of dependencies. They will always be installed unless --no-default is given.

2. Where is the definition of species documented? (In English, species is a biological term I've never seen it used in IT outside PDM.)

Despite of many groups, there are two "species", or "kinds" of dependencies that can have groups: optional dependencies and development dependencies. So what are selected by --group :all depend on whether --dev/-d is passed, to pick all optional dependency groups, or all dev dependency groups, respectively.

[sanmai-NL]

I'll note that to add to the docs one day.

[sigma67]

Should definitely be in the docs and CLI help. The :all syntax is not documented anywhere afaik.

[frostming]

Should definitely be in the docs and CLI help. The :all syntax is not documented anywhere afaik.

It's here: https://pdm.fming.dev/latest/reference/cli/#lock in the -G, --group GROUP description