Reuse input lock files

[roberth]

Is your feature request related to a problem? Please describe.

Lock files contain more nodes than needed.
This leads to an incentive to put inputs that are not for consumers of a flake elsewhere, degrading the developer experience.
Typically this involves a second flake lock in a subdirectory for development dependencies, which needs to be managed separately, may go out of sync with the root flake, etc.

Describe the solution you'd like

Only store the nodes that are direct inputs or those where it makes sense to store a copy, when a follows makes it possible to access the otherwise transitive input as a direct input.
This way, we have no loss of performance.
"Missing" transitive nodes are sourced from the inputs' lock files that contain them.

Status

Not implemented yet.
Migration plan below.

Previous issue description, initial substantiation

(was: Solve dev(/test) dependencies and bloated, diverging lock files, by reconsidering the flattened lock file)

Is your feature request related to a problem? Please describe.

We've previously identified issues with the way the lock file currently works.

• Irrelevant development dependencies are added to the lock files of all dependent flakes
  • Development only flake inputs or different category of inputs #6124
• Mututally dependent flakes cause diverging lock files as each lock file update incorporates a new version with a chain of irrelevant dependencies.
  • issue?
• Some users just complain about the size of their lock file. Partly pedantic, but they're actually kind of right:
• Ideally, lock file changes are reviewable, so that nothing weird creeps in through contributions that have such updates.

Describe the solution you'd like

We may consider two scenarios; one with follows and one without. Let's start without follows.

Observing that we do not exploit the apparent benefit of knowing all inputs upfront, we may consider loading the required parts of dependency lockfiles on demand.

Example scenario

• Dependencies . -> foo -> nixpkgs
• No follows.

Evaluation order, current situation:

• need packages.default
• need inputs.foo.packages.foo
• read foo details from ./flake.lock
• fetch foo
• need foo.inputs.nixpkgs
• read nixpkgs_1 details from local lock file
• fetch nixpkgs_1

Same evaluation, partial lock file

• need packages.default
• need inputs.foo.packages.foo
• read foo details from local lock file
• fetch foo
• need foo.inputs.nixpkgs
• read nixpkgs details from foo/flake.lock
• fetch nixpkgs

As you can see, the fetching characteristics remain the same.

Let's reintroduce follows, and first consider the example:

inputs.foo.nixpkgs.follows = "nixpkgs";

In other words, this injects the local nixpkgs dependency into foo. The lazy fetching behavior remains the same. This injection should be handled by call-flake.nix and does not require changes to the lock file.

Now consider the opposite follows:

inputs.nixpkgs.follows = "foo.nixpkgs";

In this case, if we were not to include a copy of foo.nixpkgs details in the local lock file, we'd have a slightly worse lazy fetching behavior, as nixpkgs would only be fetchable after foo's lock has been fetched. For this case, we do want "flat lock file" behavior, but limited to this dependency. The rule for which transitive dependencies to include in the lock file might be as simple as direct inputs + inputs referenced in follows.

This suggests the following implementation strategy:

• When generating a lock, apply the above rule to determine which flakes to traverse and include in the lock file
• When calling a flake, call it such that its inputs are those from its own lock // those set in the current lock
  • Lazy trees #6530 already adds an overrides parameter in call-flake.nix that appears to enable this. The missing bit is that some/many nodes would be created by reinvoking call-flake.nix rather than sharing the node identifiers. Node identifiers are local to each call-flake.nix call. i.e. overrides is a prerequisite for calling dependency locks.

Describe alternatives you've considered

Mark development dependencies explicitly.

• This is more configuration which is generally not good.
• This adds more corner cases and error messages.
• This does not solve the problem of having mutually recursive development dependencies. Note that we can barely manage to make builds not mutually recursive, but flakes encompass more than builds, so expecting flakes not to be mutually recursive is not realistic.

Additional context

• Alternative or complement to Development only flake inputs or different category of inputs #6124

Priorities

Add 👍 to issues you find important.

[roberth]

I quite like the syntax @mkenigs used in #4004 to talk about overrides

A (override B.C.D.E) -> B (override C.D)-> C -> D -> E

It is also a useful vehicle for talking about the locking algorithm, which seems that it would recurse into dependencies until the overrides are (), and would only record the inputs that are not already locked or overridden somewhere.

Similarly, it would probably reflect the overrides parameter in call-flake.nix. This parameter does not exist in master yet, and lazy-trees defines it as overrides on sources. It would have to become a structure representing overrides containing invoked flakes rather than source inputs, and it perhaps need a somewhat richer structure to represent deep overrides.