Add devenv.nix enheritance mode (instead of composition)

[Andrew15-5]

In my project, I have the root config and a few sub-projects. The sub-projects are basically a copy of the root, but with some changes. So I want to switch from composing sub-config into main one to inheriting the things from the main config into sub-projects. The easiest example is the use of same packages and same scripts/aliases.

Currently, I need to duplicate everything in all config, but in the root one additionally use lib.mkForce. This is super bloated, even worse than having separate configs (because you don't have to use lib.mkForce).

What is the best way to "reverse" the current behavior? Maybe there is another tool for this?

[liammcdermott]

I feel what you're looking for should be possible. I guess we'd need to see your Devenv configuration to say for sure, but it sounds like your root project's devenv.yaml points to the sub-projects, when really you would like it to be the other way around.

Looking at how drupal-devenv works might help. I think it has a very similar objective to your root config + sub-projects idea. That is, it provides a root config that any project can build on (but override if necessary).

Besides the way that project sets up its imports, it uses lib.mkDefault everywhere, so 'sub-projects' can override its options if necessary.

Does this help? If not, it would be good to see your actual config, or at least a cut-down sample of it, to get more context.

[liammcdermott]

I think my last post may not be helpful after all, sorry.

Assuming the desired setup includes importing nix config from a parent directory, a limitation with Nix Flakes stops this currently. See #1239

Does the configuration in that issue demonstrate the problem you're having?

Edit: it may still be possible to use parent config, as long as you can add an input for the parent directory like this example

Also, damn this is confusing.

[Andrew15-5]

I don't know if I can use relative path in devenv.yaml, I might've already tried it, and it didn't work.

The example is huge, but I don't see how I can import the base config, which is the main problem.

Because of this, I had to move to a flake, and with a bit of help from people I realized that I can define all devshells in a single flake and then just choose the right one in different directories. Though I don't know if I can inherit with flake, maybe by using proper module system, but I was able to achieve the same result with a template function.

The function produces the base config, subdirs can override stuff in it, but if they don't need something, then it will be added separately to the base config when calling the function. So the subdirs will also call the same function and add something else. I actually later realized that I only need the same config for all subdirs, so in total I only need 2 devshells.

flake

{
  inputs = {
    nixpkgs.url = "github:cachix/devenv-nixpkgs/rolling";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
    devenv.inputs.nixpkgs.follows = "nixpkgs";
  };

  nixConfig = {
    extra-trusted-public-keys = "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = { nixpkgs, devenv, systems, ... }@inputs:
    let
      configTemplate = system: overrideFunction: devenv.lib.mkShell {
        inherit inputs;
        pkgs = nixpkgs.legacyPackages.${system};
        modules = [
          (
            { pkgs, lib, ... }@localInputs:
            let
              overridesFor = x: (overrideFunction localInputs).${x} or { };
            in
            rec {
              env = {
                main = "src/main.typ";
                out-dir = "dist";
                pdf = "${env.out-dir}/out-name.pdf";
              } // (overridesFor "env");

              packages = with pkgs; with env; [
                git
                mupdf
              ] ++ (with builtins; (lib.mapAttrsToList
                (name: value: writeShellScriptBin name value.exec)
                (lib.filterAttrs (name: _: !elem name [ "setup:init" ]) tasks)
              ));

              enterShell = ''
                echo Available aliases:
                ${builtins.concatStringsSep "\n" (
                  lib.mapAttrsToList
                  (name: value: "echo ${name} = ${lib.escapeShellArg value.exec}")
                  scripts
                )}
                echo
              '';

              scripts = with env; rec {
                edit.exec = ''"$EDITOR" ${root}/${main}'';
                compile.exec = "app:compile";
                clean.exec = "clean:everything";
                e = edit;
                co = compile; # Can't unalias c, r, w, cl, ca, cln
                cle = clean;
                # ...
              } // (overridesFor "scripts");

              tasks = lib.recursiveUpdate
                (with env; {
                  "app:compile".exec = ''
                    mkdir -p ${root}/${out-dir}
                    compile --root ${root} ${root}/${main} ${root}/${pdf}
                  '';
                  # ...
                  "clean:everything".exec = "rm -rf ${root}/${out-dir}";
                  "setup:init".before = [ "devenv:enterShell" ];
                })
                (overridesFor "tasks");
            }
          )
        ];
      };

      root = ''"$DEVENV_ROOT"'';
      forEachSystem = nixpkgs.lib.genAttrs (import systems);
    in
    {
      devShells = forEachSystem (system:
        rec {
          default = base;

          base = configTemplate system ({ pkgs, config, ... }:
            {
              scripts = rec {
                zip-all.exec = "zip:all";
                zi = zip-all;
                za = zip-all;
              };
              tasks = with config.env; {
                "zip:all".exec = ''
                  cd ${root} || exit 1
                  mkdir -p ${out-dir}/${out-name}
                  cp ${pdf} ${out-dir}/${out-name}
                  ${pkgs.fd}/bin/fd '.*' ${root}/src -It f -e pdf -X cp '{}' ${out-dir}/${out-name}
                  rm -rf ${out-dir}/${out-name}.zip
                  ${pkgs.zip}/bin/zip -r ${out-dir}/${out-name}.zip ${out-dir}/${out-name}
                  rm -rf ${out-dir}/${out-name}
                '';
              };
            });

          lab = configTemplate system ({ config, ... }: {
            env.pdf = ''...'';
          });
        });
    };
}

The boilerplate is pretty big, since I practically do everything myself, but it does work. Then I just use use flake . --no-pure-eval and use flake ../..#lab --no-pure-eval.

But having a more native approach would be nice, as it will not only significantly cut down on the LoC, but also return the ability to run tasks, since with flake version there is no devenv tasks command. This is why I had to hack together the additional packages to be able to run them. This also forced me to duplicate things as I can't add dependency tasks, only depending on "devenv:enterShell" works. Not to mention that I can't override my own global aliases, but that is a limitation of direnv.

[Andrew15-5]

Does the configuration in that issue demonstrate the problem you're having?

Yeah, I think so.

Edit: it may still be possible to use parent config, as long as you can add an input for the parent directory like this example

Also, damn this is confusing.

Uhhh, I cannot comprehend what that file does. Truly a peak of hackery.

[liammcdermott]

The example is huge, but I don't see how I can import the base config, which is the main problem.

Ah, I didn't explain myself very well. In the example of drupal-devenv: drupal-devenv is the root config and the user's project is the sub-project.

The Getting Started section shows how to import that root config into your sub-project.

The difference in your case: the root project is ../ (and not gitlab:woolwichweb/drupal-devenv), which gets us a devenv.yaml that looks like:

inputs:
  nixpkgs:
    url: github:cachix/devenv-nixpkgs/rolling
  rootProject:
    url: ../
    flake: false

imports:
  - rootProject

When I tried this, however, it resulted in some buggy behaviour from enterShell. Then I tried using the full path instead of ../, so my rootProject input looks like this:

  rootProject:
    url: /home/liam/code/devenvtest
    flake: false

Which worked!

For context, the full path to my sub-project devenv.yaml is /home/liam/code/devenvtest/sub-project/devenv.yaml

So, I think the reason it didn't work is a bug in Devenv when using a relative path for an input, and a workaround is to use an absolute path instead. That's unsatisfying and not portable, but it may be less complex than using a Flake.

I like that usage of an override function in your Flake. It's a clever solution to the problem 👍

[Andrew15-5]

Thanks. Yeah, I don't want to use the absolute path, since it's all version-tracked. From #1239 (comment), I guess it confirms that this is due to flake shenanigans (rules). So unless something clever is done, I don't think this will be possible.

[liammcdermott]

I don't want to use the absolute path, since it's all version-tracked.

That's understandable. Another workaround might be to reference the repo by URL, using access-tokens to allow access (assuming it's private), that's also not the best and probably involves too much manual setup.

From #1239 (comment), I guess it confirms that this is due to flake shenanigans (rules).

Yeah, my attempt at importing the parent directory did it slightly differently than that comment, and didn't hit that rule. However, it still didn't work.

In case you (or anyone else reading this) is interested: instead of concat-ing the lines from the two devenv's enterShell then running the result (which is the expected behaviour), Nix would concat the sub-project's enterShell with itself, and run it.

So, given these enterShells:

  enterShell = ''
    echo "hello from parent";
  '';

  enterShell = ''
    echo "hello from sub-project";
  '';

devenv shell results in:

hello from sub-project
hello from sub-project

Hopefully this issue will be fixed by #1548.