Allow templating items in _exclude

[jzazo]

Actual Situation

I would like to exclude files / folders conditionally in the copier.yaml file. Currently, if you add a file/folder to the exclude list, it is always excluded, even if there is a suffixed file that converts. Therefore, the exclusion list applies to the result (this has been discussed in past issues).

My real use case is that I have a few configuration files that apply to my template, and may or not apply to the destination depending on a certain variable. If they don't apply to the destination I would like to exclude them and not copy irrelevant files. If they apply, I would not exclude and normal overwrite rules apply.

Desired Situation

Being able to exclude files/folders conditionally would allow having config files in my template that are copied over conditionally on variables. This would be a new powerful feature.

Proposed solution

I would like to have something in the copier.yaml that would allow me to exclude files/folders:

_exclude:
  - "{% if variable is defined %}myconfig.yml{% endif %}"

I think this solution would not impact current behavior of exlusions applying to final names. It just modifies applying the exclusion conditionally.

[pawamoy]

Are you aware that Copier allows to do that thanks to file names?

A file named {% if some_condition %}filename.ext{% endif %}.jinja will only be created if the some_condition evaluates to true.

More generally, a file with an empty name will not be created (it would be impossible to create it anyway). Same for directories, although they don't need the .jinja suffix.

[jzazo]

Not exactly, because the file in question is configuring the template, it cannot have any other name. That file has to exist as is, and excluded on copy conditionally. At the moment I have to overwrite it and make it clean not to copy my template's config over, but it is not used in the destination. Does it make sense?

[pawamoy]

Oh, IIUC you have files that are used both for maintaining the template, and as files to copy to destination projects. Interesting. I'd recommend keeping two separate copies indeed, and using the subdirectory feature to separate them. Just my opinion but if you need to change config for your template you might not want this change to apply to projects too? Better explicit than implicit?

[pawamoy]

Other than that, no strong opinion :) If other maintainers think this feature is worth it, I won't mind.

[jzazo]

Yeah, let me give you an example. I have a acl/access.yml file that specifies who has right permissions in the template repo. When I create a copy of the template to apply to that same server, I ask the pertinent question of who should have access, and format a brand new file from acl/access.yml.jinja file, which will replace the template's original. If I am not creating a project for that server, I don't want to have this file at all in the new repo, so I would exclude the folder.

subdirectory feature is not ideal. I like to keep files in the root because I can test everything. I can run CI and builds exactly as they are going to run in the copies (or pretty close).

[pawamoy]

Actually could you post snippets of your file layout? Because it's quite hard to understand how exactly you're using Copier from text alone.

subdirectory feature is not ideal. I like to keep files in the root because I can test everything. I can run CI and builds exactly as they are going to run in the copies (or pretty close).

Could you elaborate on that? I'm not sure to understand either.

[jzazo]

Here an example. The access.yml file is used for compliance of the template repo. Copies of the template will inherit from the processed access.yml.jinja, encoding their own access. Same for the compliance folder. Copies of the template where I could conditionally exclude folders from, would not get the folder.

Regarding the CI that I mentioned. I have a .github/workflows/ci.yaml file running pytest and mkdocs builds. Those tests are run on the template repo, so I know that tests are being found and run, and documentation is being built. Because the same modules and CI is being copied to new repos, I can check that tests running for the template repo, will more likely pass in new copies. If I had a subdirectory with all my template files, I would have to have one ci workflow for the template, and another ci workflow for the subdirectory, completely different between themselves, I would not be able to test the subdirectory workflow at all (because it wouldn't be run, only the template's workflow would be run).

Hope this makes it all clearer.

[pawamoy]

OK I see, thanks a lot 🙂

If I had a subdirectory with all my template files, I would have to have one ci workflow for the template, and another ci workflow for the subdirectory, completely different between themselves, I would not be able to test the subdirectory workflow at all (because it wouldn't be run, only the template's workflow would be run).

Yeah that's one of the challenges I'm facing indeed. Tracking my progress here: pawamoy/copier-uv#24.

[michaelrode]

I am running into the same issue. I can't edit the file name because it is needed for the template itself to run some checks. I tried to create a duplicate filename and then add that to the exclude but it ends up skipping both directories.

[chancez]

I'd like an option that doesn't involve messing with the filename too because the conditionals can become pretty unwieldy and it also seems my editor (vim) doesn't really like it either, for some reason. I can open the file, but vim fails to save it because it seems to interpret some things incorrectly.

[yajo]

I'm not sure I'm understanding the use case properly, but if I do, it seems to me like you can solve it with a combination of subdirectory and symlinks.

One live example here:

• The project itself uses CI definitions you can find in .gitlab/ci/*.yml
• The template lives under template, and there's a symlink from template/{% if gitlab_ci != "none" %}.gitlab{% endif %}/ci to that parent folder.
• A question lets the user choose if they want to enable gitlab ci

As a result, the template CI itself uses the same CI primitives as the produced projects. But produced projects can configure that CI at will.

Would this strategy solve your problems?

---

Note: To make it even more fun, the template is applied to itself!

[chancez]

I mean it would but at some point the conditional expressions get really large, and trying to embed all that logic into a file name becomes untenable.

[yajo]

You can use a macro then, or a computed field

[jzazo]

Thank you @yajo for your input. Yes, using subdirectory would work, and I could have specific pipelines to test the template folder. As I mentioned above, it is not ideal, I think it becomes pretty cumbersome and repetitive.

This is btw the solution I currently have. I have many symlinked files to try to not repeat myself a lot. But it is not clear and it is not easy to mantain. I think the solution I am proposing would be cleaner and easier, without any breaking changes.

[sisp]

There's a subtle difference between using _exclude and Jinja conditions in directory/file names: _exclude refers to generated directories/files, so it may be difficult/unintuitive/verbose to come up with a correct pattern when using dynamic directory/file names.

Think of this example template for a Python project with configurable project layout and optional CLI:

layout:
  type: str
  help: Which Python project layout would you like?
  choices:
    flat: ""
    src: null

package_slug:
  type: str
  help: Your Python package name/slug

cli:
  type: bool
  help: Do you need a CLI?

.
└── {{ layout }}
    └── {{ package_slug }}
        ├── {% if cli %}__main__.py{% endif %}.jinja
        └── __init__.py

Here, the Jinja condition for the CLI-related file is part of its filename, so it works for either project layout. But if we were to use the _exclude setting, we'd need to include path-related template variables to construct the generated file path (without the .jinja suffix):

+_exclude:
+  - "{% if not cli %}/{{ layout }}/{{ package_slug }}/__main__.py{% endif %}"

 layout:
   type: str
   help: Which Python project layout would you like?
   choices:
     flat: ""
     src: null

 package_slug:
   type: str
   help: Your Python package name/slug

 cli:
   type: bool
   help: Do you need a CLI?

 .
 └── {{ layout }}
     └── {{ package_slug }}
-        ├── {% if cli %}__main__.py{% endif %}.jinja
+        ├── __main__.py.jinja
         └── __init__.py

TBH, I find the latter neither more intuitive nor more robust. If the conditions become more complex, a computed value may be used as @yajo already suggested. Note that macros may not work at the moment because of #1164, and my upstream PR pallets/jinja#1852 does not seem to get merged for some reason.

[jzazo]

thanks @sisp! I agree my proposal being more intuitive / robust is subjective. But it gives the user the option to choose how they want to encode the logic. I personally prefer to read simpler file names and a use the exlude pattern that is still pretty explicit. Using dynamic names & computed values in the exclude section would be pretty powerful. Maybe it does not become simpler, but it is cleaner to me, your exclude logic is confined clearly.

Furthermore, it allows not requiring the subdirectory feature, which we have to use otherwise. Using subdirectory makes everything messier in my opinion, as you may have to symlink or duplicate files.

[yajo]

I agree with @sisp but it's also true that I don't see that as a strong reason to oppose including templating in _exclude. However it's a strong reason to me to not want to develop such a feature. 😆

But if some contributor wants to open the PR, I'd be glad to review it. I'll label the issue as such.