How to refactor Copier templates without submodules?

[verdaatt]

I'm building a template for a Terraform project folder structure that includes some resources by default, and am running into the problem that it has a lot of duplicate parts of the tree.

For example, if you have a folder for each hosting environment (like dev, test, staging, prod) then a lot of the content in each of those folders is mostly the same:

├── copier.yaml
└── template
    ├── dev
    │   ├── eu-west-1
    │   │   └── general
    │   │       ├── ecs
    │   │       ├── route53
    │   │       └── vpc
    │   └── us-east-1
    │       └── general
    │           ├── ecs
    │           ├── route53
    │           └── vpc
    ├── prod
    │   ├── eu-west-1
    │   │   └── general
    │   │       ├── ecs
    │   │       ├── route53
    │   │       └── vpc
    │   └── us-east-1
    │       └── general
    │           ├── ecs
    │           ├── route53
    │           └── vpc
    ├── staging
    │   ├── eu-west-1
    │   │   └── general
    │   │       ├── ecs
    │   │       ├── route53
    │   │       └── vpc
    │   └── us-east-1
    │       └── general
    │           ├── ecs
    │           ├── route53
    │           └── vpc
    └── test
        ├── eu-west-1
        │   └── general
        │       ├── ecs
        │       ├── route53
        │       └── vpc
        └── us-east-1
            └── general
                ├── ecs
                ├── route53
                └── vpc

This means any change to one of those resources requires finding all the copies. As the template becomes more advanced with if-statements etcetera the number of duplicate folders can grow exponentially. In software engineering you'd refactor any duplicate code into a function, and some MVC frameworks refer to reusable chunks of code as "partials".

I'm wondering if it's possible to refactor in Copier by creating template blocks, which can then be reused within the main template? Read the manual about using git submodules, but that requires putting every little "partial" into a separate git repository. This would result in us creating half a dozen or more repo's, and how it all fits together being very hard to understand.

Is there any way to use something resembling a macro function, but then to generate parts of the template tree, using a folder in the same repo to store those partials? Perhaps something that works like this:

├── copier.yaml
├── partials
│   └── general
│       ├── ecs
│       ├── route53
│       └── vpc
└── template
    ├── dev
    │   ├── eu-west-1
    │   │   └── {% partial general -%}
    │   └── us-east-1
    │       └── {% partial general -%}
    ├── prod
    │   ├── eu-west-1
    │   │   └── {% partial general -%}
    │   └── us-east-1
    │       └── {% partial general -%}
    ├── staging
    │   ├── eu-west-1
    │   │   └── {% partial general -%}
    │   └── us-east-1
    │       └── {% partial general -%}
    └── test
        ├── eu-west-1
        │   └── {% partial general -%}
        └── us-east-1
            └── {% partial general -%}

Would much appreciate any solutions. Thanks!

[yajo]

Have you tried using symlinks?

[verdaatt]

I have not. That could work, but it would break the code for Windows users. Are there any code-based solutions?

[pawamoy]

I believe this would work even on Windows. Sure, when cloning the repo, symlinks would be replaced by actual files, but the contents would be the same as the originally linked file, and rendered the same way.

[verdaatt]

Update: Tried the symlinks and that works quite well. Got a lot of refactoring done. Now I need to figure out how to make the refactored piece smart. While the prod branch of the tree needs the same resources as the dev branch, some values need to be different. Since they are both symlinked to the same target there is no differentiation yet. Would very much welcome any suggestions!

[pawamoy]

You're looking for macros, well... Jinja has macros! https://jinja.palletsprojects.com/en/3.1.x/templates/#macros.

In a file called general in your repository root:

{% macro ecs(...) -%}
    ...
{%- endmacro %}

{% macro route53(...) -%}
    ...
{%- endmacro %}

{% macro vpc(...) -%}
    ...
{%- endmacro %}

(the dots are to be replaced with actual parameters/contents)

Then in your templated files (that you still have to duplicate everywhere, but that now require no maintenance):

{% import "general" as general %}
{{ general.ecs(...) }}

{% import "general" as general %}
{{ general.route53(...) }}

{% import "general" as general %}
{{ general.vpc(...) }}

(again, dots to be replaced with actual arguments)

[verdaatt]

So we've been playing around with this for a while and in general the symlinks work well. We didn't go with the suggested macro solution, because the trees aren't always 100% similar.

Still need to figure out how we can make a symlinked partial "smart". Right now it assumes every symlinked partial is identical, but that's not really the case. Imagine this simplified example:

├── copier.yaml
├── partials
│   └── region
│       ├── ecs
│       ├── route53
│       ├── vpc
│       └── region.yaml.jinja
└── template
    ├── dev
    │   └── us-east-1  ~> ../../partials/region
    ├── prod
    │   ├── eu-west-1  ~> ../../partials/region
    │   └── us-east-1  ~> ../../partials/region
    ├── staging
    │   ├── eu-west-1  ~> ../../partials/region
    │   └── us-east-1  ~> ../../partials/region
    └── test
        └── us-east-1  ~> ../../partials/region

We would want to write the region name into a variable in region.yaml during copier rendering, but the partial currently doesn't have any way of detecting where it's at. I believe this needs to be resolved with calling a Jinja macro inside region.yaml.jinja, but haven't figured this part out yet. Suspect #695 / copier-org/copier-templates-extensions#2 could be the solution to this.

[yajo]

https://jinja.palletsprojects.com/en/3.0.x/templates/#template-inheritance explains another way of reusing templates where only a part of the code changes.