Load terraformsh config from parent directory

[pwillis-els]

No description provided.

[pwillis-els]

Hi @Th0masL,

I reopened a pull request with some of the changes you requested. I liked your proposal, but I'm just trying to find a way to formulate the code so the functionality is based on reusable functions, rather than one function that searches for files in two different ways. (Your suggestion is completely appropriate for most software projects; but since this is more of a pet project, I'm being annoying and trying to make the code better for its own sake 😉 ) )

Will make another attempt at it, but if I don't get far we can just go with your original suggestion.

[pwillis-els]

One problem to consider in looking for parent terraformsh configs is overloading/precedence.

If there's a config in the current directory and a config in the parent directory, how do we load them?

• If the current directory's file is taken as authoritative, we shouldn't load the parent file, because it will overload things like CD_DIR.
• If the parent directory's file is taken as authoritative, do we load the current directory's file and then the parent's? That may result in a mix of options that conflict with each other, because the current directory and parent directory may have been intended to run two different Terraform modules with different Terraform versions and options.

It seems that the simplest/most reliable option is to only load one config. And it seems most logical to load the current dir's file as that is how the current behavior works, and generally you'd expect to load a config from the directory you're currently in with most tools.

Similarly, it seems like loading the file "closest" to your current dir makes the most sense, if it isn't in your current dir. So that leaves situations when there's a config in a parent dir and a config in a child dir: do we assume we always want to load a parent dir's config, even if it might not relate to the current dir? And if there's a child dir, do we want to instead seek to that directory and run Terraform there?

[Th0masL]

I think that it makes sense to only read terraformsh.conf files from the current and parent folders, as it would be to risky to load the ones in the child folder, since terraformsh cannot predict what to do with the child folders.

Regarding where to stop the loading of files in the parents folders, I think it makes sense to either:

• decide to only read the first match of terraformsh.conf file (first try to find in the current folder, then try any parent folders until finding a match)
• OR allow to read multiple terraformsh.conf files in a row, but the behavior of that might be tricky to predict as things the different terraformsh.conf files could conflict. If we do that, I think it's important to an Environment Variable that could be exported by the terraformsh.conf file that would tell terraformsh to stop reading any other parent files.

[Th0masL]

Just for example, in my current implementation, in my repo/branch, I'm exiting after the first match.

See https://github.com/Th0masL/terraformsh/blob/main/terraformsh#L496

Full list of changes I added are visible in the custom version v0.12.1

[pwillis-els]

I think either of those options could work, but I think this is one of those times where we have a lot of options but not really know which is the better choice because we haven't used them yet.

Can you tell me more about your use case for this function?

For myself, I use terraformsh with the idea that I only run it from a directory that I want to deploy one Terraform module from. I also load the ~/.terraformshrc file so that I have a general global option to turn on Terraform's compact mode, but that breaks with older versions of Terraform, so I have to add options in certain modules' configs to override those settings. Kind of annoying but I couldn't find a better way (particularly because I have one directory hierarchy full of modules with a half-dozen different Terraform versions)

[Th0masL]

In my setup, I'm using a parent/child folder structure to breakdown Terraform stacks by folder, but I want a single terraform.conf script file for enforcing the same behavior in all Terraform Stacks.

Here is an overview of my folder structure:

$ tree
.
├── aws
│   ├── 1-vpc
│   │   ├── 0-backend-dynamic.tf # Generated dynamically by terraformsh.conf
│   │   ├── 1-variables-dynamic.tf # Generated dynamically by terraformsh.conf
│   │   ├── 1-variables-dynamic.tfvars # Generated dynamically by terraformsh.conf
│   │   ├── 2-locals.tf
│   │   ├── 3-errorcheck.tf
│   │   ├── 4-providers.tf
│   │   ├── 5-modules.tf
│   │   └── 6-output.tf
│   ├── 2-ecs-cluster
│   │   ├── <... dynamically generated files ...>
│   │   ├── 2-locals.tf
│   │   ├── 3-errorcheck.tf
│   │   ├── 4-providers.tf
│   │   ├── 5-modules.tf
│   │   └── 6-output.tf
│   ├── 3-ecs-service-frontend
│   │   ├── <... dynamically generated files ...>
│   │   ├── 2-locals.tf
│   │   ├── 3-errorcheck.tf
│   │   ├── 4-providers.tf
│   │   ├── 5-modules.tf
│   │   └── 6-output.tf
│   ├── <... other stacks ...>
├── terraformsh.conf  # Single terraformsh.conf file, stored in parent folder
└── terraformsh.conf.json # JSON file that helps configuring terraformsh.conf

What I do is cd into the stack folder, like aws/1-vpc, an then run terraformsh.

Also, in my case, there are multiple people running terraformsh, including CI/CD scripts, so basically I cannot rely on any files in my $HOME folder, or anywhere else than in the actual git-repo's folder strucutre.

One of the main goal of terraformsh.conf in my case is to generate dynamic files in the Terraform Stack folder, and enforce a very opinionated view on what and how Terraform should run.

If you need a more clear example, I'm happy to share the content of my terraformsh.conf and terraformsh.conf.json files.

[pwillis-els]

@Th0masL sure, if you could share the content that would be great!

So what I've done in the past is in aws/1-vpc/terraformsh.conf I will have content like:

source ../../terraformsh.conf
source ../terraformsh.conf
CD_DIR=../../aws/modules/k8s-vpc

Since terraformsh.conf is just a shell script, I can use the shell command source or (better for POSIX but worse for readability) . to load in a parent terraformsh.conf file. The order is for the highest parent first, so subsequent directories inherit and overload the parent configuration.

To run this in CI/CD, I just check out the repo, then cd aws/1-vpc && terraformsh, and it inherits parent configs.

I also use symlinks to commonly-used configs to keep the configs DRY. (In the past I have experimented with functionality that auto-generates temporary config directories by overlaying multiple directories on top of one another, but some weird problems crop up because of that, and the current system ends up being much simpler while still working)

[pwillis-els]

Would still like to see your sample configs, but I'm leaning toward the terraformsh env var option that would tell it to look in parent directories for more configs to load. The value could be a number to indicate how far back to look. That way the user could either opt to do the source trick above, or auto-load parent configs.

Of course... if there is such an option, what happens when more than one config sets that option? Do we assume the second one is redundant? Or say that they both indicate how many parents to traverse; do we assume that the second instance of the variable increases the number of parents to traverse? Do we ignore it? Is it possible someone made a mistake by configuring that option in a parent file?

With the source trick, the answer is explicit: if a parent also uses source, the shell is actually just reading that file and executing it in the context of the current shell (I'm pretty sure, I should double check), so it would be relative to the current directory and fail.

So in order to use the source trick as shown above, the module has to explicitly load every single config from every parent directory that it wants; it cannot expect parent files to load more files. This makes the operation and configuration very explicit; each module must specify exactly what configs it wants to load. So by looking at any module's config, you can see exactly what will be loaded, which makes troubleshooting much easier, but the files sliiiightly more annoying to maintain x_x

[Th0masL]

I've been pretty busy recently so I couldn't spend too much time trying to implement a more complex way to parse parent folders, ans you suggested.

I had a look at your suggestion, and that's true that the following approach would work fine for my use case:

So what I've done in the past is in aws/1-vpc/terraformsh.conf I will have content like:

source ../../terraformsh.conf
source ../terraformsh.conf
CD_DIR=../../aws/modules/k8s-vpc

So I'm going to use your suggested approach for the time being, and I might work on an improved semi-automated way to load files from parent folders in the future, but for the moment that's fine :)

Thanks !

[Th0masL]

You can close this PR for now. Thanks !