Initalizing dependencies fails when state is encrypted #3495

norman-zon · 2024-10-18T14:20:01Z

Describe the bug

When the state of a dependency is encrypted and the dependency is not initialised, any terragrunt command in the dependant module fails.

Steps To Reproduce

Create two terraform modules A and B.
B is a dependency of A.
run terraform apply in both.
verify terragrunt plan works in A.
rm -rf .terragrunt-cache in B.
verify terragrunt plan still works in A
encrypt the state of B and rm -rf .terraform-cache again.
run terragrunt plan in A.

A fails with:

ERROR  Failed to execute "tofu output -json" in ../B/.terragrunt-cache/779213818
│ Error: Required plugins are not installed

un-encrypt state of B and rm -rf .terraform-cache.
verify terragrunt plan works again in A.

I created a minimal example repo unfortunately in this minimal setup I can not reproduce the issue.
Any advice what I should look out for in my real production setup, that could make the difference?

Expected behavior

State encryption does not alter the behaviour of dependencies

Versions

Terragrunt version: 0.68.3
OpenTofu/Terraform version: 1.8.3
Environment details: MacOS, can reproduce in CI under Ubuntu22.04

If you need further details I am happy to help out.

The text was updated successfully, but these errors were encountered:

norman-zon · 2024-10-18T15:10:10Z

When running with debug logging I see

16:59:56.571 STDERR [../B] tofu: Error loading state: Unsupported state file format: This state file is encrypted and can not be read without an encryption configuration

The encryption config for B is added via a generate block, like in my example. But I notice, that after the failed run .terragrunt-cache in B is empty. Is this somehow an issue of order of execution maybe?

I think the main difference between the example and my real environment is the example using local state.

norman-zon · 2024-10-18T15:21:33Z

Indeed!
When using a remote_state {} block in my example I can reproduce the issue.

So my guess is remote state is accessed before the generate blocks are executed? And therefore the encryption config is missing, if it is defined in a generate block.

denis256 · 2024-10-18T18:16:35Z

Hi,
AFAIK, Terragrunt accesses the remote state before running the generate block to check for configuration drifts.

I bypass this was thinking about:

try running with --terragrunt-disable-bucket-update to see if it will skip checking this part
extract config generation in an external script and invoke it through run_cmd

https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-disable-bucket-update

norman-zon · 2024-10-22T07:22:58Z

Hi @denis256, thanks for the suggestions.

--terragrunt-disable-bucket-update does not help unfortunately.
I tried using run_cmd with a script to generate the config, but this does not work either.

Also, this only happens in such a dependency setup as described above. Without dependencies I can use encryption config from a generate block.

So I, personally, would rathe classify this as a bug.

norman-zon · 2024-10-22T07:44:37Z

I tried using a before_hook instead of the generate, yielding the same result

norman-zon · 2024-10-29T12:26:42Z

I looked further into the issue.

Here is a complete debug output:

❯ terragrunt plan --terragrunt-log-level debug --terragrunt-debug
12:36:57.653 DEBUG  Terragrunt Version: 0.68.3
12:36:57.657 INFO   Terragrunt Cache server is listening on 127.0.0.1:53531
12:36:57.658 INFO   Start Terragrunt Cache server
12:36:57.658 DEBUG  Provider cache dir "../../../../.tf-plugin-cache"
12:36:57.664 DEBUG  Did not find any locals block: skipping evaluation.
12:36:57.664 DEBUG  Running command: tofu --version
12:36:57.664 DEBUG  Engine is not enabled, running command directly in .
12:36:57.684 DEBUG  tofu version: 1.8.3
12:36:57.684 DEBUG  Reading Terragrunt config file at ./terragrunt.hcl
12:36:57.685 DEBUG  Did not find any locals block: skipping evaluation.
12:36:57.685 DEBUG  Did not find any locals block: skipping evaluation.
12:36:57.687 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.688 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.688 DEBUG  [../B] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
12:36:57.688 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.689 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.689 DEBUG  [../B] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
12:36:57.701 DEBUG  [../B] git show-toplevel result:


..

12:36:57.702 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.702 DEBUG  [../B] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
12:36:57.702 DEBUG  Getting output of dependency ../B/terragrunt.hcl for config ./terragrunt.hcl
12:36:57.703 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.703 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.704 DEBUG  [../B] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
12:36:57.704 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.704 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.705 DEBUG  [../B] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
12:36:57.705 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.705 DEBUG  [../B] Did not find any locals block: skipping evaluation.
12:36:57.705 DEBUG  [../B] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
12:36:57.705 DEBUG  [../B] Detected remote state block with generate config. Resolving dependency by pulling remote state.
12:36:57.705 DEBUG  [../B] Setting dependency working directory to ../B/.terragrunt-cache/932801613
12:36:57.706 DEBUG  [../B] Generated file ../B/.terragrunt-cache/932801613/_backend.tf.
12:36:57.706 DEBUG  [../B] Generated remote state configuration in working dir ../B/.terragrunt-cache/932801613
12:36:57.706 DEBUG  [../B] Copying lock file from ../B/.terraform.lock.hcl to ../B/.terragrunt-cache/932801613
12:36:57.706 DEBUG  [../B] Running command: tofu init -get=false
12:36:57.706 DEBUG  [../B] Engine is not enabled, running command directly in ../B/.terragrunt-cache/932801613
12:36:58.256 DEBUG  [../B] Ignoring expected error from dependency init call
12:36:58.256 DEBUG  [../B] Init call stderr:
12:36:58.256 DEBUG  [../B] 12:36:57.729 STDOUT [../B] tofu: Initializing the backend...
12:36:58.117 STDOUT [../B] tofu:
12:36:58.117 STDOUT [../B] tofu: Successfully configured the backend "gcs"! OpenTofu will automatically
12:36:58.117 STDOUT [../B] tofu: use this backend unless the backend configuration changes.
12:36:58.251 STDERR [../B] tofu: Error loading state: Unsupported state file format: This state file is encrypted and can not be read without an encryption configuration

12:36:58.256 DEBUG  [../B] Running command: tofu output -json
12:36:58.256 DEBUG  [../B] Engine is not enabled, running command directly in ../B/.terragrunt-cache/932801613
╷
│ Error: Required plugins are not installed
│
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.opentofu.org/hashicorp/null: there is no package for registry.opentofu.org/hashicorp/null 3.2.3 cached in .terraform/providers
│
│ OpenTofu uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   tofu init
╵
12:36:58.296 INFO   Shutting down Terragrunt Cache server...
12:36:58.296 INFO   Terragrunt Cache server stopped
12:36:58.296 ERROR  Failed to execute "tofu output -json" in ../B/.terragrunt-cache/932801613
╷
│ Error: Required plugins are not installed
│
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.opentofu.org/hashicorp/null: there is no package for registry.opentofu.org/hashicorp/null 3.2.3 cached in .terraform/providers
│
│ OpenTofu uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   tofu init
╵

exit status 1
12:36:58.475 DEBUG  util.ProcessExecutionError Failed to execute "tofu output -json" in ../B/.terragrunt-cache/932801613
╷
│ Error: Required plugins are not installed
│
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.opentofu.org/hashicorp/null: there is no package for registry.opentofu.org/hashicorp/null 3.2.3 cached in .terraform/providers
│
│ OpenTofu uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   tofu init
╵

exit status 1
/home/circleci/project/shell/run_shell_cmd.go:216 (0x101320371)
/home/circleci/project/telemetry/metrics.go:42 (0x100c76d88)
/home/circleci/project/telemetry/telemetry.go:80 (0x10131f014)
/home/circleci/project/telemetry/traces.go:38 (0x100c78fa8)
/home/circleci/project/telemetry/telemetry.go:79 (0x10131ef2c)
/home/circleci/project/shell/run_shell_cmd.go:111 (0x10131ec99)
/home/circleci/project/shell/run_shell_cmd.go:72 (0x10131eb98)
/home/circleci/project/config/dependency.go:949 (0x10169b35c)
/home/circleci/project/config/dependency.go:781 (0x10169a4b4)
/home/circleci/project/config/dependency.go:616 (0x1016997e4)
/home/circleci/project/config/dependency.go:550 (0x101698fd4)
/home/circleci/project/config/dependency.go:474 (0x101698710)
/home/circleci/project/config/dependency.go:164 (0x1016966a4)
/home/circleci/project/config/dependency.go:414 (0x101698008)
/home/circleci/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 (0x1012d9314)
/usr/local/go/src/runtime/asm_arm64.s:1223 (0x1006936e4)

So Generated file ../B/.terragrunt-cache/932801613/_backend.tf. shows me that the remote_state get's templated, and tofu gets executed.
The line

Ignoring expected error from dependency init call

leads me to this function.

So terragrunt would have to not only generate the backendconfig, but also run all generate blocks before being sure that the remote state config is ready.

Either this, or encryption config could be a part of the remote_state block, or even become a block of its own, that is handled here like remote_state.

norman-zon · 2024-10-29T14:05:58Z

Digging deeper into the code it seems to me, that generate blocks should maybe be added to config_partial, so they can be handled the same as remoteStateTGConfig for dependencies.

Since this would implicate a design decision, I would love to hear back from @denis256 or @yhakbar before trying to put together a PR (which may not be successful, for I am not experienced in writing Golang).

yhakbar · 2024-11-06T20:52:17Z

Hey @norman-zon ,

Unless you or @denis256 disagree with me, what I think makes the most sense is to have encryption configurations pulled into remote_state, and to treat it like a first class citizen.

The reason I would recommend this is that it seems sensible to have _backend.tf contain all the logic used for state configurations, rather than looking across two files to determine what's happening to state. I would prefer that over adjusting how generate works.

yhakbar · 2024-11-06T20:54:56Z

I should also point out that there's special logic used for directly parsing dependencies from the backend, rather than going through tofu output here. That won't work for encrypted state, so it seems like remote_state will eventually need first-class support of interacting with encrypted state.

norman-zon · 2024-11-12T07:43:34Z

Thanks, @yhakbar for having a look at this and giving your opinion. I agree with you that it makes sense, to have all state related config in one place. Using generate only came to mind, because the encryption block is quite flexible compared to the backend block, so my initial thought was so handle it separately. But grouping functionally makes more sense, now that you point it out.

I would be really happy to see this implemented soon, since we are stuck in the middle of the process of state migration here.

yhakbar · 2024-11-12T14:14:23Z

If you're interested in taking a stab at the implementation, @norman-zon , we'd be happy to help. Otherwise, it might take a little longer for us to get around to this. We have docs on contributing.

Let me know either way, though. It's a really useful feature in OpenTofu, and I'm sure Terragrunt users will benefit from better support.

norman-zon · 2024-11-12T14:19:47Z

I will give it a try, but have to tell to you right away, my experiences with Go are very limited.
Nevertheless, let me see if I can get a PR up this week to get things going.

norman-zon · 2024-11-14T15:58:01Z

@yhakbar one more design question.

Would you like to have the encryption config as a block inside the remote_state block and completely defined, like the config block is now. Making it neccesary to consider all different possible key providers (pbkdf2, aws, gcp, openbao) separetely. Something like this:

encryption = {
    key_provider = "pbkdf2"
    passphrase   = "PASSPHRASE"
    method          = "aes_gcm"
    ...
}

Or would you rather like to have an arbitrary block allowing to write the encryption config directly, without any validation or help from terragrunt that just gets rendered into _backend.tf as is. Like this:

encryption = {
  key_provider "pbkdf2" "default" {
    passphrase = "PASSPHRASE"
  }
  method "aes_gcm" "default" {
    keys = key_provider.pbkdf2.default
  }
...
}

Or would you even nest the encryption block inside the config block?

yhakbar · 2024-11-14T20:48:29Z

Hey @norman-zon ,

I don't think it's possible to nest a configuration block into an attribute in HCL, but I may be mistaken on that.

If you're not aware of the difference, anything that has an = to the right of it is an attribute.

I think the simplest thing to do is likely to have an encryption configuration block added to remote_state, and to match the OpenTofu schema.

I'm not sure about the context for why we've setup remote_state to use attributes instead of configuration blocks, however, so @denis256 or @levkohimins might have different opinions on the best way to tackle this.

norman-zon · 2024-11-21T14:40:44Z

@yhakbar I just opened a PR with the most basic implementation of this feature. Could you take a look and give me feedback, if this is going in a direction you like? If so I would continue to add assertions etc.
Especially the HCL Traversal feels like I'm doing something not as intended there.

norman-zon added the bug Something isn't working label Oct 18, 2024

norman-zon mentioned this issue Oct 30, 2024

[WIP] feat: add encryption block #3525

Closed

4 tasks

norman-zon linked a pull request Nov 21, 2024 that will close this issue

feat: add encryption to remote_state #3586

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initalizing dependencies fails when state is encrypted #3495

Initalizing dependencies fails when state is encrypted #3495

norman-zon commented Oct 18, 2024 •

edited

Loading

norman-zon commented Oct 18, 2024 •

edited

Loading

norman-zon commented Oct 18, 2024

denis256 commented Oct 18, 2024

norman-zon commented Oct 22, 2024 •

edited

Loading

norman-zon commented Oct 22, 2024

norman-zon commented Oct 29, 2024

norman-zon commented Oct 29, 2024 •

edited

Loading

yhakbar commented Nov 6, 2024

yhakbar commented Nov 6, 2024

norman-zon commented Nov 12, 2024

yhakbar commented Nov 12, 2024

norman-zon commented Nov 12, 2024

norman-zon commented Nov 14, 2024 •

edited

Loading

yhakbar commented Nov 14, 2024

norman-zon commented Nov 21, 2024

Initalizing dependencies fails when state is encrypted #3495

Initalizing dependencies fails when state is encrypted #3495

Comments

norman-zon commented Oct 18, 2024 • edited Loading

Describe the bug

Steps To Reproduce

Expected behavior

Versions

norman-zon commented Oct 18, 2024 • edited Loading

norman-zon commented Oct 18, 2024

denis256 commented Oct 18, 2024

norman-zon commented Oct 22, 2024 • edited Loading

norman-zon commented Oct 22, 2024

norman-zon commented Oct 29, 2024

norman-zon commented Oct 29, 2024 • edited Loading

yhakbar commented Nov 6, 2024

yhakbar commented Nov 6, 2024

norman-zon commented Nov 12, 2024

yhakbar commented Nov 12, 2024

norman-zon commented Nov 12, 2024

norman-zon commented Nov 14, 2024 • edited Loading

yhakbar commented Nov 14, 2024

norman-zon commented Nov 21, 2024

norman-zon commented Oct 18, 2024 •

edited

Loading

norman-zon commented Oct 18, 2024 •

edited

Loading

norman-zon commented Oct 22, 2024 •

edited

Loading

norman-zon commented Oct 29, 2024 •

edited

Loading

norman-zon commented Nov 14, 2024 •

edited

Loading