Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Override exactly according to the Terraform spec #2124

Merged
merged 9 commits into from
Oct 14, 2024
Merged

Conversation

wata727
Copy link
Member

@wata727 wata727 commented Sep 16, 2024

Fixes #2114

As explained in #2114, the current override behavior is incomplete. The override behavior up to v0.53 is as follows:

  • Attributes in blocks with matching headers (type and labels) are merged.
  • Nested blocks are also merged recursively.
  • If there are multiple blocks with the same header (e.g. terraform, locals), only one of them will be overridden. Which one gets overwritten is non-deterministic.
  • The order in which override files are processed is completely random, not lexicographical.
# main.tf
resource "aws_instance" "foo" {
  instance_type = "t2.micro" # => Should be c5.xlarge, but can be m5.xlarge

  # The entire block is overwritten, the volume_type should be discarded, but only the volume_size is overwritten
  ebs_block_device {
    volume_type = "gp3"
    volume_size = 20
  }
}

# main1_override.tf
resource "aws_instance" "foo" {
  instance_type = "m5.xlarge"

  ebs_block_device {
    volume_size = 50
  }
}

# main2_override.tf
resource "aws_instance" "foo" {
  instance_type = "c5.xlarge"
}
# main.tf
terraform {
  backend "s3" {}
}

terraform {
  # If the terraform block below is overridden then "google" will be merged,
  # but if the one above is overridden then it will be ignored.
  required_providers {
    aws = {}
  }
}

# main_override.tf
terraform {
  required_providers {
    google = {}
  }
}

This PR fixes the override behavior to follow the Terraform spec. The following changes will be made:

In most cases, this change should be considered a bug fix as it results in stricter Terraform override behavior, and should have little to no impact unless you are doing complex overrides.

Finally, the behavior of overriding for multiply declarable blocks (e.g. terraform, locals) requires some caution: the GetModuleContent API returns different formats depending on whether there is one or more blocks.

For example, if only one required_providers is declared, the override will apply for that block:

# main.tf
terraform {
  # Only one block with required_providers is returned, containing 3 attributes: "aws", "google", and "azurerm".
  required_providers {
    aws = {}
  }
}

# main_override.tf
terraform {
  required_providers {
    google = {}
  }
}

terraform {
  required_providers {
    azurerm= {}
  }
}

On the other hand, if multiple required_providers are declared, the new attributes will be returned as a new block:

# main.tf
terraform {
  required_providers {
    aws = {}
  }
}

terraform {
  required_providers {
    google = {} # => This will be overridden
  }
}

# main_override.tf
terraform {
  required_providers {
    google = {}
    azurerm = {} # => This is not merged with either one and is returned as a new block, meaning the caller receives 3 "terraform" blocks.
  }
}

This is because it is not obvious which block the new attribute should be merged into.

See https://developer.hashicorp.com/terraform/language/files/override#merging-behavior

- Within a top-level block, any nested blocks within an override block replace all blocks of the same type in the original block. Any block types that do not appear in the override block remain from the original block.
- The contents of nested configuration blocks are not merged.
See https://developer.hashicorp.com/terraform/language/files/override#merging-behavior

If more than one override file defines the same top-level block, the overriding effect is compounded, with later blocks taking precedence over earlier blocks. Overrides are processed in order first by filename (in lexicographical order) and then by position in each file.

Regarding the position in each file, no additional considerations are necessary since hclext.Blocks are already sorted.
See https://developer.hashicorp.com/terraform/language/files/override#merging-terraform-blocks

The settings within terraform blocks are considered individually when merging.

If the required_providers argument is set, its value is merged on an element-by-element basis, which allows an override block to adjust the constraint for a single provider without affecting the constraints for other providers.

In both the required_version and required_providers settings, each override constraint entirely replaces the constraints for the same component in the original block. If both the base block and the override block both set required_version then the constraints in the base block are entirely ignored.

The presence of a block defining a backend (either cloud or backend) in an override file always takes precedence over a block defining a backend in the original configuration. That is, if a cloud block is set within the original configuration and a backend block is set in the override file, Terraform will use the backend block specified in the override file upon merging. Similarly, if a backend block is set within the original configuration and a cloud block is set in the override file, Terraform will use the cloud block specified in the override file upon merging.
See also https://developer.hashicorp.com/terraform/language/files/override#merging-locals-blocks

Each locals block defines a number of named values. Overrides are applied on a value-by-value basis, ignoring which locals block they are defined in.
See https://developer.hashicorp.com/terraform/language/files/override#merging-resource-and-data-blocks

Within a resource block, the contents of any lifecycle nested block are merged on an argument-by-argument basis. For example, if an override block sets only the create_before_destroy argument then any ignore_changes argument in the original block will be preserved.

If an overriding resource block contains one or more provisioner blocks then any provisioner blocks in the original block are ignored.

If an overriding resource block contains a connection block then it completely overrides any connection block present in the original block.
If there is only a single locals/required_providers in the primary,
we will merge it rather than append it to maintain backwards compatibility.
For optimal performance, sort override files at parse time
instead of sorting in PartialContent all the time.
@wata727 wata727 marked this pull request as ready for review October 6, 2024 16:26
@wata727 wata727 merged commit 47067d0 into master Oct 14, 2024
14 checks passed
@wata727 wata727 deleted the override_blocks branch October 14, 2024 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Override files are not compliant with the Terraform spec
1 participant