Use signals not only for error-handling

[norman-zon]

Describe the enhancement

I really like the idea of the new error block. Especially the signals attribute for usage in automated CI environments.

My request is to expand the usage of signals to more than just error handling, but to be able to handle any context.

One use case that comes immediately to mind is to have a signal for the plan result, to the CI can react differently on 0-diff, changes and/or deletions.

I would imagine a usage like this:

signal "signal_changes" {
  matches = [
    "^Plan:.*\b[1-9][0-9]*\b to change\b" # match non-zero changes
  ]
  message = "Plan contains changes" # Optional
  signals = {
      plan_changes = true
  }
}
signal "signal_deletions" {
  matches = [
    "^Plan:.*\b[1-9][0-9]*\b to destroy\b" # match non-zero deletions
  ]
  message = "Plan contains deletions" # Optional
  signals = {
    plan_destroy = true
  }
}

Additional context

• I cannot tell how much of an performance impact it would be to regex-match the whole stream instead of only the error-stream.

• Alternatively, for this exact use-case the plan output could be written to a JSON file directly. But expanding the usage of signals is much more flexible.

RFC Not Needed

• I have evaluated the complexity of this enhancement, and I believe it does not require an RFC.

[yhakbar]

Hey @norman-zon ,

Folks typically use plan -out for this kind of thing, which is a lot more reliable than inspecting stdout, as you have a serialized representation of the plan that can be analyzed by external tools more efficiently and precisely.

Is there a reason you wouldn't just use plan -out and an after_hook?

[norman-zon]

Yes, there is a reason. I use state and plan encryption, so I can't process the plan-file with other tools than tofu.

[yhakbar]

Sorry, I haven't played with state encryption yet, so I might be missing some important context here.

Are you not able to have tofu emit the JSON representation of your plan like this?

# main.tf
resource "local_file" "example" {
	filename = "example.txt"
	content  = "Hello, World!"
}

output "example" {
	value = local_file.example.content
}
# terragrunt.hcl
# Intentionally empty.

$ terragrunt plan -out plan.out
...
$ terragrunt show -json plan.out | jq
{
  "format_version": "1.2",
  "terraform_version": "1.8.5",
  "planned_values": {
    "outputs": {
      "example": {
        "sensitive": false,
        "type": "string",
        "value": "Hello, World!"
      }
    },
    "root_module": {
      "resources": [
        {
          "address": "local_file.example",
          "mode": "managed",
          "type": "local_file",
          "name": "example",
          "provider_name": "registry.opentofu.org/hashicorp/local",
          "schema_version": 0,
          "values": {
            "content": "Hello, World!",
            "content_base64": null,
            "directory_permission": "0777",
            "file_permission": "0777",
            "filename": "example.txt",
            "sensitive_content": null,
            "source": null
          },
          "sensitive_values": {
            "sensitive_content": true
          }
        }
      ]
    }
  },
  "resource_changes": [
    {
      "address": "local_file.example",
      "mode": "managed",
      "type": "local_file",
      "name": "example",
      "provider_name": "registry.opentofu.org/hashicorp/local",
      "change": {
        "actions": [
          "create"
        ],
        "before": null,
        "after": {
          "content": "Hello, World!",
          "content_base64": null,
          "directory_permission": "0777",
          "file_permission": "0777",
          "filename": "example.txt",
          "sensitive_content": null,
          "source": null
        },
        "after_unknown": {
          "content_base64sha256": true,
          "content_base64sha512": true,
          "content_md5": true,
          "content_sha1": true,
          "content_sha256": true,
          "content_sha512": true,
          "id": true
        },
        "before_sensitive": false,
        "after_sensitive": {
          "sensitive_content": true
        }
      }
    }
  ],
  "output_changes": {
    "example": {
      "actions": [
        "create"
      ],
      "before": null,
      "after": "Hello, World!",
      "after_unknown": false,
      "before_sensitive": false,
      "after_sensitive": false
    }
  },
  "prior_state": {
    "format_version": "1.0",
    "terraform_version": "1.8.5",
    "values": {
      "outputs": {
        "example": {
          "sensitive": false,
          "value": "Hello, World!",
          "type": "string"
        }
      },
      "root_module": {}
    }
  },
  "configuration": {
    "provider_config": {
      "local": {
        "name": "local",
        "full_name": "registry.opentofu.org/hashicorp/local"
      }
    },
    "root_module": {
      "outputs": {
        "example": {
          "expression": {
            "references": [
              "local_file.example.content",
              "local_file.example"
            ]
          }
        }
      },
      "resources": [
        {
          "address": "local_file.example",
          "mode": "managed",
          "type": "local_file",
          "name": "example",
          "provider_config_key": "local",
          "expressions": {
            "content": {
              "constant_value": "Hello, World!"
            },
            "filename": {
              "constant_value": "example.txt"
            }
          },
          "schema_version": 0
        }
      ]
    }
  },
  "relevant_attributes": [
    {
      "resource": "local_file.example",
      "attribute": [
        "content"
      ]
    }
  ],
  "timestamp": "2024-12-04T14:30:48Z",
  "errored": false
}

I don't think anything about state encryption should prevent you from having tofu write out the plan in json format like that for processing with another tool.

The convenient way to have this always happen with Terragrunt would be to set this up:

# terragrunt.hcl
terraform{
    extra_arguments "write_to_out" {
        commands  = ["plan"]
        arguments = ["-out=plan.out"]
    }
    after_hook "show_plan_as_json" {
        commands = ["plan"]
        execute  = ["bash", "-c", "$TG_CTX_TF_PATH show -json plan.out > plan.json"]
    }
    after_hook "detect_actions" {
        commands = ["plan"]
        # If you only care about one kind of action:
        # execute  = ["bash", "-c", "jq '.resource_changes[] | select(.change.actions[] | contains(\"create\")) | .address' < plan.json > creates.txt"]
        # If you care about all actions:
        execute = ["bash", "-c", "jq '.resource_changes[] | select(.change.actions[]) | { action: .change.actions[0], address: .address }' plan.json > changes.json"]
    }
}

In my testing, it results in this changes.json

{
  "action": "create",
  "address": "local_file.example"
}

I wrote this up quickly and tested minimally. You might have to do more tweaking to get it to do what you want.

[norman-zon]

Yes, that would work, but my intention of using state encryption is to not have any unencrypted state in my CI logs at all.

I have to ponder wether directly piping into jq without an intemediate file like terragrunt show -json plan.out | jq '.resource_changes[] | select(.change.actions[]) | { action: .change.actions[0], address: .address }' would be safe enough.

Thanks a lot for the jq-query by the way! This is really handy.

[yhakbar]

Hey @norman-zon , what do you think about this enhancement? Should we still pursue it?

One thing I didn't see described is how it would integrate with the existing signals system in the errors block. Would that be deprecated in favor of a dedicated signals block that also hooks into errors?

[norman-zon]

I think it would be a helpful feature for CI in general, despite my first idea of an use case might better be handled like you described above.

And yes, I would see it as a separate system, that can also hook into errors.

[yhakbar]

In that case, I would like to see the proposal fleshed out a bit more with an RFC, as it would be a significant new feature in Terragrunt.

Considerations like the following would be important there:

1. What is the name used for the signal file? Can it be changed?
2. Where will it end up? Can that change?
3. How are triggers controlled? Is it always reading stdout/stderr? Should there be a dedicated trigger block that can be used to extend behavior?
4. How do conflicting signals get handled? (e.g. two signals that write the same file with different content).
5. What is the deprecation path for having the errors block signal being pulled into this new error system? Would it be a trigger that included exit codes? Should the existing signal system on errors be supported indefinitely?
6. Is there any notion of "capture"? In your example, you have a regex pattern, which could have included a capture group: "^Plan:.*\b[1-9][0-9]*\b to change\b". Is it a good idea to have that be something that can be emitted into a signal?

[norman-zon]

Title: Enhance Terragrunt's signals Functionality for Comprehensive Output Handling

Summary:
This RFC proposes extending Terragrunt's signals feature to handle various command outputs beyond error messages. The enhancement includes introducing a configurable signal block, allowing users to define custom patterns for matching outputs from stdout, stderr, or both. Matched patterns will trigger the generation of a JSON file (default: signals.json) containing user-defined signals, facilitating more nuanced automation workflows in CI/CD environments.

Motivation:
The current signals attribute within the error block is limited to error handling. Expanding this functionality to process general command outputs enables CI systems to react dynamically based on specific conditions, such as detecting infrastructure changes or deletions during a Terraform plan.

Detailed Design:

1. Signal Block Configuration:
   Introduce a new signal block in the Terragrunt configuration file, allowing users to define custom patterns and associated signals.

   signal "detect_changes" {
     matches = [
       "^Plan:.*\\b([1-9][0-9]*)\\b to add\\b",    # Matches non-zero additions
       "^Plan:.*\\b([1-9][0-9]*)\\b to change\\b", # Matches non-zero changes
       "^Plan:.*\\b([1-9][0-9]*)\\b to destroy\\b" # Matches non-zero deletions
     ]
     message = "Plan indicates infrastructure changes." # Optional message
     signals = {
       plan_changes = true
     }
     output_file = "custom_signals.json" # Optional; defaults to "signals.json"
   }

2. Output File Configuration:

   • Default Behavior: The signals will be written to a JSON file named signals.json in the Terragrunt root directory.
   • Customization: Users can specify a different filename using the output_file attribute within the signal block.
   • CLI Override: A new CLI flag --signal-output-dir will allow users to define a custom directory for the output file, overriding the default location.

3. Trigger Sources:

   • Configuration: Introduce an attribute trigger_source within the signal block to specify the source of the output to monitor: stdout, stderr, or both.
   • Default Value: If not specified, the default will be both.

   signal "detect_errors" {
     matches = [
       "ERROR:.*"
     ]
     message = "Error detected in stderr output."
     signals = {
       error_detected = true
     }
     trigger_source = "stderr"
   }

4. File Locking Mechanism:
   Implement file locking to prevent concurrent write conflicts to the output file. This ensures data integrity when multiple processes attempt to write signals simultaneously.

5. Deprecation of Existing Signals in Error Block:
   Given the recent introduction of the signals feature, the existing signals attribute within the error block will be deprecated and removed. Users will need to transition to the new signal block for defining signals, acknowledging this as a breaking change.

6. Output Capturing:
   Terragrunt will capture the specified outputs (stdout, stderr, or both) during command execution to facilitate pattern matching as defined in the signal blocks.

Example of Output Structure in JSON with Capture Groups:

When a signal block matches a pattern in the command output, Terragrunt will generate a JSON file (e.g., signals.json) with the following structure:

{
  "signals": {
    "plan_changes": true
  },
  "matches": [
    {
      "pattern": "^Plan:.*\\b([1-9][0-9]*)\\b to add\\b",
      "matched_text": "Plan: 2 to add, 0 to change, 0 to destroy",
      "capture_groups": {
        "1": "2"
      }
    },
    {
      "pattern": "^Plan:.*\\b([1-9][0-9]*)\\b to change\\b",
      "matched_text": null,
      "capture_groups": {}
    },
    {
      "pattern": "^Plan:.*\\b([1-9][0-9]*)\\b to destroy\\b",
      "matched_text": null,
      "capture_groups": {}
    }
  ],
  "message": "Plan indicates infrastructure changes."
}

Implementation Steps:

1. Define the signal Block Syntax: Establish the configuration structure for the new signal block, including attributes like matches, message, signals, output_file, and trigger_source.
2. Pattern Matching Logic: Implement functionality to parse specified command outputs and match them against user-defined patterns within the signal blocks.
3. Signal Handling Mechanism: Develop a system to manage and expose the signals set by the signal blocks, writing them to the configured JSON output file.
4. File Locking Implementation: Incorporate file locking mechanisms to ensure safe concurrent access to the output file.
5. CLI Flag Addition: Add the --signal-output-dir CLI flag to allow users to specify a custom directory for the output file.
6. Deprecation Notice: Update documentation and provide guidance for users to transition from the deprecated signals attribute in the error block to the new signal block.
7. Documentation and Examples: Update Terragrunt's documentation to include the new signal block, providing examples and guidelines for users.

Backward Compatibility:
This proposal introduces a breaking change by deprecating the existing signals attribute within the error block. Users will need to update their configurations to adopt the new signal block. Comprehensive documentation and migration guides will be provided to facilitate this transition.

Drawbacks:

• Breaking Change: Deprecating the existing signals attribute requires users to modify their configurations, which may cause inconvenience.
• Increased Complexity: Introducing additional configuration options adds complexity to Terragrunt's setup, potentially increasing the learning curve for new users.

Alternatives:

• Custom Scripting: Users can implement custom scripts in their CI/CD pipelines to parse Terraform outputs and determine actions. However, this approach lacks the integration and convenience of having the functionality within Terragrunt.

Unresolved Questions

• Conflicting signals: Using file locking to prevent conflicts in the output file is one possible option, but may not always lead to the expected result.