Revert Recent Symlink Work and Reintroduce Behind Experiment Flag

[yhakbar]

Describe the enhancement

The recent effort to add support for symlink support in Terragrunt:

• feat: support discovery of symlinked modules #3562

Has resulted in some unexpected regression for users:

• --terragrunt-exclude-dir change in behavior #3622
• Recent Symlink Work Breaks MacOS Tests #3616

To avoid issues like this for users while we resolve edge cases and improve testing, revert the recent feature, and reintroduce it behind an experiment flag. This will give users that want to continue using the current unstable behavior a chance to adopt it, while work continues to handle edge cases.

In addition to doing this, add a standard process for managing experimental features like this.

Do something very similar to what is available for Strict Controls, but instead of allowing opt-in breakage of future breaking functionality, add a system for opt-in of future stable functionality that is still unstable.

Additional context

We need to develop a systematic approach to handling experiments anyways, and this is a good test case for the new system.

This will end up being a medium effort endeavor, with documentation requirements, introducing new flags (experiment and experiment mode), and a breaking change to make symlinks no longer work by default.

Things you might want to address include:

• Changes required.
• Implications of the feature.
• Alternatives considered.
• Level of effort.

PoC (Proof of Concept)

Link to a Proof of Concept if you have one:

• PoC

Including a PoC can help others understand the feature better and implement it faster.

RFC Not Needed

• I have evaluated the complexity of this enhancement, and I believe it does not require an RFC.

[yhakbar]

FYI, @rodrigorfk . Let me know if you object to this plan of action.

[rodrigorfk]

Hi @yhakbar, I feel really sorry for introducing the regressions and for missing your message here, I only saw this today when the release was created.

No problem from my side and thank you very much for working on it, I should have payed closer attention to the discussions here.

Moving forward, how can I help mitigating the regressions introduced? I can take a look on the issues mentioned above and look back on my original implementation and see why it is not working for those cases.

[yhakbar]

@rodrigorfk , nothing to apologize for! Sometimes bugs are the price of progress!

It does seem like folks are very actively interested in this feature, so I'm happy to hear you're interested in contributing to addressing the regression.

Take a look at these docs:
https://terragrunt.gruntwork.io/docs/reference/experiments/#symlinks

Going forward, we want to have documentation like that to explicitly detail what is required to consider an experiment stable for general use.

Those items largely relate to the issues linked above. If you can think of anything else we should add to safely make it the default, please add to the list of tasks before release.

Adding the macOS integration tests will probably go a long way, as almost all of our tests run in temporary directories, and it seems like macOS puts temp dirs in symlinked directories for some reason.