[Merged by Bors] - Initial work on config merging

[nightkr]

Description

This is a starting point for #362, but is intended to be usable outside of that context (in this case, the primary motivation for getting this done now is #365).

Work Checklist until R4R

• Merging "primitive" types like u8 or String
• Merging arbitrarily nested structs
• Merging enums
• Merging boxed stuff
• Merging lists
  • This will probably depend on the lists in question, might want to defer this for a later PR
• Docs

Review Checklist

• Code contains useful comments
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[nightkr]

We might want to reorganize this into a "virtual root crate + members" layout, but I'd rather split that into a separate PR to avoid cluttering the diff view too much.

[maltesander]

Im still on the code, just 2 findings to not get lost.
stackable-operator-derive needs to be added to exceptions in the deny.toml.

[nightkr]

Hopefully cargo deny should be happy now...

[maltesander]

LGTM. Works fine for the usecases i could come up with.
As discussed with @teozkr, im fine with merging for now and see what we stumble across later on.

[nightkr]

I've been experimenting a bit further with this in feature/merge-experiments, but never really got to a point that I was happy with there. In particular:

• cbb7f0b changes our uses of Option into a new Inheritable enum to try to clarify the semantics we want a bit better. This leaves Option available for semantically optional fields. However, this has a bunch of downsides:
  • Serde doesn't like differentiating between missing and null fields, the closest I could get required sprinkling #[serde(default)] on every field, which is.. far from ideal
  • This is pretty much impossible to represent idiomatically in Go, which I suspect will cause problems for getting K8s to accept the schema definitions down the line (I didn't even try this yet at that point)
• 19bdb83 and 0bebbe7 further split up Inheritable into the Atomic and Enum enums, allowing any type (that conforms to the regular Deserialize+Serialize+JsonSchema requirements) to be used as an atomic, but arguably this is an even worse situation to be in because it further increases complexity without really fixing the root issues
• 7dc134c (the current state of the feature/merge branch that this PR concerns) overloads Option between "this is optional" and "this should be inherited", leaving the differentiation between those two cases for a future validation step.. this also has a few disadvantages
  • It's impossible to override a field to null once a value has been set
  • It's unclear from the struct itself whether a field should actually be optional

I'm starting to believe that "Option everywhere" is still the most viable approach though, and that we'll have to handle cases where we want to allow override-to-null separately. I think the "unclear optionality" problem should get better once we have the validation layer in place, at which point the mergeable fragments would likely be generated by the validation layer anyway.

[soenkeliebau]

I have tested the code in here with my draft PR in NiFi and everything works as expected. I'd be happy to merge this.

However, I f****d up when trying to merge main into this PR earlier today and may have broken other code, not sure..

[nightkr]

bors r+

[bors]

Pull request successfully merged into main.

Build succeeded:

• All tests passed