feat(core): configurable merge options for Delta merge modes

[malon64]

Summary

• add optional sink.accepted.merge config for Delta merge write modes with:
  • ignore_columns
  • compare_columns
  • scd2.current_flag_column, scd2.valid_from_column, scd2.valid_to_column
• keep backward compatibility: when merge is omitted, merge behavior remains unchanged
• wire options into runtime merge execution:
  • SCD1 excludes configured ignore_columns from update set
  • SCD2 uses configured compare set (compare_columns if provided; else non-key business columns minus ignore_columns)
  • SCD2 uses configured system column names instead of hardcoded defaults

Validation Rules

• sink.accepted.merge is only valid when:
  • sink.accepted.format = delta
  • sink.write_mode is merge_scd1 or merge_scd2
• ignore_columns and compare_columns must exist in schema
• merge key columns (schema.primary_key) are forbidden in ignore_columns and compare_columns
• SCD2 system column names must be non-empty, unique, and must not collide with business columns
• merge.scd2 options are only allowed with merge_scd2

Defaults and Migration Notes

• defaults remain:
  • __floe_is_current
  • __floe_valid_from
  • __floe_valid_to
• existing configs do not require changes
• teams can opt in incrementally by adding sink.accepted.merge

Docs

• update docs/config.md with sink.accepted.merge block and examples
• update docs/write_modes.md with merge option semantics
• add architecture note in docs/design/merge_strategy_architecture.md

Tests

• unit tests added for parse + validation matrix under crates/floe-core/tests/unit/config/
• integration tests added in crates/floe-core/tests/integration/delta_run.rs for:
  • SCD2 custom system column names
  • compare/ignore-driven change detection
  • default behavior unchanged when merge block is absent

Validation

• cargo fmt --all
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test --all

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80e6070833

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".