[Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline

[felipepessoto]

The Delta Spark UT pipeline landed in #12388 (fixes #9296). It runs delta-io/delta's spark test suite against a Gluten Velox bundle and gates each run against a committed baseline of known failures.

That baseline is currently 735 known failures out of 19,073 tests. This issue tracks the work to bring that number down and to improve the pipeline itself. It is a tracking issue only — discussion belongs in the linked issues.

1. Correctness gaps behind the 735 baseline entries

These are not a consequence of Gluten declining to offload: a plan Gluten cannot offload should fall back to vanilla Spark and the test should still pass. Most entries below are bugs -- fallback not happening, metrics differing from vanilla, or the native side failing. A separate group is expected rather than fixable here: tests that assert on the query plan see a different plan once Gluten offloads, which is by design. Those are worth splitting out during triage, since they should either be fixed upstream (as delta-io/delta#7104 and delta-io/delta#7105 were) or stay in the baseline permanently.

The largest clusters, counted from known-failures.txt by suite:

Tests	Area	Issue
130	DescribeDeltaHistorySuite + ...WithCatalogOwnedBatch100Suite	needs triage
100	stats.DataSkippingDeltaV1* (7 suite variants, 34 of them column-mapping)	needs triage
64	ImplicitMergeCastingSuite + ImplicitStreamingMergeCastingSuite (overflow / storeAssignmentPolicy)	needs triage
42	commands.backfill.RowTrackingBackfillConflicts(DV)Suite	needs triage
14	io.delta.tables.DeltaTableHadoopOptionsSuite — Hadoop FS options	#4963

Two clusters cut across suites, so they don't show up in the table above (counts from the triage in #12388): ~226 tests failing on IncrementMetric (#9003) and ~47 on the Delta stats tracker ClassCastException when the stats plan can't be offloaded (#12538).

Related type-support gap: TIMESTAMP_NTZ (#11622), which is one trigger of #12538.

• [VL] IncrementMetric doesn't work in some cases #9003 — IncrementMetric
• [VL][Delta] Gluten Delta stats tracker should fall back when the stats plan can't be offloaded (ClassCastException on TIMESTAMP_NTZ) #12538 — Delta stats tracker should fall back instead of throwing
• [VL] Support TIMESTAMP_NTZ Type #11622 — TIMESTAMP_NTZ support
• Hadoop file system options doesn't work #4963 — Hadoop file system options
• Triage the DescribeDeltaHistory, ImplicitMergeCasting, RowTrackingBackfill and DataSkipping clusters and file one issue per root cause

2. Native bugs currently worked around in CI

Both of these are hidden by the pipeline today, so the workarounds should be removed together with the fixes.

• [VL] Native Delta DV bitmap aggregator aborts on a Long.MAX_VALUE sentinel row index during MERGE with deletion vectors (intermittent VeloxRuntimeError) #12377 — DV bitmap aggregator aborts on an invalid row index. Currently quarantined by error signature in flaky-error-patterns.txt (2 patterns). Remove those entries when fixed.
• [VL] Native memory OOMs the test/executor JVM when reading large (billions-of-rows) Delta tables with deletion vectors #12387 — native memory OOM when reading billions-of-rows tables. Two DeletionVectorsSuite 2B-row tests are force-failed in setup-delta.sh to stop them OOM-killing the shard. Remove that patch when fixed.
• Unsupported operations should induce a fallback for Delta plans rather than failing (raised by @zhztheplayer in [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline #12388; [VL][Delta] Gluten Delta stats tracker should fall back when the stats plan can't be offloaded (ClassCastException on TIMESTAMP_NTZ) #12538 is the concrete instance)

3. Pipeline improvements

• Automatically remove now-passing tests from the Delta baseline. [MINOR][CI] Automate nightly Delta known-failures updates #12855 opens or updates a deletion-only PR from the scheduled run, covering core-library fixes such as [GLUTEN-10134][CI] Remove Delta known-failures fixed by cast mode preservation #12815 and [MINOR][CI] Remove now-passing Delta type widening tests from known failures #12851 that did not trigger Delta CI when merged.
• Move compiler and dependency caches to Apache Stash. Migrated ccache, Maven, and sbt/Ivy/Coursier caches in [MINOR][CI] Move Delta Spark UT caches to Apache Stash #12820.
• Cache and reuse the fully built Velox native library across workflows (not ccache). gluten-delta/** and backends-velox/src-delta*/** match both velox_backend_x86.yml and delta_spark_ut.yml, so those PRs build the native lib twice (~10 min). Suggested by @philo-he and @zhouyuan. A prebuilt-library cache is preferable to artifact-passing: artifacts are scoped to a workflow run and there is no cross-workflow needs, so the Delta workflow would have to poll a sibling run it races with, whereas a prebuilt-library cache populated by push-to-main has no ordering problem. The key must cover cpp/**, the build flags and the base image — unlike ccache, where a bad key only costs a rebuild, a stale hit on a prebuilt lib means silently testing the wrong binary. A miss must fall back to building.
• Cover more Spark versions. The workflow already takes spark_version / delta_ref inputs, and pom.xml pairs Spark 3.5→Delta 3.3.2, 4.0→4.0.1, 4.1→4.1.0. The blocker is the baseline: each combination needs its own known-failures.txt and a bootstrap run. Suggested starting point: Spark 3.5 as a nightly-only second leg, which keeps per-PR cost unchanged. Requested by @zhouyuan.
• Remove the overlapping imported Delta suites. @malinjawi identified 8 suites imported from delta-io/delta (DeltaSuite, DeleteSQLSuite + DV variants, UpdateSQLSuite, DeltaDDLSuite, DeltaInsertIntoTableSuite, ClusteredTableClusteringSuite, OptimizeMetadataOnlyDeltaQuerySuite, OptimizedWritesSuite) that this pipeline now duplicates — but only on the Spark 4.1 leg. They are the only Delta coverage on Spark 3.5 and 4.0, so this is blocked on the Spark 3.5 leg above. Raised by @zhztheplayer.
• Delta 4.3.0 support. Attempted during [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline #12388: the bundle is compiled against Delta 4.1.0 and hits a binary-incompatible change (IdentityColumn.logTableWrite, first param Snapshot → SnapshotDescriptor), which NoSuchMethodErrors on every write. Needs the bundle built against 4.3.0.
• /delta-test PR comment opt-in. Replaces the proposed run-delta-ci label: contributors cannot apply labels, but anyone can comment /delta-test to force the full suite on a PR skipped by the paths: filter. Implemented in [GLUTEN-12743][CI] Let contributors trigger the Delta Spark UT with a /delta-test PR comment #12781.
• Drop the upstream cherry-picks ([Spark] [Test] Collect scans by FileSourceScanLike in ScanReportHelper delta-io/delta#7104 + [VL] Remove a limit for BHJ in stage fallback policy #7105, FileSourceScanExec → FileSourceScanLike) once DELTA_REF moves to a release that contains them. setup-delta.sh already skips a cherry-pick that is already applied, so this is cleanup.

Notes for whoever picks these up

• Refresh the baseline by running Delta Spark UT (Gluten) with update_baseline=true and committing the produced artifact; see .github/workflows/util/delta-spark-ut/README.md. - A test that fails consistently belongs in known-failures.txt. flaky-tests.txt / flaky-error-patterns.txt are only for tests that genuinely pass on some runs and fail on others — quarantining a deterministic failure hides a real bug. - The nightly run enforces fail_on_fixed=true, so when a fix lands, the nightly goes red until the now-passing tests are removed from the baseline. That is the intended signal.