From 1d1d345aee1d4119dde7cd952e3663a9dab7f2b9 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 25 Jun 2026 20:20:58 +0000 Subject: [PATCH 01/28] [GLUTEN][CI] Add Delta Spark UT pipeline gated against a known-failures baseline Run delta-io/delta's `spark` ScalaTest suite against a Gluten Velox bundle in CI and gate the results against a committed baseline so the many expected Delta-on- Gluten failures stay manageable and can be fixed incrementally without letting currently-passing tests silently regress. What it adds (.github/workflows/util/delta-spark-ut/): - delta_spark_ut.yml: builds the native lib + Gluten bundle, then runs the Delta spark suite sharded by suite into 4 shards x 4 forked test JVMs (~16-way), and gates each shard against the baseline. - compare-test-results.py: the gate. Per shard, regressions (failed not in the baseline) fail the build; newly-passing baselined tests are flagged so the baseline can be tightened. Also supports seed/aggregate modes. - known-failures.txt: the committed baseline of expected failures. - setup-delta.sh: clones Delta, injects the Gluten bundle, patches DeltaSQLCommandTest, and force-fails the two DeletionVectorsSuite 2B-row tests whose native row-index materialization OOM-kills the runner and hangs the shard. - README.md: how the pipeline, gating and baseline-refresh work. The workflow also carries a hang watchdog that thread-dumps and kills a wedged fork, and tunes the per-fork heap (2G) and off-heap (2G) to fit the ~16G runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 667 ++++++++++++ .../workflows/util/delta-spark-ut/README.md | 112 ++ .../delta-spark-ut/compare-test-results.py | 467 +++++++++ .../util/delta-spark-ut/known-failures.txt | 977 ++++++++++++++++++ .../util/delta-spark-ut/setup-delta.sh | 177 ++++ 5 files changed, 2400 insertions(+) create mode 100644 .github/workflows/delta_spark_ut.yml create mode 100644 .github/workflows/util/delta-spark-ut/README.md create mode 100644 .github/workflows/util/delta-spark-ut/compare-test-results.py create mode 100644 .github/workflows/util/delta-spark-ut/known-failures.txt create mode 100755 .github/workflows/util/delta-spark-ut/setup-delta.sh diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml new file mode 100644 index 00000000000..8a7d91f2c06 --- /dev/null +++ b/.github/workflows/delta_spark_ut.yml @@ -0,0 +1,667 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle +# that is built from the source in this repository. The pipeline: +# +# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64). +# 2. Builds the Gluten Java/Scala jars and assembles the +# `gluten-velox-bundle-spark_-linux_amd64-.jar` +# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile. +# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`), +# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/` +# -- see setup-delta.sh for the unmanagedJars scoping rationale), +# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin, +# and runs `sbt spark/test` sharded across the matrix. +# +# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose +# of validating Gluten changes against the latest Delta release. + +name: Delta Spark UT (Gluten) + +on: + workflow_dispatch: + inputs: + delta_ref: + description: 'delta-io/delta git ref (tag/branch/SHA) to test against' + required: true + default: 'v4.2.0' + spark_version: + description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)' + required: true + default: '4.1' + test_parallelism: + description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)' + required: true + default: '4' + update_baseline: + description: 'Seed/refresh the known-failures baseline instead of enforcing it' + type: boolean + required: false + default: false + fail_on_fixed: + description: 'Fail when a baseline test now passes (keeps the baseline honest)' + type: boolean + required: false + default: true + pull_request: + paths: + - '.github/workflows/delta_spark_ut.yml' + - '.github/workflows/util/delta-spark-ut/**' + - 'gluten-delta/**' + - 'backends-velox/src-delta40/**/DeltaSQLCommandTest.scala' + +env: + ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true + MVN_CMD: 'build/mvn -ntp' + CCACHE_DIR: "${{ github.workspace }}/.ccache" + # Gluten profile / bundle naming for the build-gluten-bundle and + # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's + # default Spark version (4.1.0) from project/CrossSparkVersions.scala. + GLUTEN_SPARK_PROFILE: 'spark-4.1' + GLUTEN_SCALA_PROFILE: 'scala-2.13' + GLUTEN_JAVA_PROFILE: 'java-17' + GLUTEN_BUNDLE_SPARK_VERSION: '4.1' + GLUTEN_BUNDLE_SCALA_VERSION: '2.13' + # Default values used when the workflow is triggered by pull_request + # (where `inputs.*` is empty). Keep these in sync with the workflow_dispatch + # defaults above. + DELTA_REF_DEFAULT: 'v4.2.0' + DELTA_SPARK_VERSION_DEFAULT: '4.1' + DELTA_TEST_PARALLELISM_DEFAULT: '4' + # Default mode for pull_request runs (where inputs.* is empty): enforce the + # committed baseline and fail when a baseline test starts passing. Override + # via the workflow_dispatch inputs above. + DELTA_UPDATE_BASELINE_DEFAULT: 'false' + DELTA_FAIL_ON_FIXED_DEFAULT: 'true' + DELTA_SCALA_VERSION: '2.13.16' + # Number of shards in the delta-spark-test matrix. Must equal the length of + # the `shard` matrix below. + # + # EXPERIMENT: 4 shards x TEST_PARALLELISM_COUNT=4 (vs production 16 shards x 1). + # Both give ~16-way parallelism, but this packs it into 4 runner jobs (4 forks + # each) instead of 16 single-fork jobs -- fewer concurrent runners for the same + # throughput. Sharding is by SUITE; total work (~1250 shard-minutes) is fixed. + # RISK: each forked test JVM uses ~4G (2G heap + 2G off-heap), so 4 forks atop + # the sbt launcher push the ~16G runner to its limit and may OOM on heavy suites + # -- which is why production uses TEST_PARALLELISM_COUNT=1. Measuring whether it + # fits now that the worst memory hog (DeletionVectorsSuite 2B-row) is force-failed. + DELTA_NUM_SHARDS: '4' + +concurrency: + group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} + cancel-in-progress: true + +jobs: + build-native-lib-centos-7: + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Get Ccache + uses: actions/cache/restore@v4 + with: + path: '${{ env.CCACHE_DIR }}' + key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}} + restore-keys: | + ccache-delta-spark-ut-centos7-release-default + ccache-centos7-release-default + - name: Build Gluten native libraries + run: | + docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c " + set -e + yum install tzdata -y + df -a + cd /work + export CCACHE_DIR=/work/.ccache + export CCACHE_MAXSIZE=1G + mkdir -p /work/.ccache + ccache -sz + bash dev/ci-velox-buildstatic-centos-7.sh + ccache -s + mkdir -p /work/.m2/repository/org/apache/arrow/ + cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/ + " + - name: Save Ccache + if: always() + uses: actions/cache/save@v4 + with: + path: '${{ env.CCACHE_DIR }}' + key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}} + - uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-native-lib-centos-7-${{github.sha}} + path: ./cpp/build/ + if-no-files-found: error + - uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} + path: .m2/repository/org/apache/arrow/ + if-no-files-found: error + + build-gluten-bundle: + needs: build-native-lib-centos-7 + runs-on: ubuntu-22.04 + container: apache/gluten:centos-9-jdk17 + steps: + - uses: actions/checkout@v4 + - name: Download native artifacts + uses: actions/download-artifact@v4 + with: + name: delta-spark-ut-native-lib-centos-7-${{github.sha}} + path: ./cpp/build/ + - name: Download Arrow jars + uses: actions/download-artifact@v4 + with: + name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} + path: /root/.m2/repository/org/apache/arrow/ + - name: Cache Maven repository + uses: actions/cache@v4 + with: + path: /root/.m2/repository + key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }} + restore-keys: | + m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}- + m2-delta-spark-ut-bundle- + - name: Build Gluten Velox + Delta bundle + run: | + set -euo pipefail + yum install -y java-17-openjdk-devel + export JAVA_HOME=/usr/lib/jvm/java-17-openjdk + export PATH=$JAVA_HOME/bin:$PATH + java -version + cd "$GITHUB_WORKSPACE" + # `install` (not `package`) so the gluten-delta artifact is in the local + # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17` + # overrides any user settings.xml that may pin release=1.8 for Java 17 builds. + $MVN_CMD clean install \ + -P${{ env.GLUTEN_SPARK_PROFILE }} \ + -P${{ env.GLUTEN_SCALA_PROFILE }} \ + -P${{ env.GLUTEN_JAVA_PROFILE }} \ + -Pbackends-velox -Pdelta \ + -DskipTests -Dmaven.compiler.release=17 + - name: Stage bundle jar + run: | + set -euo pipefail + mkdir -p bundle-out + # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar + # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT), + # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match + # `ls` from aborting the step under `set -o pipefail`, so the explicit + # check below runs instead of dying with a generic "cannot access". + jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true) + if [ -z "$jar" ] || [ ! -f "$jar" ]; then + echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2 + ls -la package/target/ || true + exit 1 + fi + cp "$jar" bundle-out/ + ls -lh bundle-out/ + - uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-gluten-bundle-${{github.sha}} + path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar + if-no-files-found: error + + delta-spark-test: + needs: build-gluten-bundle + runs-on: ubuntu-22.04 + container: apache/gluten:centos-9-jdk17 + # EXPERIMENT (4 shards x 4 forks): back to 350 -- per-shard suites now run + # 4-at-a-time, so each shard should finish well under the cap again. + timeout-minutes: 350 + strategy: + fail-fast: false + matrix: + # Length of this list MUST equal env.DELTA_NUM_SHARDS. + shard: [0, 1, 2, 3] + env: + # Mirror Delta's spark_test.yaml env vars used by run-tests.py / + # TestParallelization.scala. + SHARD_ID: ${{ matrix.shard }} + steps: + - uses: actions/checkout@v4 + + - name: Resolve workflow inputs + id: resolve + run: | + set -euo pipefail + delta_ref='${{ github.event.inputs.delta_ref }}' + spark_version='${{ github.event.inputs.spark_version }}' + test_parallelism='${{ github.event.inputs.test_parallelism }}' + update_baseline='${{ github.event.inputs.update_baseline }}' + fail_on_fixed='${{ github.event.inputs.fail_on_fixed }}' + : "${delta_ref:=${DELTA_REF_DEFAULT}}" + : "${spark_version:=${DELTA_SPARK_VERSION_DEFAULT}}" + : "${test_parallelism:=${DELTA_TEST_PARALLELISM_DEFAULT}}" + : "${update_baseline:=${DELTA_UPDATE_BASELINE_DEFAULT}}" + : "${fail_on_fixed:=${DELTA_FAIL_ON_FIXED_DEFAULT}}" + { + echo "delta_ref=${delta_ref}" + echo "spark_version=${spark_version}" + echo "test_parallelism=${test_parallelism}" + echo "update_baseline=${update_baseline}" + echo "fail_on_fixed=${fail_on_fixed}" + } | tee -a "$GITHUB_OUTPUT" + + - name: Download Gluten bundle jar + uses: actions/download-artifact@v4 + with: + name: delta-spark-ut-gluten-bundle-${{github.sha}} + path: gluten-bundle + + - name: Install minimal tools + run: | + set -euo pipefail + # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX + # shell, and curl-minimal (which provides the `curl` command sbt's + # launcher needs). Install the rest of what Delta's build/sbt and the + # tests may need. We deliberately do NOT install the full `curl` + # package -- it conflicts with the pre-installed curl-minimal. + yum install -y java-17-openjdk-devel which findutils gzip python3 + export JAVA_HOME=/usr/lib/jvm/java-17-openjdk + export PATH=$JAVA_HOME/bin:$PATH + java -version + git --version + curl --version | head -n 1 + + - name: Cache sbt / Ivy / Coursier + uses: actions/cache@v4 + with: + path: | + /root/.sbt + /root/.ivy2 + /root/.cache/coursier + # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share + # the same dependency tree, so a single shared cache (with parallel + # save races resolved by GH on a first-write-wins basis) gives the + # best storage / hit-rate tradeoff. + key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }} + restore-keys: | + delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}- + delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}- + + - name: Clone and patch Delta + run: | + set -euo pipefail + # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step + # under `set -o pipefail`, so the explicit check below emits a clear + # error instead of a generic "cannot access". + GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true) + if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then + echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2 + ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true + exit 1 + fi + echo "Using Gluten bundle: $GLUTEN_JAR" + bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \ + "${{ steps.resolve.outputs.delta_ref }}" \ + "$GITHUB_WORKSPACE/delta" \ + "$GLUTEN_JAR" \ + "$GITHUB_WORKSPACE" + + - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }}) + env: + NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }} + TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }} + # Required by Delta to enable testing-only code paths + # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1"). + DELTA_TESTING: '1' + # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the + # `io.netty.tryReflectionSetAccessible` system property; otherwise + # the forked test JVM fails with + # java.lang.UnsupportedOperationException: sun.misc.Unsafe or + # java.nio.DirectByteBuffer.(long, int) not available + # as soon as Gluten's bundled Arrow allocator initializes Netty + # direct buffers. Delta's own `Test / javaOptions` (see + # project/CrossSparkVersions.scala `java17TestSettings`) sets the + # base add-opens but NOT the Netty property -- Delta's own tests + # don't load Arrow/Netty buffers in a way that triggers it. + # + # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt + # launcher JVM and the forked test JVM (sbt forks tests and the + # child inherits the parent's env). The set below mirrors + # `extraJavaTestArgs` from Gluten's own root pom.xml (the + # canonical Gluten test JVM flag set). + # + # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS + # is processed BEFORE the JVM command line, so Delta's explicit + # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still + # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via + # an sbt `set spark / Test / javaOptions ++= ...` command below, + # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER + # `-Xmx1024m` and wins. + JAVA_TOOL_OPTIONS: >- + -XX:+IgnoreUnrecognizedVMOptions + --add-opens=java.base/java.lang=ALL-UNNAMED + --add-opens=java.base/java.lang.invoke=ALL-UNNAMED + --add-opens=java.base/java.lang.reflect=ALL-UNNAMED + --add-opens=java.base/java.io=ALL-UNNAMED + --add-opens=java.base/java.net=ALL-UNNAMED + --add-opens=java.base/java.nio=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.base/java.util.concurrent=ALL-UNNAMED + --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED + --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED + --add-opens=java.base/sun.nio.ch=ALL-UNNAMED + --add-opens=java.base/sun.nio.cs=ALL-UNNAMED + --add-opens=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/sun.util.calendar=ALL-UNNAMED + -Djdk.reflect.useDirectMethodHandle=false + -Dio.netty.tryReflectionSetAccessible=true + -Dfile.encoding=UTF-8 + run: | + set -euo pipefail + export JAVA_HOME=/usr/lib/jvm/java-17-openjdk + export PATH=$JAVA_HOME/bin:$PATH + cd "$GITHUB_WORKSPACE/delta" + chmod +x build/sbt + # Only run the unified `spark` sbt project, NOT `sparkGroup/test` -- + # `sparkGroup` aggregates many other projects (sparkV2, contribs, + # sharing, connect*, ...) that are out of scope for this pipeline. + # + # JVM heap layout (16 GB ubuntu-22.04 runner, TEST_PARALLELISM=1): + # * sbt launcher JVM: -J-Xmx4G, BUT made to RETURN idle memory (see the + # G1 periodic-GC flags below). The per-minute MEM profiler in the + # watchdog (run #18) DISPROVED the old "launcher RSS is well under 4G" + # assumption: the launcher grew to a ROCK-STEADY 5.3G (RSS) during the + # test-compile and then HELD it for the entire run -- ~3.8G of pure + # idle waste during the (long) test phase, where it only relays the + # fork's test events. That fixed 5.3G + the fork's native spike in a + # heavy suite is what pushed the cgroup to the ~16G OOM-kill. G1 does + # NOT uncommit on its own here because the idle launcher never GCs. + # FIX (behaviour-neutral -- touches NO Gluten/Spark runtime config, so + # it cannot pollute the measured pass/fail signal): keep -Xmx4G for the + # compile headroom but force a periodic GC every 10s when idle + # (G1PeriodicGCInterval) with the system-load gate disabled + # (G1PeriodicGCSystemLoadThreshold=0, since the busy fork would + # otherwise suppress it) as a full STW collection + # (-G1PeriodicGCInvokesConcurrent) that uncommits down to a tight free + # ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low -Xms512m + # floor. The idle launcher then drops from ~5.3G back to ~1-2G during + # the test phase, cutting the cgroup peak by ~3.8G (~15.9G -> ~12G) -- + # real headroom under the OOM threshold -- with zero compile-OOM risk. + # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command + # below. Delta v4.2.0 caps its test fork at `-Xmx1024m` in build.sbt; + # Gluten OFFLOADS data to Velox off-heap (capped at 2g via + # spark.memory.offHeap.size in the patched DeltaSQLCommandTest), so the + # fork's JVM HEAP need is modest -- 2G is generous. + # HISTORY: briefly bumped to 8G then 4G to absorb a DV+CDC merge suite's + # giant RoaringBitmapArray heap allocation, but that was a Gluten bug + # (garbage native _metadata.row_index) FIXED UPSTREAM by #12269 -- so the + # large heap is no longer needed. Worse, on the ~16G runner the cgroup + # memory.peak hit 15.97G with a 4G fork heap and the kernel OOM-killed + # the fork mid-shard (sudden death, no hs_err / no heap dump), which + # wedged sbt's main process forever in ScalaTestRunner.done -> Thread + # .join (the chronic "shard 2 hang"). The JVM heaps -- not off-heap -- + # drive that peak, so 2G fork heap (+ the unchanged 4G sbt launcher, + # which needs its heap to COMPILE the tests) brings the peak to ~13G, + # leaving real headroom. The `++=` appends to Delta's own Test/javaOptions + # seq so our `-Xmx2G` comes AFTER `-Xmx1024m` and wins (last `-Xmx` + # wins). Keep heap-dump-on-OOM so a genuine >2G heap OOM is analyzable. + # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so + # every suite writes per-test results. Delta itself only configures + # the console reporter (-oDF), so without this we'd have no machine- + # readable results to gate on. The path is relative to the forked + # test JVM's working dir (Test / baseDirectory = spark/), i.e. + # delta/spark/target/test-reports/TEST-*.xml. + # + # We deliberately do NOT let an sbt non-zero exit (which fires on the + # MANY expected Delta-on-Gluten failures) fail this step directly. + # Instead the known-failures gate below decides pass/fail: the build + # is green when the only failures are ones already recorded in the + # baseline, and red on a genuine regression. + set +e + # --- hang watchdog --------------------------------------------------- + # Shard 2 (and occasionally others) hangs indefinitely after a suite's + # last test with no further output. ScalaTest's failAfter only wraps + # individual test BODIES, so a wedge in suite teardown/afterAll -- or in + # a non-interruptible native Velox/JNI call that ignores + # Thread.interrupt() -- has no timeout and stalls until the 350-min job + # limit with zero diagnostics. This watchdog dumps the forked test JVM's + # threads (to the job log, and to a file for the artifact) once the test + # output has been silent for too long, so the deadlock is diagnosable. + SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log" + : > "$SBT_LOG" + rm -f /tmp/sbt-done + ( + # CRITICAL: the step shell runs with `bash -eo pipefail`, which the + # subshell inherits. Without `set +e` here, ANY non-zero command -- + # e.g. fork detection finding no match, or `kill`/`jps` returning + # non-zero -- silently kills this watchdog. That errexit kill (plus a + # /proc detection miss) is why the watchdog captured ZERO dumps in + # runs #12 and #13. A diagnostic must never abort on a failed probe. + set +e +o pipefail + JSTACK="${JAVA_HOME}/bin/jstack" + JPS="${JAVA_HOME}/bin/jps" + silent_limit=900 # 15 min with no new test output => treat as hung + dumps=0 + fork_pids() { + # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads + # the main class from hsperfdata, robust to sbt's @argfile launch); + # fall back to scanning /proc cmdline + @argfile. + "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}' + local p cl arg + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue + cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)" + case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac + arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)" + [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \ + && echo "${p##*/}" + done + } + all_java_pids() { + "$JPS" -q 2>/dev/null + local p + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}" + done + } + echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence" + hb=0 + while [ ! -f /tmp/sbt-done ]; do + sleep 60 + [ -f "$SBT_LOG" ] || continue + now=$(date +%s) + mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now") + silent=$(( now - mtime )) + # Per-minute memory profile: heap tuning proved the ~16G OOM peak is + # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows + # toward it -- the last lines before a hang reveal the real hog to cut. + # Read /proc directly (no `ps` dependency in the minimal container). + memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null) + jvmrss="" + for mp in $(all_java_pids 2>/dev/null | sort -un); do + r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null) + [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)" + done + echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]" + hb=$(( hb + 1 )) + # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how + # long the test has been silent) without waiting for a hang. + [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago" + if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then + dumps=$(( dumps + 1 )) + pids="$(fork_pids | sort -un)" + # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM. + [ -n "$pids" ] || pids="$(all_java_pids | sort -un)" + echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))" + [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump" + for pid in $pids; do + # SIGQUIT makes the JVM print a full thread dump to its OWN stderr, + # which sbt relays into the test log via the SAME stream as test + # output -- so it lands in the job log even when a separately + # spawned jstack child's output would be buffered/lost. Also write + # jstack to a file for the per-shard artifact. + echo "----- SIGQUIT + jstack pid ${pid} -----" + kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}" + timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \ + || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}" + done + echo "::endgroup::" + # The dump is now captured (job log via SIGQUIT + artifact via + # jstack file). A hung JVM otherwise stalls the whole shard until + # the 350-min job timeout AND keeps the job log frozen so the dump + # never becomes reachable. So KILL the wedged JVM(s): the suite + # fails fast (acceptable -- errors are expected; only an + # unrecoverable hang blocks CI), the job proceeds/ends, and the log + # + artifacts flush. Give SIGQUIT a moment to print first. + sleep 20 + echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)" + for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done + fi + done + ) & + WATCHDOG_PID=$! + + ./build/sbt \ + -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \ + -v \ + -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \ + -J-XX:G1PeriodicGCInterval=10000 \ + -J-XX:G1PeriodicGCSystemLoadThreshold=0 \ + -J-XX:-G1PeriodicGCInvokesConcurrent \ + -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \ + "++ ${DELTA_SCALA_VERSION}" \ + 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \ + 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \ + "spark/test" 2>&1 | tee "$SBT_LOG" + SBT_EXIT=${PIPESTATUS[0]} + touch /tmp/sbt-done + kill "$WATCHDOG_PID" 2>/dev/null || true + set -e + echo "sbt spark/test exited with ${SBT_EXIT}" + + # Memory forensics: a sudden forked-JVM death with no hs_err and no heap + # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM + # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill + # count so we can confirm/measure it (cgroup v2 paths; best-effort). + ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ===" + for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \ + /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do + [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; } + done ) || true + + # A compile/launch failure leaves no reports at all. In that case the + # gate would see zero failures and pass spuriously, so fail loudly. + REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true) + echo "Found ${REPORT_COUNT} JUnit XML report file(s)." + if [ "${REPORT_COUNT}" -eq 0 ]; then + echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures." + exit 1 + fi + + # update_baseline=true -> SEED mode (record failures, never fail) so the + # baseline can be (re)generated. Otherwise ENFORCE against the baseline. + GATE_MODE=enforce + if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then + GATE_MODE=seed + fi + mkdir -p "$GITHUB_WORKSPACE/gate-out" + python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \ + --mode "${GATE_MODE}" \ + --reports-dir "$GITHUB_WORKSPACE/delta" \ + --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \ + --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \ + --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \ + --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}" + + - name: Compress heap dumps (if any) + if: ${{ failure() }} + run: | + set -euo pipefail + if compgen -G "/tmp/*.hprof" > /dev/null; then + echo "Found heap dump(s); compressing..." + ls -lh /tmp/*.hprof + # gzip is single-threaded and slow on multi-GB heaps but is + # always present in the centos image. Heap dumps compress ~10x. + gzip -1 /tmp/*.hprof + ls -lh /tmp/*.hprof.gz + else + echo "No heap dumps found in /tmp/." + fi + + - name: Upload per-shard gate lists + if: always() + uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-gate-lists-shard-${{ matrix.shard }} + path: gate-out/*.txt + if-no-files-found: warn + + - name: Upload test reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-reports-shard-${{ matrix.shard }} + path: | + delta/**/target/test-reports/**/*.xml + delta/**/target/surefire-reports/**/*.xml + if-no-files-found: warn + + - name: Upload hang watchdog thread dumps + if: always() + uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-threaddumps-shard-${{ matrix.shard }} + path: /tmp/threaddump-shard-${{ matrix.shard }}-*.txt + if-no-files-found: ignore + + - name: Upload JVM crash logs and other failure artifacts + if: ${{ failure() }} + uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-failure-logs-shard-${{ matrix.shard }} + path: | + delta/**/target/*.log + delta/**/hs_err_pid*.log + delta/**/core.* + /tmp/*.hprof + /tmp/*.hprof.gz + if-no-files-found: ignore + + # Merges every shard's failure/ran lists into a single, sorted, ready-to-commit + # known-failures.txt and reports global regressions / now-passing / stale + # entries. Runs even when some shards went red (if: always()) so the refreshed + # baseline artifact is always available -- this is what you download and commit + # to bootstrap or refresh the baseline (see util/delta-spark-ut/README.md). + delta-spark-aggregate: + needs: delta-spark-test + if: always() + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Download per-shard gate lists + uses: actions/download-artifact@v4 + continue-on-error: true + with: + pattern: delta-spark-ut-gate-lists-shard-* + path: gate-lists + merge-multiple: true + - name: Aggregate known failures + run: | + set -euo pipefail + python3 .github/workflows/util/delta-spark-ut/compare-test-results.py \ + --mode aggregate \ + --inputs-dir gate-lists \ + --known-failures .github/workflows/util/delta-spark-ut/known-failures.txt \ + --baseline-out aggregated/known-failures.txt + - name: Upload refreshed baseline + if: always() + uses: actions/upload-artifact@v4 + with: + name: delta-spark-ut-known-failures + path: aggregated/known-failures.txt + if-no-files-found: warn diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md new file mode 100644 index 00000000000..ea2cc6af190 --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -0,0 +1,112 @@ + + +# Delta Spark UT (Gluten) — managing expected failures + +Running delta-io/delta's `spark` ScalaTest suite against the Gluten Velox +bundle produces **many expected failures**: Gluten does not yet offload every +Delta code path, and falls back or behaves differently in places. If CI simply +went red on any failure, the signal would be useless and we could never tell a +*new* breakage from the hundreds of already-known ones. + +To make this manageable we keep a **baseline of known failures** and gate each +run against it. The build is green when the only failing tests are ones already +recorded in the baseline; it goes red the moment a **previously-passing test +starts failing** (a regression). + +## Files + +| File | Purpose | +|---|---| +| `known-failures.txt` | Committed baseline: the tests currently expected to fail. One `#` per line. | +| `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and gates / seeds / aggregates against the baseline. Standard-library only. | +| `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches `DeltaSQLCommandTest`. | + +## How the gate works + +Each test shard: + +1. Runs `sbt spark/test` with ScalaTest's JUnit XML reporter enabled + (`-u target/test-reports`), so every suite writes per-test results. (Delta + itself only configures the console reporter, so the workflow injects this.) +2. Runs `compare-test-results.py --mode enforce`, which classifies every test: + - **regression** — failed, but not in the baseline → **fails the shard**. + - **expected** — failed and in the baseline → ignored. + - **now-passing** — in the baseline but passed this run → fails the shard + (so the baseline is kept honest), unless `fail_on_fixed=false`. + +A final `aggregate` job merges every shard's results into a single, sorted, +ready-to-commit `known-failures.txt` artifact and reports **stale** baseline +entries (tests no longer present in any shard, e.g. after a Delta version bump). + +Because Delta shards **by suite**, every suite (and therefore every test) runs +in exactly one shard, so per-shard enforcement sees complete suites and never +double-counts. + +## Bootstrapping the baseline (first time) + +While `known-failures.txt` has no entries the gate auto-runs in **seed mode** +(it never fails — it only records failures). To create the initial baseline: + +1. Trigger **Actions → Delta Spark UT (Gluten) → Run workflow** with + `update_baseline = true`. +2. When it finishes, download the **`delta-spark-ut-known-failures`** artifact. +3. Replace `known-failures.txt` with the file from that artifact and commit it. + +From the next run onward the gate enforces the baseline. + +## Day-to-day: fixing tests incrementally + +- **You fixed Gluten and some Delta tests now pass.** CI will flag them as + *now-passing*. Delete those lines from `known-failures.txt` in your PR. That + is the whole point — the baseline only ever shrinks as coverage improves. +- **You intentionally added a new expected failure** (e.g. a Delta path Gluten + can't offload yet). Add the exact `Suite#test` line(s) the gate prints under + *Regressions* to `known-failures.txt`, ideally with a comment explaining why. +- **A genuine regression.** Fix it; do **not** add it to the baseline. + +The error log prints copy-pasteable `Suite#test` lines for both regressions and +now-passing tests, and each run's job summary shows the full breakdown. + +## Regenerating / refreshing the whole baseline + +After a Delta version bump or a large Gluten change, regenerate from scratch the +same way as bootstrapping: run the workflow with `update_baseline=true`, download +the `delta-spark-ut-known-failures` artifact, and commit it. The aggregate job +also lists **stale** entries you can prune. + +## Caveats + +- **Flaky tests.** A flaky test that usually passes will be flagged as a + regression when it flakes; one that usually fails (and is in the baseline) + may be flagged as now-passing when it happens to pass. Re-run, or set + `fail_on_fixed=false` for that run, and keep genuinely flaky tests out of the + enforced set. +- **Known failures still execute** (and fail) — they are gated *after* the run, + not skipped — so they still consume CI time. This keeps us decoupled from + Delta's sources; skipping them at runtime would require patching Delta. + +## Running the comparison locally + +```bash +# after an sbt spark/test run that wrote delta/**/target/test-reports/*.xml +python3 .github/workflows/util/delta-spark-ut/compare-test-results.py \ + --mode enforce \ + --reports-dir delta \ + --known-failures .github/workflows/util/delta-spark-ut/known-failures.txt \ + --failures-out /tmp/failures.txt --ran-out /tmp/ran.txt +``` diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py new file mode 100644 index 00000000000..bed6d18712e --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Gate / seed / aggregate the Delta-on-Gluten unit test results. + +Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle +produces many *expected* failures (Gluten does not yet support every Delta +code path). To keep the red/green signal meaningful while we fix those +failures incrementally, we maintain a committed baseline of known failing +tests (``known-failures.txt``) and compare each CI run against it. + +This script has three modes: + +``enforce`` (default, per shard) + Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u`` + reporter) and compare against the baseline: + + * regression -- a test that FAILED but is NOT in the baseline. These + fail the build: a previously-passing test just started failing. + * expected -- a test that failed and IS in the baseline. Ignored. + * fixed -- a baseline test that now PASSES. By default these also + fail the build (``--fail-on-fixed true``) so the baseline stays honest + and contributors remove entries as they fix them. + + If the baseline is empty (not yet bootstrapped) the mode automatically + degrades to ``seed`` so the first run is never spuriously red. + +``seed`` (bootstrap / ``update_baseline``) + Never fails. Just writes the current shard's failing tests so the baseline + can be (re)generated from a real run. + +``aggregate`` (final job) + Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single, + sorted, ready-to-commit ``known-failures.txt`` and report stale baseline + entries (tests no longer present in any shard). + +Baseline file format (``known-failures.txt``):: + + # comment lines start with '#' + # + +The suite is always a JVM class name (dot-separated, never starts with '#'), +so a line whose first non-space character is '#' is unambiguously a comment, +and the FIRST '#' after the suite separates suite from the (possibly +'#'-containing) test name. + +Only the Python standard library is used so the script runs in the bare +centos image used by the Delta UT pipeline with no ``pip install``. +""" + +import argparse +import glob +import os +import sys +import xml.etree.ElementTree as ET + +# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll +# throws) so that the JUnit XML reports a suite-level error with no per-test +# . Without this, a suite that used to pass but now aborts entirely +# would record zero failing testcases and the regression would be missed. +SUITE_ABORTED = "" + +SEP = "#" + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +# --------------------------------------------------------------------------- # +# Baseline (known-failures.txt) parsing / formatting +# --------------------------------------------------------------------------- # +def format_entry(suite, test): + return "{}{}{}".format(suite, SEP, test) + + +def parse_entry(line): + """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments.""" + stripped = line.strip() + if not stripped or stripped.startswith("#"): + return None + idx = stripped.find(SEP) + if idx < 0: + # No separator: treat the whole line as a suite-level entry. + return (stripped, SUITE_ABORTED) + return (stripped[:idx], stripped[idx + len(SEP) :]) + + +def load_entries(path): + """Load a set of (suite, test) tuples from a baseline/shard-list file.""" + entries = set() + if not path or not os.path.exists(path): + return entries + with open(path, "r", encoding="utf-8") as fh: + for line in fh: + parsed = parse_entry(line) + if parsed is not None: + entries.add(parsed) + return entries + + +def write_entries(path, entries, header=None): + """Write a sorted set of (suite, test) tuples to a file.""" + os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8") as fh: + if header: + for hl in header.splitlines(): + fh.write(hl.rstrip() + "\n") + for suite, test in sorted(entries): + # Defensive: collapse any stray newlines so each entry stays on one line. + safe_test = test.replace("\r", " ").replace("\n", " ") + fh.write(format_entry(suite, safe_test) + "\n") + + +# --------------------------------------------------------------------------- # +# JUnit XML parsing +# --------------------------------------------------------------------------- # +def _iter_testsuites(root): + """Yield every element regardless of whether the file root is + (wrapper) or a single .""" + tag = root.tag.split("}")[-1] # strip any namespace + if tag == "testsuites": + for child in root: + if child.tag.split("}")[-1] == "testsuite": + yield child + elif tag == "testsuite": + yield root + + +def _child_local_tags(elem): + return {c.tag.split("}")[-1] for c in elem} + + +def parse_reports(reports_dir): + """Walk reports_dir for JUnit XML and classify every test. + + Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is + 'failed' if its has a or child, 'skipped' if + it has a child, otherwise 'passed'. Suite-level aborts (a + reporting errors/failures with no failing ) are + recorded as a synthetic (suite, SUITE_ABORTED) failure. + """ + passed, failed, skipped = set(), set(), set() + + xml_files = [] + # ScalaTest's -u reporter and Maven surefire both write `TEST-.xml` + # under a `target/.../*-reports/` dir. Restrict the secondary glob to + # `target/` so we never parse Delta's own XML *test resources* (which live + # under src/test/resources and are not reports). The -root guard + # below is a final safety net. + for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"): + xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True)) + xml_files = sorted(set(xml_files)) + + parsed_any = False + for xml_file in xml_files: + try: + tree = ET.parse(xml_file) + except ET.ParseError as exc: + eprint("WARNING: could not parse {}: {}".format(xml_file, exc)) + continue + root = tree.getroot() + root_tag = root.tag.split("}")[-1] + if root_tag not in ("testsuites", "testsuite"): + continue # not a JUnit report + + for ts in _iter_testsuites(root): + parsed_any = True + suite_name = ts.get("name") or "" + suite_has_failing_tc = False + for tc in ts: + if tc.tag.split("}")[-1] != "testcase": + continue + suite = tc.get("classname") or suite_name + name = tc.get("name") or "" + key = (suite, name) + tags = _child_local_tags(tc) + if "failure" in tags or "error" in tags: + failed.add(key) + suite_has_failing_tc = True + elif "skipped" in tags: + skipped.add(key) + else: + passed.add(key) + + # Suite-level abort: counters say something failed but no testcase + # carried the failure (the suite blew up in beforeAll/constructor). + # Record a + # synthetic entry so the regression is visible. + try: + errors = int(ts.get("errors", "0") or "0") + failures = int(ts.get("failures", "0") or "0") + except ValueError: + errors = failures = 0 + if (errors + failures) > 0 and not suite_has_failing_tc: + failed.add((suite_name, SUITE_ABORTED)) + + if not parsed_any: + eprint( + "WARNING: no JUnit elements found under {}".format(reports_dir) + ) + + # A test can't be both passed and failed; failure wins. Skipped only counts + # if the test was not otherwise seen (e.g. retried). + passed -= failed + skipped -= failed + skipped -= passed + return passed, failed, skipped + + +# --------------------------------------------------------------------------- # +# Reporting helpers +# --------------------------------------------------------------------------- # +def _summary_sink(): + """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available.""" + path = os.environ.get("GITHUB_STEP_SUMMARY") + handle = open(path, "a", encoding="utf-8") if path else None + + def write(line=""): + print(line) + if handle: + handle.write(line + "\n") + + return write, handle + + +def _print_block(write, title, entries, limit=50): + write("") + write("### {} ({})".format(title, len(entries))) + if not entries: + return + write("") + write("```") + for i, (suite, test) in enumerate(sorted(entries)): + if i >= limit: + write("... and {} more".format(len(entries) - limit)) + break + write(format_entry(suite, test)) + write("```") + + +# --------------------------------------------------------------------------- # +# Modes +# --------------------------------------------------------------------------- # +def run_enforce(args): + baseline = load_entries(args.known_failures) + passed, failed, skipped = parse_reports(args.reports_dir) + + # Always emit this shard's artifacts for the aggregation job. + if args.failures_out: + write_entries(args.failures_out, failed) + if args.ran_out: + write_entries(args.ran_out, passed | failed) + + write, handle = _summary_sink() + try: + seeding = args.mode == "seed" or not baseline + if seeding and args.mode != "seed": + write( + "> NOTE: baseline `{}` is empty -- running in SEED mode " + "(no failures will be enforced). Bootstrap the baseline from " + "the aggregated artifact, commit it, then enforcement begins.".format( + args.known_failures + ) + ) + + write( + "## Delta-on-Gluten test gate -- shard {}".format( + os.environ.get("SHARD_ID", "?") + ) + ) + write("") + write("| Category | Count |") + write("|---|---:|") + write("| Ran (pass+fail) | {} |".format(len(passed) + len(failed))) + write("| Passed | {} |".format(len(passed))) + write("| Failed | {} |".format(len(failed))) + write("| Skipped | {} |".format(len(skipped))) + write("| Baseline (known failures) | {} |".format(len(baseline))) + + if seeding: + write("") + write( + "Seed mode: recorded {} failing test(s) for this shard. " + "Nothing enforced.".format(len(failed)) + ) + return 0 + + regressions = failed - baseline + fixed = baseline & passed + expected = failed & baseline + + write("") + write("| Gate result | Count |") + write("|---|---:|") + write("| Expected failures (in baseline) | {} |".format(len(expected))) + write("| **Regressions (new failures)** | {} |".format(len(regressions))) + write("| Now-passing (remove from baseline) | {} |".format(len(fixed))) + + _print_block( + write, "Regressions -- new failures NOT in the baseline", regressions + ) + if regressions: + write("") + write( + "These tests were not previously known to fail. Either fix " + "the regression, or (if it is a genuinely new expected " + "failure) add the lines above to `known-failures.txt`." + ) + + if args.fail_on_fixed: + _print_block( + write, "Now-passing -- delete these lines from the baseline", fixed + ) + + exit_code = 0 + if regressions: + for suite, test in sorted(regressions): + eprint("::error::REGRESSION {}".format(format_entry(suite, test))) + exit_code = 1 + if args.fail_on_fixed and fixed: + for suite, test in sorted(fixed): + eprint( + "::error::NOW-PASSING (remove from baseline) {}".format( + format_entry(suite, test) + ) + ) + exit_code = 1 + + if exit_code == 0: + write("") + write("All failures are expected (in the baseline). Gate passed.") + return exit_code + finally: + if handle: + handle.close() + + +def run_aggregate(args): + failure_files = sorted( + glob.glob(os.path.join(args.inputs_dir, "**", "failures-*.txt"), recursive=True) + ) + ran_files = sorted( + glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True) + ) + + union_failed = set() + for f in failure_files: + union_failed |= load_entries(f) + union_ran = set() + for f in ran_files: + union_ran |= load_entries(f) + + header = ( + "# Known Delta-on-Gluten unit test failures.\n" + "#\n" + "# Auto-generated by compare-test-results.py --mode aggregate.\n" + "# Format: #\n" + "# Lines starting with '#' are comments.\n" + "#\n" + "# Regenerate by running the 'Delta Spark UT (Gluten)' workflow with\n" + "# update_baseline=true and committing the produced artifact.\n" + ) + if args.baseline_out: + write_entries(args.baseline_out, union_failed, header=header) + + write, handle = _summary_sink() + try: + write("## Delta-on-Gluten aggregated results") + write("") + write("| Metric | Count |") + write("|---|---:|") + write("| Shards with failure lists | {} |".format(len(failure_files))) + write("| Distinct failing tests | {} |".format(len(union_failed))) + write("| Distinct tests run | {} |".format(len(union_ran))) + + exit_code = 0 + if args.known_failures and os.path.exists(args.known_failures): + baseline = load_entries(args.known_failures) + if baseline: + regressions = union_failed - baseline + fixed = baseline & (union_ran - union_failed) + stale = baseline - union_ran + write("| Baseline entries | {} |".format(len(baseline))) + write("| Regressions (global) | {} |".format(len(regressions))) + write("| Now-passing (global) | {} |".format(len(fixed))) + write("| Stale (not seen this run) | {} |".format(len(stale))) + _print_block(write, "Regressions (global)", regressions) + _print_block(write, "Now-passing (global)", fixed) + _print_block(write, "Stale baseline entries (suite/test gone)", stale) + if args.fail_on_regression and regressions: + exit_code = 1 + return exit_code + finally: + if handle: + handle.close() + + +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # +def str2bool(value): + return str(value).strip().lower() in ("1", "true", "yes", "y", "on") + + +def main(argv=None): + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument( + "--mode", choices=("enforce", "seed", "aggregate"), default="enforce" + ) + parser.add_argument( + "--known-failures", help="Path to the committed known-failures.txt baseline." + ) + parser.add_argument( + "--reports-dir", help="Root dir to search for JUnit XML (enforce/seed)." + ) + parser.add_argument( + "--failures-out", help="Write this shard's failing tests here (enforce/seed)." + ) + parser.add_argument( + "--ran-out", help="Write this shard's run tests (pass+fail) here." + ) + parser.add_argument( + "--fail-on-fixed", + type=str2bool, + default=True, + help="Fail when a baseline test now passes (default true).", + ) + parser.add_argument( + "--inputs-dir", help="Dir with per-shard failures-*/ran-* files (aggregate)." + ) + parser.add_argument( + "--baseline-out", help="Write the merged baseline here (aggregate)." + ) + parser.add_argument( + "--fail-on-regression", + type=str2bool, + default=False, + help="In aggregate mode, fail if global regressions exist.", + ) + args = parser.parse_args(argv) + + if args.mode in ("enforce", "seed"): + if not args.reports_dir: + parser.error("--reports-dir is required for --mode {}".format(args.mode)) + return run_enforce(args) + return run_aggregate(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt new file mode 100644 index 00000000000..ae7b300084b --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -0,0 +1,977 @@ +# Known Delta-on-Gluten unit test failures. +# +# Baseline of delta-io/delta `spark` ScalaTest tests EXPECTED to fail under the +# Gluten Velox bundle. The Delta Spark UT (Gluten) workflow enforces this list: +# a failing test NOT listed here is a regression (fails CI); a listed test that +# now passes should be removed. Format: #. +# Lines starting with '#' are comments. See README.md in this directory. +# +# --------------------------------------------------------------------------- +# Full 16-shard baseline. Originally seeded from 15 of 16 shards (run +# 27490052632). Shard 2 used to hang/OOM-crash on DeletionVectorsSuite's 2B-row +# DV tests; those two tests are now force-failed in setup-delta.sh, so shard 2 +# runs to completion and contributes 69 failures. 963 known failures total. +# --------------------------------------------------------------------------- +io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using SQL conf +io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using withExtensions +io.delta.sql.JavaDeltaSparkSessionExtensionSuite#testSQLConf +io.delta.tables.DeltaTableHadoopOptionsSuite#delete - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#details - with filesystem options. +io.delta.tables.DeltaTableHadoopOptionsSuite#forPath - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#forPath error out without filesystem options passed in. +io.delta.tables.DeltaTableHadoopOptionsSuite#forPath with unsupported options +io.delta.tables.DeltaTableHadoopOptionsSuite#forPath: as/alias/toDF with filesystem options. +io.delta.tables.DeltaTableHadoopOptionsSuite#generate - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#history - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#merge - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#optimize - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#restoreTable - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#update - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#updateExpr - with filesystem options +io.delta.tables.DeltaTableHadoopOptionsSuite#vacuum - with filesystem options +org.apache.spark.sql.delta.AutoCompactExecutionIdColumnMappingSuite#auto-compact-enabled-conf: auto compact should kick in when enabled - session config - column mapping id mode +org.apache.spark.sql.delta.AutoCompactExecutionIdColumnMappingSuite#auto-compact-enabled-property: auto compact should kick in when enabled - table config - column mapping id mode +org.apache.spark.sql.delta.AutoCompactExecutionIdColumnMappingSuite#auto-compact-enabled-property: auto compact should not kick in when session config is off - column mapping id mode +org.apache.spark.sql.delta.AutoCompactExecutionIdColumnMappingSuite#variant auto compact kicks in when enabled - session config - column mapping id mode +org.apache.spark.sql.delta.AutoCompactExecutionIdColumnMappingSuite#variant auto compact kicks in when enabled - table config - column mapping id mode +org.apache.spark.sql.delta.AutoCompactExecutionNameColumnMappingSuite#auto-compact-enabled-conf: auto compact should kick in when enabled - session config - column mapping name mode +org.apache.spark.sql.delta.AutoCompactExecutionNameColumnMappingSuite#auto-compact-enabled-property: auto compact should kick in when enabled - table config - column mapping name mode +org.apache.spark.sql.delta.AutoCompactExecutionNameColumnMappingSuite#auto-compact-enabled-property: auto compact should not kick in when session config is off - column mapping name mode +org.apache.spark.sql.delta.AutoCompactExecutionNameColumnMappingSuite#variant auto compact kicks in when enabled - session config - column mapping name mode +org.apache.spark.sql.delta.AutoCompactExecutionNameColumnMappingSuite#variant auto compact kicks in when enabled - table config - column mapping name mode +org.apache.spark.sql.delta.AutoCompactExecutionSuite#auto-compact-enabled-conf: auto compact should kick in when enabled - session config +org.apache.spark.sql.delta.AutoCompactExecutionSuite#auto-compact-enabled-property: auto compact should kick in when enabled - table config +org.apache.spark.sql.delta.AutoCompactExecutionSuite#auto-compact-enabled-property: auto compact should not kick in when session config is off +org.apache.spark.sql.delta.AutoCompactExecutionSuite#variant auto compact kicks in when enabled - session config +org.apache.spark.sql.delta.AutoCompactExecutionSuite#variant auto compact kicks in when enabled - table config +org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled +org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsSuite#SC-86940: writing a GCS checkpoint should happen in a new thread +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#SC-86940: writing a GCS checkpoint should happen in a new thread +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#SC-86940: writing a GCS checkpoint should happen in a new thread +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled +org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#SC-86940: writing a GCS checkpoint should happen in a new thread +org.apache.spark.sql.delta.CloneTableSQLSuite#shallow clone across file systems +org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch100Suite#shallow clone across file systems +org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch1Suite#shallow clone across file systems +org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch2Suite#shallow clone across file systems +org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#Cloning table with persistent DVs and absolute parquet paths +org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#Shallow clone round-trip with DVs +org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#shallow clone across file systems +org.apache.spark.sql.delta.CloneTableScalaSuite#shallow clone across file systems +org.apache.spark.sql.delta.ConvertToDeltaSQLSuite#external tables use correct path scheme +org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#external tables use correct path scheme +org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row per file - Partitioned = false, cdfEnabled = false +org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row per file - Partitioned = false, cdfEnabled = true +org.apache.spark.sql.delta.DeltaAllFilesInCrcSuite#test all-files-in-crc verification failure also triggers and logs incremental-commit verification result +org.apache.spark.sql.delta.DeltaAlterTableByNameIdColumnMappingSuite#CHANGE COLUMN - case insensitive - column mapping id mode +org.apache.spark.sql.delta.DeltaAlterTableByNameIdColumnMappingSuite#CHANGE COLUMN - move to first (nested) - column mapping id mode +org.apache.spark.sql.delta.DeltaAlterTableByNameNameColumnMappingSuite#CHANGE COLUMN - case insensitive - column mapping name mode +org.apache.spark.sql.delta.DeltaAlterTableByNameNameColumnMappingSuite#CHANGE COLUMN - move to first (nested) - column mapping name mode +org.apache.spark.sql.delta.DeltaArbitraryColumnNameSuite#create table +org.apache.spark.sql.delta.DeltaCDCStreamDeletionVectorSuite#cdc streams with noop merge +org.apache.spark.sql.delta.DeltaCDCStreamSuite#cdc streams with noop merge +org.apache.spark.sql.delta.DeltaCDCStreamWithCatalogManagedBatch100Suite#cdc streams with noop merge +org.apache.spark.sql.delta.DeltaCDCStreamWithCatalogManagedBatch1Suite#cdc streams with noop merge +org.apache.spark.sql.delta.DeltaCDCStreamWithCatalogManagedBatch2Suite#cdc streams with noop merge +org.apache.spark.sql.delta.DeltaColumnMappingSuite#add nested column in schema on new protocol +org.apache.spark.sql.delta.DeltaColumnMappingSuite#alter column order in schema on new protocol +org.apache.spark.sql.delta.DeltaColumnMappingSuite#explicit id matching +org.apache.spark.sql.delta.DeltaColumnMappingSuite#id and name mode should write field_id in parquet schema +org.apache.spark.sql.delta.DeltaColumnMappingSuite#try modifying restricted max id property should fail +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#SC-86916: Delta log cache should respect options +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#SC-86916: checkpoint should pick up Hadoop file system options +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#SC-86916: invalidateCache should invalidate all DeltaLogs of the given path +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#SC-86916: read/write Delta paths using DataFrame should pick up Hadoop file system options +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#all operations should propagate Hadoop file system options +org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#operations without Hadoop options should fail for fake:// filesystem +org.apache.spark.sql.delta.DeltaFastDropFeatureSuite#Vacuum does not delete deletion vector files.generateDVTombstones: false +org.apache.spark.sql.delta.DeltaFastDropFeatureSuite#We do not create redundant DV tombstones after cloning isShallowClone: true +org.apache.spark.sql.delta.DeltaGenerateSymlinkManifestSuite#incremental manifest: failure to generate manifest throws exception +org.apache.spark.sql.delta.DeltaGenerateSymlinkManifestSuite#special partition column values +org.apache.spark.sql.delta.DeltaHistoryManagerSuite#data skipping still works with time travel +org.apache.spark.sql.delta.DeltaHistoryManagerWithCatalogOwnedBatch100Suite#data skipping still works with time travel +org.apache.spark.sql.delta.DeltaHistoryManagerWithCatalogOwnedBatch1Suite#data skipping still works with time travel +org.apache.spark.sql.delta.DeltaHistoryManagerWithCatalogOwnedBatch2Suite#data skipping still works with time travel +org.apache.spark.sql.delta.DeltaInsertIntoDataFrameByPathSuite#insertInto: timestamp partition values with different precisions +org.apache.spark.sql.delta.DeltaInsertIntoDataFrameSuite#insertInto: timestamp partition values with different precisions +org.apache.spark.sql.delta.DeltaInsertIntoSQLByPathSuite#insertInto: timestamp partition values with different precisions +org.apache.spark.sql.delta.DeltaInsertIntoSQLSuite#insertInto: timestamp partition values with different precisions +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#Works with union +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#limit larger than total +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#limit push-down flag +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#no filter or projection +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with non-partition filter +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with partition filter only +org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with projection only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#Works with union +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#limit larger than total +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#limit push-down flag +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#no filter or projection +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with non-partition filter +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with partition filter only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with projection only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#Works with union +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#limit larger than total +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#limit push-down flag +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#no filter or projection +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with non-partition filter +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with partition filter only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with projection only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#Works with union +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#limit larger than total +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#limit push-down flag +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#no filter or projection +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with non-partition filter +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with partition filter only +org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with projection only +org.apache.spark.sql.delta.DeltaLiteVacuumSuite#vacuum for cdc - delete tombstones +org.apache.spark.sql.delta.DeltaLiteVacuumSuite#vacuum for cdc - update/merge +org.apache.spark.sql.delta.DeltaNameColumnMappingSuite#query with predicates should skip partitions - column mapping name mode +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=true, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=true, read DV metadata columns: with isRowDeletedCol=true, with rowIndexCol=true, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_CONTAINED, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_CONTAINED, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_CONTAINED, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_NOT_CONTAINED, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_NOT_CONTAINED, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false +org.apache.spark.sql.delta.DeltaParquetFileFormatWithPredicatePushdownSuite#read DV metadata columns: with rowIndexFilterType=IF_NOT_CONTAINED, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true +org.apache.spark.sql.delta.DeltaSinkIdColumnMappingSuite#partitioned writing and batch reading - column mapping id mode +org.apache.spark.sql.delta.DeltaSinkNameColumnMappingSuite#partitioned writing and batch reading - column mapping name mode +org.apache.spark.sql.delta.DeltaSuite#SC-8810: skip deleted file +org.apache.spark.sql.delta.DeltaSuite#SC-8810: skipping deleted file still throws on corrupted file +org.apache.spark.sql.delta.DeltaSuite#all operations with special characters in path +org.apache.spark.sql.delta.DeltaSuite#deleted files cause failure by default +org.apache.spark.sql.delta.DeltaSuite#invalid replaceWhere +org.apache.spark.sql.delta.DeltaSuite#query with predicates should skip partitions +org.apache.spark.sql.delta.DeltaSuite#replaceArbitrary should enforce proper usage of backtick +org.apache.spark.sql.delta.DeltaTableCreationSuite#Default column values: CONVERT TO DELTA keeps EXISTS_DEFAULT +org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#convert to delta with partitioning change +org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#partitioned convert to delta with schema change +org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - delete tombstones +org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - update/merge +org.apache.spark.sql.delta.DeltaVariantShreddingSuite#Infer schema for Delta table +org.apache.spark.sql.delta.DeltaVariantSuite#DISABLE_VARIANT_TABLE_FEATURE_FOR_SPARK_40 - config disabled does not block +org.apache.spark.sql.delta.DeltaVariantSuite#DISABLE_VARIANT_TABLE_FEATURE_FOR_SPARK_40 - no-op on Spark 4.1+ +org.apache.spark.sql.delta.DeltaVariantSuite#Existing table with variant type can enable CDF +org.apache.spark.sql.delta.DeltaVariantSuite#Table with variant type can use CDF +org.apache.spark.sql.delta.DeltaVariantSuite#Variant can be used as a source for generated columns +org.apache.spark.sql.delta.DeltaVariantSuite#Variant can have default value set +org.apache.spark.sql.delta.DeltaVariantSuite#Variant cannot be created as a generated column +org.apache.spark.sql.delta.DeltaVariantSuite#Variant respects Delta table CHECK constraints +org.apache.spark.sql.delta.DeltaVariantSuite#Variant respects Delta table IS NOT NULL constraints +org.apache.spark.sql.delta.DeltaVariantSuite#Zorder is not supported for Variant +org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - id - false +org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - id - true +org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - name - false +org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - name - true +org.apache.spark.sql.delta.DeltaVariantSuite#optimize variant +org.apache.spark.sql.delta.DeltaVariantSuite#shallow cloning table with variant +org.apache.spark.sql.delta.DeltaVariantSuite#streaming variant delta table +org.apache.spark.sql.delta.DeltaVariantSuite#time travel with variant column works +org.apache.spark.sql.delta.DeltaVariantSuite#variant works with schema evolution for INSERT +org.apache.spark.sql.delta.DeltaVariantSuite#variant works with schema evolution for MERGE +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skip deleted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skipping deleted file still throws on corrupted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#deleted files cause failure by default +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#invalid replaceWhere +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#query with predicates should skip partitions +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#replaceArbitrary should enforce proper usage of backtick +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#SC-8810: skip deleted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#SC-8810: skipping deleted file still throws on corrupted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#deleted files cause failure by default +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#invalid replaceWhere +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#query with predicates should skip partitions +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#replaceArbitrary should enforce proper usage of backtick +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#SC-8810: skip deleted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#SC-8810: skipping deleted file still throws on corrupted file +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#deleted files cause failure by default +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#invalid replaceWhere +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#query with predicates should skip partitions +org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#replaceArbitrary should enforce proper usage of backtick +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only delete all rows - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only delete all rows - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only delete all rows - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only delete all rows - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with duplicates - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with duplicates - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with duplicates - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with duplicates - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with skipping - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with skipping - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with skipping - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only with skipping - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join with source with 1 row - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join with source with 1 row - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join with source with 1 row - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only without join with source with 1 row - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with skipping - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with skipping - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with skipping - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with skipping - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: not matched by source update only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: not matched by source update only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: not matched by source update only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: not matched by source update only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: replace target with source - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: replace target with source - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: replace target with source - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: replace target with source - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert and delete with conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert and delete with conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert and delete with conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: upsert and delete with conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistorySuite#operation metrics - merge +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only delete all rows - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only delete all rows - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only delete all rows - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only delete all rows - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with condition on delete and insert with no matching rows - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with duplicates - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with duplicates - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with duplicates - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with duplicates - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with skipping - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with skipping - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with skipping - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only with skipping - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join with source with 1 row - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join with source with 1 row - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join with source with 1 row - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: delete-only without join with source with 1 row - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: insert-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with skipping - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with skipping - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with skipping - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with skipping - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: match-only with update/delete with unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: not matched by source update only - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: not matched by source update only - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: not matched by source update only - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: not matched by source update only - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: replace target with source - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: replace target with source - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: replace target with source - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: replace target with source - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: update/delete/insert with some unsatisfied conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert and delete with conditions - Partitioned = false, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert and delete with conditions - Partitioned = false, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert and delete with conditions - Partitioned = true, CDF = false +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#merge-metrics: upsert and delete with conditions - Partitioned = true, CDF = true +org.apache.spark.sql.delta.DescribeDeltaHistoryWithCatalogOwnedBatch100Suite#operation metrics - merge +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Disable Deletion Vectors feature - withUnset: false +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Disable Deletion Vectors feature - withUnset: true +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Disable row tracking feature - withUnset: false +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Disable row tracking feature - withUnset: true +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Enable column mapping feature - txnInterleaved: true +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Enable deletion vectors feature +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Enable row tracking feature concurrent txn: delete +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Removing column mapping mode produces conflict - startMode: id +org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite#Removing column mapping mode produces conflict - startMode: name +org.apache.spark.sql.delta.FileSizeHistogramSuite#check CommitStats with deletes +org.apache.spark.sql.delta.FileSizeHistogramSuite#histogram is re-calculated when files are removed +org.apache.spark.sql.delta.GeneratedColumnSuite#update_generated_column_with_incorrect_value +org.apache.spark.sql.delta.GeneratedColumnSuite#update_source_and_generated_columns_with_incorrect_value +org.apache.spark.sql.delta.HDFSLogStoreSuite#No AbstractFileSystem - end to end test using data frame +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#Convert a partitioned parquet table with partition schema autofill +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#can convert a partition-like table path +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#can convert table with partition overwrite +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#catalog partition values contain special characters +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a Hive based external parquet table +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a Hive based parquet table +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a delta table where metadata does not reflect that the table is already converted should update the metadata +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a parquet path to delta while database called parquet exists +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a parquet table to delta with database name as parquet +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a parquet table using table name +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a parquet table with catalog schema - false +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert a parquet table with catalog schema - true +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert an external parquet table +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert partitioned parquet table with catalog partitions - false +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert partitioned parquet table with catalog partitions - true +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert to delta using table name without database name +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert two external tables pointing to same underlying files with differing table properties should error if conf enabled otherwise merge properties +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert with statistics +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#convert without statistics +org.apache.spark.sql.delta.HiveConvertToDeltaSuite#negative case: convert parquet path to delta when there is a database called parquet but no table or path exists +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: INT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: INT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: SMALLINT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: SMALLINT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: SMALLINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: SMALLINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: TINYINT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: TINYINT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: TINYINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: INT, targetType: TINYINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: MAP, targetType: MAP followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: MAP, targetType: MAP followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: MAP, targetType: MAP followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: MAP, targetType: MAP followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: STRING, targetType: INT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: STRING, targetType: INT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: STRING, targetType: INT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: STRING, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: STRING, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: Struct, targetType: Struct followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: Struct, targetType: Struct followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: Struct, targetType: Struct followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: Struct, targetType: Struct followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: DECIMAL(3,1), targetType: DECIMAL(3,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: DOUBLE, targetType: BIGINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: INT, targetType: SMALLINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: INT, targetType: SMALLINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: INT, targetType: TINYINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: INT, targetType: TINYINT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: MAP, targetType: MAP followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: MAP, targetType: MAP followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: STRING, targetType: INT followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: STRING, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: STRING, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: Struct, targetType: Struct followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: Struct, targetType: Struct followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.PublicHDFSLogStoreSuite#No AbstractFileSystem - end to end test using data frame +org.apache.spark.sql.delta.RestoreTableSQLSuite#cdf + RESTORE +org.apache.spark.sql.delta.RestoreTableSQLSuite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableSQLSuite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch100Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch100Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch1Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch1Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch2Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableSQLWithCatalogOwnedBatch2Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableScalaDeletionVectorSuite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableScalaDeletionVectorSuite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableScalaSuite#cdf + RESTORE +org.apache.spark.sql.delta.RestoreTableScalaSuite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableScalaSuite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch100Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch100Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch1Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch1Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch2Suite#restore command output metrics +org.apache.spark.sql.delta.RestoreTableScalaWithCatalogOwnedBatch2Suite#restore operation metrics in Delta table history +org.apache.spark.sql.delta.SnapshotManagementSuite#recover from a corrupt checkpoint: previous checkpoint doesn't exist +org.apache.spark.sql.delta.SnapshotManagementSuite#should not recover when both the current and previous checkpoints are broken +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch100Suite#recover from a corrupt checkpoint: previous checkpoint doesn't exist +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch100Suite#should not recover when both the current and previous checkpoints are broken +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch1Suite#recover from a corrupt checkpoint: previous checkpoint doesn't exist +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch1Suite#should not recover when both the current and previous checkpoints are broken +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch2Suite#recover from a corrupt checkpoint: previous checkpoint doesn't exist +org.apache.spark.sql.delta.SnapshotManagementWithCoordinatedCommitsBatch2Suite#should not recover when both the current and previous checkpoints are broken +org.apache.spark.sql.delta.UpdateMetricsSuite#update-metrics: update one row per file - Partitioned = false, cdfEnabled = false +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#DELETE - Scenario 7 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#OPTIMIZE - Scenario 7 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsDVSuite#UPDATE - Scenario 7 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#DELETE - Scenario 7 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#OPTIMIZE - Scenario 7 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 1 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 2 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 3 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 4 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 5 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 6 +org.apache.spark.sql.delta.commands.backfill.RowTrackingBackfillConflictsSuite#UPDATE - Scenario 7 +org.apache.spark.sql.delta.concurrency.TransactionExecutionObserverSuite#Phase Locking - delete command +org.apache.spark.sql.delta.coordinatedcommits.CoordinatedCommitsSuite#Incomplete backfills are handled properly by next commit after CC to FS conversion +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#DELETE with DVs with column mapping mode=id +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#huge table: delete a small number of rows from tables of 2B rows with DVs +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#huge table: read from tables of 2B rows with existing DV of many zeros +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#variant types DELETE with DVs with column mapping mode=id +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#variant types DELETE with DVs with column mapping mode=name +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite#(It is not a test it is a sbt.testing.SuiteSelector) +org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite# +org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLNameBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedCDCOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedDVPredPushOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedDVPredPushOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteBaseScalaSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLNameBasedSuite#test delete on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLNameBasedSuite#test delete on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedCDCOnSuite#test delete on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedCDCOnSuite#test delete on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedDVPredPushOffSuite#test delete on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedDVPredPushOffSuite#test delete on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedDVPredPushOnSuite#test delete on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedDVPredPushOnSuite#test delete on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedSuite#test delete on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedSuite#test delete on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.MergeCDCSQLPathBasedCDCOnSuite#merge CDC - all conditions failed for all rows +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOffSuite#Merge with DVs metrics - Incremental Updates +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOffSuite#Merge with DVs metrics - delete entire file +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOffSuite#Verify error is produced when paths are not joined correctly +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOnSuite#Merge with DVs metrics - Incremental Updates +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOnSuite#Merge with DVs metrics - delete entire file +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedCDCOnDVsPredPushOnSuite#Verify error is produced when paths are not joined correctly +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOffSuite#Merge with DVs metrics - Incremental Updates +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOffSuite#Merge with DVs metrics - delete entire file +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOffSuite#Verify error is produced when paths are not joined correctly +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOnSuite#Merge with DVs metrics - Incremental Updates +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOnSuite#Merge with DVs metrics - delete entire file +org.apache.spark.sql.delta.generatedsuites.MergeIntoDVsSQLPathBasedDVsPredPushOnSuite#Verify error is produced when paths are not joined correctly +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, non-null target leaves, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, non-null target leaves, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null source nested map, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null source nested map, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target col, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target col, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target leaves, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target leaves, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target nested struct, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedMapStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested map-of-struct - non-null source leaves, null target nested struct, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLNameBasedSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLNameBasedSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedCDCOnSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertSQLPathBasedSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertScalaSuite#schema evolution - struct in different order +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionInsertScalaSuite#schema evolution - struct in different order - with evolution disabled +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, non-null target leaves, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, non-null target leaves, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target col, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target col, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target leaves, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target leaves, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target nested struct, UPDATE * +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionNullnessSQLNameBasedSuite#schema evolution - nested struct - non-null source leaves, null target nested struct, UPDATE t.col = s.col +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLNameBasedSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLNameBasedSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLNameBasedSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedCDCOnSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlySQLPathBasedSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlyScalaSuite#schema evolution - extra nested column in source - update, isPartitioned=false +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlyScalaSuite#schema evolution - extra nested column in source - update, isPartitioned=true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructEvolutionUpdateOnlyScalaSuite#schema evolution - extra nested column in source - update, partition on unused column +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLNameBasedSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLNameBasedSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedCDCOnSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedDVsPredPushOffSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedDVsPredPushOnSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionSQLPathBasedSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionScalaSuite#schema evolution - new source column in map struct key +org.apache.spark.sql.delta.generatedsuites.MergeIntoNestedStructInMapEvolutionScalaSuite#schema evolution - source nested map struct key contains less columns than target +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLNameBasedSuite#not matched by source - all 3 clauses - no changes - isPartitioned: false - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLNameBasedSuite#not matched by source - all 3 clauses - no changes - isPartitioned: true - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLPathBasedCDCOnSuite#not matched by source - all 3 clauses - no changes - isPartitioned: false - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLPathBasedCDCOnSuite#not matched by source - all 3 clauses - no changes - isPartitioned: true - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLPathBasedSuite#not matched by source - all 3 clauses - no changes - isPartitioned: false - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2SQLPathBasedSuite#not matched by source - all 3 clauses - no changes - isPartitioned: true - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2ScalaSuite#not matched by source - all 3 clauses - no changes - isPartitioned: false - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoNotMatchedBySourceCDCPart2ScalaSuite#not matched by source - all 3 clauses - no changes - isPartitioned: true - cdcEnabled: true +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLNameBasedSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedCDCOnDVsPredPushOffSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedCDCOnDVsPredPushOnSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedCDCOnSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedDVsPredPushOffSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedDVsPredPushOnSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSQLSQLPathBasedSuite#CTE as a source in MERGE +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLNameBasedSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedCDCOnDVsPredPushOffSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedCDCOnDVsPredPushOnSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedCDCOnSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnScalaSuite#schema evolution - extra nested column in source - update - single target partition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - partition on multiple columns +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - partition on multiple columns +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - partition on multiple columns +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - partition on multiple columns +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#UDT Data Types - simple and nested +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#data skipping - target-only condition +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#data skipping with matched predicates - with insert clause +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#insert only merge - target data skipping +org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#merge with repartition - insert only merge +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLNameBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnDVSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnRowTrackingOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnRowTrackingOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedDVPredPushOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedDVPredPushOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedRowTrackingOffSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedRowTrackingOnSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscScalaSuite#Variant type +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLNameBasedSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLNameBasedSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnDVSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnDVSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnRowTrackingOffSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnRowTrackingOffSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedDVPredPushOffSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedDVPredPushOffSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedDVPredPushOnSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedDVPredPushOnSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedRowTrackingOffSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedRowTrackingOffSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedSuite#test update on temp view - nontrivial projection - Dataset TempView +org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedSuite#test update on temp view - nontrivial projection - SQL TempView +org.apache.spark.sql.delta.optimize.OptimizeCompactionSQLSuite#optimize - multiple jobs start executing at once +org.apache.spark.sql.delta.optimize.OptimizeCompactionScalaSuite#optimize - multiple jobs start executing at once +org.apache.spark.sql.delta.optimize.OptimizeConflictSuite#conflict handling between Optimize and Business Txn +org.apache.spark.sql.delta.optimize.OptimizeMetricsSuite#optimize ZOrderBy operation metrics in Delta table history +org.apache.spark.sql.delta.optimize.OptimizeMetricsSuite#optimize metrics on idempotent operations +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(day,yyyy-MM-dd) from timestamp +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(day,yyyy-MM-dd) from timestamp nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(hour,yyyy-MM-dd-HH) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(hour,yyyy-MM-dd-HH) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from cast(date) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from cast(date) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from timestamp +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from timestamp nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(date) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(date) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(timestamp) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(timestamp) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part1) escaped field names +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(my.substr,1,3) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(my.substr,1,3) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,0,3) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,0,3) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) deeply nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,2,3) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,2,3) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(DD,eventTimeTrunc) from date_trunc(cast(date)) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(DD,eventTimeTrunc) from date_trunc(cast(date)) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(YEAR,eventTimeTrunc) from date_trunc(timestamp) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(YEAR,eventTimeTrunc) from date_trunc(timestamp) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,month) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,month) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,quarter) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,quarter) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,year) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,year) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayHourPartitionExpr(year,month,day,hour) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayHourPartitionExpr(year,month,day,hour) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayPartitionExpr(year,month,day) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayPartitionExpr(year,month,day) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthPartitionExpr(year,month) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthPartitionExpr(year,month) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(cast(date)) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(cast(date)) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(date) +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(date) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) nested +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#end-to-end optimizable partition expression +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#end-to-end test of behaviors of write/read null on partition column +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#five digits year in a date_format yyyy-MM partition column +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#five digits year in a date_format yyyy-MM-dd-HH partition column +org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#substring on multibyte characters +org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#Re-added files keep their row IDs after conflict with txn not updating high watermark +org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#concurrent transactions do not assign overlapping row IDs +org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#re-added files keep their row ids +org.apache.spark.sql.delta.rowid.RowIdSuite#Filter by base Row IDs +org.apache.spark.sql.delta.rowid.RowIdSuite#Filter by base Row IDs in subquery +org.apache.spark.sql.delta.rowid.RowIdSuite#No dictionary filtering on _metadata.row_id +org.apache.spark.sql.delta.rowid.RowIdSuite#No row-group skipping on _metadata.row_id +org.apache.spark.sql.delta.rowid.RowIdSuite#missing base row ids and default row commit versions +org.apache.spark.sql.delta.rowid.RowIdSuite#row ids can be read back +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencySuite#Interleaved delete right after protocol downgrade should abort due to protocol change +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencySuite#Interleaved update right after protocol downgrade should abort due to protocol change +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencySuite#Single Unbackfill batch interleaves delete +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencySuite#Single Unbackfill batch interleaves update +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencyWithoutDVsSuite#Interleaved delete right after protocol downgrade should abort due to protocol change +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencyWithoutDVsSuite#Interleaved update right after protocol downgrade should abort due to protocol change +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencyWithoutDVsSuite#Single Unbackfill batch interleaves delete +org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencyWithoutDVsSuite#Single Unbackfill batch interleaves update +org.apache.spark.sql.delta.rowtracking.RowTrackingReadWriteSuite#write and read table with all-null materialized columns +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping stats before and after optimize +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping stats before and after optimize - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#Test file pruning metrics with data skipping - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#Test file pruning metrics with data skipping - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - double nested, single 1 - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - double nested, single 1 - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - backtick escapes work as expected - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - backtick escapes work as expected - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - index only a subset of leaf columns - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - index only a subset of leaf columns - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - naming a nested column allows nested complex types - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - naming a nested column allows nested complex types - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - naming a nested column indexes all leaf fields of that column - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - naming a nested column indexes all leaf fields of that column - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 3 - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 3 - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 6 - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 6 - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 9 - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested schema - # indexed column = 9 - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested, single 1 - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - nested, single 1 - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - starts with, nested - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - starts with, nested - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping flags - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping flags - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping shouldn't use expressions involving a subquery - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping shouldn't use expressions involving a subquery - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping stats before and after optimize - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping stats before and after optimize - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping with a different DataFrame schema order and nested columns - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping with missing columns in DataFrame - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#loading data from Delta to parquet should skip data - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#loading data from Delta to parquet should skip data - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#support case insensitivity for partitioning filters - column mapping name mode +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#support case insensitivity for partitioning filters - column mapping name mode - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping stats before and after optimize +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping stats before and after optimize - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping stats before and after optimize +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping stats before and after optimize - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Data skipping handles aliasing for _metadata fields +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Test file pruning metrics with data skipping +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping flags +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping flags - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping shouldn't use expressions involving a subquery +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#loading data from Delta to parquet should skip data +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#support case insensitivity for partitioning filters +org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema +org.apache.spark.sql.delta.stats.PartitionLikeDataSkippingColumnMappingSuite#partition-like data skipping for expression COALESCE: COALESCE(TO_DATE(S.b), c) = '1976-07-03' - column mapping id mode +org.apache.spark.sql.delta.stats.StatsCollectionSuite#gather stats +org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute stats multiple columns and files +org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute variant stats +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening BIGINT -> DECIMAL(20,0), partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DATE -> TIMESTAMP_NTZ, partitioned=false +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DATE -> TIMESTAMP_NTZ, partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DECIMAL(9,2) -> DECIMAL(19,3), partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening FLOAT -> DOUBLE, partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening INT -> DOUBLE, partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening with user-defined type in table +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#unsupported type changes DOUBLE -> FLOAT, partitioned=true +org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#unsupported type changes TIMESTAMP_NTZ -> DATE, partitioned=false +org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - always automatic type widening DATE -> TIMESTAMP_NTZ +org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - automatic type widening DATE -> TIMESTAMP_NTZ +org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - unsupported automatic type widening TIMESTAMP_NTZ -> DATE +org.apache.spark.sql.delta.typewidening.TypeWideningMergeIntoSchemaEvolutionSuite#MERGE - automatic type widening DATE -> TIMESTAMP_NTZ +org.apache.spark.sql.delta.typewidening.TypeWideningMergeIntoSchemaEvolutionSuite#MERGE - unsupported automatic type widening TIMESTAMP_NTZ -> DATE +org.apache.spark.sql.delta.typewidening.TypeWideningTableFeatureAdvancedSuite#drop feature after type change DATE -> TIMESTAMP_NTZ +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - Portable +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - invalid Int ids - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - invalid Int ids - Portable +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - invalid unsigned Int ids - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate no duplicates - invalid unsigned Int ids - Portable +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - Portable +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - invalid Int ids - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - invalid Int ids - Portable +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - invalid unsigned Int ids - Native +org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap groupBy aggregate with duplicates - invalid unsigned Int ids - Portable diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh new file mode 100755 index 00000000000..8da1b660ad7 --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -0,0 +1,177 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Prepares a delta-io/delta clone for running its `spark` module tests with the +# Gluten (Velox) bundle jar on the classpath. +# +# Usage: +# setup-delta.sh +# +# Arguments: +# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0) +# delta_dir - destination directory for the Delta clone +# gluten_bundle_jar - path to the gluten-velox-bundle fat jar +# gluten_repo_root - path to the Gluten repository root (used to locate +# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala) +# + +set -euo pipefail + +if [ "$#" -ne 4 ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +DELTA_REF="$1" +DELTA_DIR="$2" +GLUTEN_BUNDLE_JAR="$3" +GLUTEN_ROOT="$4" + +if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then + echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2 + exit 1 +fi + +# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module +# rather than maintaining a separate copy. This file is compiled as part of the +# unified `spark` project's Test scope, which has the Gluten bundle on its +# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig +# imports resolve correctly. +PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" +if [ ! -f "$PATCH_SOURCE" ]; then + echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2 + exit 1 +fi + +echo "::group::Cloning delta-io/delta @ ${DELTA_REF}" +# Shallow clone the requested tag/branch. Fall back to full clone when the ref is a SHA. +if ! git clone --depth 1 --branch "$DELTA_REF" https://github.com/delta-io/delta.git "$DELTA_DIR"; then + echo "Shallow clone of ref '${DELTA_REF}' failed, falling back to full clone." + rm -rf "$DELTA_DIR" + git clone https://github.com/delta-io/delta.git "$DELTA_DIR" + git -C "$DELTA_DIR" checkout "$DELTA_REF" +fi +git -C "$DELTA_DIR" --no-pager log -1 --oneline +echo "::endgroup::" + +echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath" +# The Gluten bundle jar must be on the spark project's TEST runtime classpath +# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but +# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds +# Delta's main sources. The bundle's transitive contents include extra symbols +# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g. +# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and +# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail +# with `reference to expression is ambiguous`. +# +# sbt auto-scans `/lib` via `unmanagedBase`. Two relevant +# projects in Delta v4.2.0 have a `lib/` baseDirectory: +# - sparkV1: `project in file("spark")` -> spark/lib +# - spark : `project in file("spark-unified")` -> spark-unified/lib +# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping +# the bundle into spark-unified/lib/ adds it to the unified `spark` project's +# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what +# we want: +# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles. +# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin. +# (Verified empirically: with bundle only in spark-unified/lib/, sbt's +# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and +# `show spark/Test/fullClasspath` includes it.) +# +# We deliberately do NOT also drop the bundle into spark/lib/, which is what +# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase, +# and putting the bundle there would re-introduce the ambiguity errors. +SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib" +mkdir -p "$SPARK_UNIFIED_LIB" +cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar" +ls -lh "$SPARK_UNIFIED_LIB" +echo "::endgroup::" + +echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin" +TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" +if [ ! -f "$TARGET" ]; then + echo "Expected file not found in Delta clone: $TARGET" >&2 + echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." + exit 1 +fi +cp "$PATCH_SOURCE" "$TARGET" +echo "Patched $TARGET" +echo "--- diff vs. upstream ---" +git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true +echo "::endgroup::" + +echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests" +# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table. +# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of +# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it. +# The dead fork then wedges sbt, hanging the whole shard until the workflow's +# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite +# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml. +# +# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST +# with a clear message: the gap stays visible in the test reports / baseline +# until the native memory blow-up is fixed, at which point this patch should be +# removed. NOTE: making the suite complete also un-skips the rest of the shard's +# suite queue, so the known-failures baseline must be refreshed after this. +DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" +if [ ! -f "$DVS" ]; then + echo "Expected file not found in Delta clone: $DVS" >&2 + echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2 + exit 1 +fi +# Inject `fail(...)` as the first statement of each test body (the line ending +# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now- +# unreachable original body compiles fine. Keep each injected line <100 chars: +# Delta's scalastyle enforces a 100-char line length on test sources. The full +# rationale lives in this comment, so the in-test message stays terse. +sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS" +sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS" +INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true) +if [ "$INJECTED" -ne 2 ]; then + echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but injected ${INJECTED}." >&2 + echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update setup-delta.sh." >&2 + exit 1 +fi +echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)." +git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" || true +echo "::endgroup::" + +echo "::group::Disabling Delta scalastyle HeaderMatchesChecker" +# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header, which +# does not match Delta's HeaderMatchesChecker regex (the regex expects either a +# Delta copyright block, or the ASF header followed by a Spark-modifications +# block and the Delta copyright block). HeaderMatchesChecker is a file-level +# checker that does NOT honor `// scalastyle:off` directives, so we instead +# disable it globally in Delta's shared scalastyle-config.xml. The config is +# applied via `ThisBuild / scalastyleConfig` in project/Checkstyle.scala, so a +# single edit covers every sbt sub-project. +SCALASTYLE_CONFIG="$DELTA_DIR/scalastyle-config.xml" +if [ ! -f "$SCALASTYLE_CONFIG" ]; then + echo "Expected scalastyle config not found: $SCALASTYLE_CONFIG" >&2 + exit 1 +fi +sed -i \ + 's|||' \ + "$SCALASTYLE_CONFIG" +if ! grep -q '' "$SCALASTYLE_CONFIG"; then + echo "Failed to disable HeaderMatchesChecker in $SCALASTYLE_CONFIG" >&2 + grep -n 'HeaderMatchesChecker' "$SCALASTYLE_CONFIG" >&2 || true + exit 1 +fi +echo "Disabled HeaderMatchesChecker in $SCALASTYLE_CONFIG" +echo "::endgroup::" From b21b577e4b9641f005842b3933b99f5d968f9400 Mon Sep 17 00:00:00 2001 From: Felipe Pessoto Date: Sun, 28 Jun 2026 02:02:36 -0700 Subject: [PATCH 02/28] Change order of steps Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 8a7d91f2c06..a96c95df940 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -161,11 +161,6 @@ jobs: with: name: delta-spark-ut-native-lib-centos-7-${{github.sha}} path: ./cpp/build/ - - name: Download Arrow jars - uses: actions/download-artifact@v4 - with: - name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} - path: /root/.m2/repository/org/apache/arrow/ - name: Cache Maven repository uses: actions/cache@v4 with: @@ -174,6 +169,11 @@ jobs: restore-keys: | m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}- m2-delta-spark-ut-bundle- + - name: Download Arrow jars + uses: actions/download-artifact@v4 + with: + name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} + path: /root/.m2/repository/org/apache/arrow/ - name: Build Gluten Velox + Delta bundle run: | set -euo pipefail From 11e4bb3a5694fe4cd9de87dba663564639d3cde2 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Sun, 28 Jun 2026 00:29:26 +0000 Subject: [PATCH 03/28] [CI] Cherry-pick Delta FileSourceScanLike test fixes and refresh baseline Delta's data-skipping, limit-push-down, column-pruning and scan-metric tests collect file-source scans by matching the concrete `FileSourceScanExec` case class. Under the Gluten Velox bundle the scan is offloaded to DeltaScanTransformer, a sibling that implements the same `FileSourceScanLike` interface but is not FileSourceScanExec, so the match misses and the scan looks absent. This surfaced as `scala.MatchError: List()` (~56 DataSkipping*/DeltaLimitPushDown* tests), empty generated-column partition filters (~45 OptimizeGeneratedColumnSuite tests) and broken column-pruning / scan-metric checks across the Delete, Update, Merge, DeletionVectors and RowId suites and the TestsStatistics helper. Gluten copies `partitionFilters` and the other accessors these tests read verbatim onto the offloaded scan, so results are identical to vanilla -- only the test's `case` match breaks. Fix it by cherry-picking the two merged upstream Delta commits that widen these matches to the shared `FileSourceScanLike` interface (behavior-preserving for vanilla, which also implements it): * delta-io/delta#7104 -- ScanReportHelper.collectScans * delta-io/delta#7105 -- the remaining 9 test sources, its follow-up Both are merged on Delta master but land after the ref this workflow builds against (v4.2.0), so setup-delta.sh cherry-picks them onto the shallow checkout. Each fetches the fix commit at depth 2 (commit + parent) so cherry-pick can compute the parent->fix diff, and uses `cherry-pick -n` so no committer identity is required. Once the pinned DELTA_REF advances to include a commit its cherry-pick becomes a clean no-op and that block can be removed. The cherry-picks run before the DeletionVectorsSuite 2B-row force-fail step: that step sed-injects fail() into DeletionVectorsSuite.scala, which delta-io/delta#7105 also edits, and git cherry-pick refuses to apply onto a working tree with uncommitted changes to a file it touches (exit 128). Refresh known-failures.txt from run 28299900971 (the delta-spark-aggregate job output), which ran all 19073 tests across 16 shards: removes 187 now-passing tests with 0 regressions, 963 -> 776. ~147 come from the fixes above (DataSkipping*, DeltaLimitPushDown*, OptimizeGeneratedColumnSuite, MergeInto*, RowIdSuite); the remaining ~40 are other suites that now pass (e.g. HiveConvertToDeltaSuite, BitmapAggregatorE2ESuite). Verified against the per-shard ran/failed lists: every baseline entry was observed this run (0 stale), so nothing was dropped due to a crashed or incomplete shard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../util/delta-spark-ut/known-failures.txt | 166 +----------------- .../util/delta-spark-ut/setup-delta.sh | 29 +++ 2 files changed, 38 insertions(+), 157 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index ae7b300084b..18487b7e75e 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -7,10 +7,15 @@ # Lines starting with '#' are comments. See README.md in this directory. # # --------------------------------------------------------------------------- -# Full 16-shard baseline. Originally seeded from 15 of 16 shards (run -# 27490052632). Shard 2 used to hang/OOM-crash on DeletionVectorsSuite's 2B-row -# DV tests; those two tests are now force-failed in setup-delta.sh, so shard 2 -# runs to completion and contributes 69 failures. 963 known failures total. +# Baseline for the committed 4-shard x 4-fork config. Originally seeded from run +# 27490052632; refreshed from run 28318129710 (4 x 4) after the FileSourceScanLike +# test fixes (delta-io/delta #7104 + #7105). 810 known failures total. +# +# IMPORTANT: regenerate this baseline under the SAME NUM_SHARDS x +# TEST_PARALLELISM_COUNT the gate runs (here 4 x 4). ~34 of these failures are +# fork-parallelism-sensitive -- they pass with 1 fork per JVM but fail with 4 (4 +# Velox forks x ~4G run close to the ~16G runner limit) -- so a baseline captured +# at a different parallelism spuriously flags them as regressions. # --------------------------------------------------------------------------- io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using SQL conf io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using withExtensions @@ -110,37 +115,8 @@ org.apache.spark.sql.delta.DeltaInsertIntoDataFrameByPathSuite#insertInto: times org.apache.spark.sql.delta.DeltaInsertIntoDataFrameSuite#insertInto: timestamp partition values with different precisions org.apache.spark.sql.delta.DeltaInsertIntoSQLByPathSuite#insertInto: timestamp partition values with different precisions org.apache.spark.sql.delta.DeltaInsertIntoSQLSuite#insertInto: timestamp partition values with different precisions -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#Works with union -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#limit larger than total -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#limit push-down flag -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#no filter or projection -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with non-partition filter -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with partition filter only -org.apache.spark.sql.delta.DeltaLimitPushDownV1Suite#with projection only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#Works with union -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#limit larger than total -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#limit push-down flag -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#no filter or projection -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with non-partition filter -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with partition filter only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch100Suite#with projection only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#Works with union -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#limit larger than total -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#limit push-down flag -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#no filter or projection -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with non-partition filter -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with partition filter only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch1Suite#with projection only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#Works with union -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#limit larger than total -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#limit push-down flag -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#no filter or projection -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with non-partition filter -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with partition filter only -org.apache.spark.sql.delta.DeltaLimitPushDownWithCatalogOwnedBatch2Suite#with projection only org.apache.spark.sql.delta.DeltaLiteVacuumSuite#vacuum for cdc - delete tombstones org.apache.spark.sql.delta.DeltaLiteVacuumSuite#vacuum for cdc - update/merge -org.apache.spark.sql.delta.DeltaNameColumnMappingSuite#query with predicates should skip partitions - column mapping name mode org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=false, with readColumnarBatchAsRows=true org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=false org.apache.spark.sql.delta.DeltaParquetFileFormatSuite#isDeletionVectorsEnabled=false, read DV metadata columns: with isRowDeletedCol=false, with rowIndexCol=false, with vectorized Parquet reader=true, with readColumnarBatchAsRows=true @@ -172,7 +148,6 @@ org.apache.spark.sql.delta.DeltaSuite#SC-8810: skipping deleted file still throw org.apache.spark.sql.delta.DeltaSuite#all operations with special characters in path org.apache.spark.sql.delta.DeltaSuite#deleted files cause failure by default org.apache.spark.sql.delta.DeltaSuite#invalid replaceWhere -org.apache.spark.sql.delta.DeltaSuite#query with predicates should skip partitions org.apache.spark.sql.delta.DeltaSuite#replaceArbitrary should enforce proper usage of backtick org.apache.spark.sql.delta.DeltaTableCreationSuite#Default column values: CONVERT TO DELTA keeps EXISTS_DEFAULT org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#convert to delta with partitioning change @@ -204,19 +179,16 @@ org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skip dele org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skipping deleted file still throws on corrupted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#deleted files cause failure by default org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#invalid replaceWhere -org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#query with predicates should skip partitions org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#replaceArbitrary should enforce proper usage of backtick org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#SC-8810: skip deleted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#SC-8810: skipping deleted file still throws on corrupted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#deleted files cause failure by default org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#invalid replaceWhere -org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#query with predicates should skip partitions org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch1Suite#replaceArbitrary should enforce proper usage of backtick org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#SC-8810: skip deleted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#SC-8810: skipping deleted file still throws on corrupted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#deleted files cause failure by default org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#invalid replaceWhere -org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#query with predicates should skip partitions org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch2Suite#replaceArbitrary should enforce proper usage of backtick org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = false, CDF = false org.apache.spark.sql.delta.DescribeDeltaHistorySuite#merge-metrics: delete-only - Partitioned = false, CDF = true @@ -639,53 +611,39 @@ org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumn org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnScalaSuite#schema evolution - extra nested column in source - update - single target partition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - partition on multiple columns org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - partition on multiple columns org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - partition on multiple columns org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - partition on multiple columns org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#UDT Data Types - simple and nested org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#data skipping - target-only condition org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#data skipping with matched predicates - with insert clause -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#insert only merge - target data skipping org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLNameBasedSuite#Variant type org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnDVSuite#Variant type @@ -719,64 +677,15 @@ org.apache.spark.sql.delta.optimize.OptimizeCompactionScalaSuite#optimize - mult org.apache.spark.sql.delta.optimize.OptimizeConflictSuite#conflict handling between Optimize and Business Txn org.apache.spark.sql.delta.optimize.OptimizeMetricsSuite#optimize ZOrderBy operation metrics in Delta table history org.apache.spark.sql.delta.optimize.OptimizeMetricsSuite#optimize metrics on idempotent operations -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(day,yyyy-MM-dd) from timestamp -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(day,yyyy-MM-dd) from timestamp nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(hour,yyyy-MM-dd-HH) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(hour,yyyy-MM-dd-HH) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from cast(date) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from cast(date) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from timestamp -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DateFormatPartitionExpr(month,yyyy-MM) from timestamp nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(date) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(date) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(timestamp) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#DatePartitionExpr(date) from cast(timestamp) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#IdentityPartitionExpr(part1) escaped field names -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(my.substr,1,3) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(my.substr,1,3) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,0,3) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,0,3) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) deeply nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,1,3) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,2,3) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#SubstringPartitionExpr(substr,2,3) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(DD,eventTimeTrunc) from date_trunc(cast(date)) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(DD,eventTimeTrunc) from date_trunc(cast(date)) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(YEAR,eventTimeTrunc) from date_trunc(timestamp) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TimestampTruncPartitionExpr(YEAR,eventTimeTrunc) from date_trunc(timestamp) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,month) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,month) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,quarter) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,quarter) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,year) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#TruncDatePartitionExpr(date,year) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayHourPartitionExpr(year,month,day,hour) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayHourPartitionExpr(year,month,day,hour) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayPartitionExpr(year,month,day) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthDayPartitionExpr(year,month,day) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthPartitionExpr(year,month) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearMonthPartitionExpr(year,month) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(cast(date)) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(cast(date)) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(date) -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) from year(date) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#YearPartitionExpr(year) nested -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#end-to-end optimizable partition expression org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#end-to-end test of behaviors of write/read null on partition column org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#five digits year in a date_format yyyy-MM partition column org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#five digits year in a date_format yyyy-MM-dd-HH partition column -org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite#substring on multibyte characters org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#Re-added files keep their row IDs after conflict with txn not updating high watermark org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#concurrent transactions do not assign overlapping row IDs org.apache.spark.sql.delta.rowid.ConflictCheckerRowIdSuite#re-added files keep their row ids org.apache.spark.sql.delta.rowid.RowIdSuite#Filter by base Row IDs org.apache.spark.sql.delta.rowid.RowIdSuite#Filter by base Row IDs in subquery org.apache.spark.sql.delta.rowid.RowIdSuite#No dictionary filtering on _metadata.row_id -org.apache.spark.sql.delta.rowid.RowIdSuite#No row-group skipping on _metadata.row_id org.apache.spark.sql.delta.rowid.RowIdSuite#missing base row ids and default row commit versions org.apache.spark.sql.delta.rowid.RowIdSuite#row ids can be read back org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencySuite#Interleaved delete right after protocol downgrade should abort due to protocol change @@ -790,8 +699,6 @@ org.apache.spark.sql.delta.rowid.RowTrackingRemovalConcurrencyWithoutDVsSuite#Si org.apache.spark.sql.delta.rowtracking.RowTrackingReadWriteSuite#write and read table with all-null materialized columns org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ @@ -800,16 +707,8 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data s org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping stats before and after optimize org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#data skipping stats before and after optimize - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1JsonCheckpointV2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#Test file pruning metrics with data skipping - column mapping name mode -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#Test file pruning metrics with data skipping - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - double nested, single 1 - column mapping name mode org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - double nested, single 1 - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping by stats - indexed column names - backtick escapes work as expected - column mapping name mode @@ -840,20 +739,12 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - column mapping name mode org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - column mapping name mode - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping shouldn't use expressions involving a subquery - column mapping name mode -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping shouldn't use expressions involving a subquery - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping stats before and after optimize - column mapping name mode org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping stats before and after optimize - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping with a different DataFrame schema order and nested columns - column mapping name mode org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#data skipping with missing columns in DataFrame - column mapping name mode -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#loading data from Delta to parquet should skip data - column mapping name mode -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#loading data from Delta to parquet should skip data - column mapping name mode - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#support case insensitivity for partitioning filters - column mapping name mode -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1NameColumnMappingSuite#support case insensitivity for partitioning filters - column mapping name mode - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ @@ -862,18 +753,10 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#dat org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping stats before and after optimize org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#data skipping stats before and after optimize - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1ParquetCheckpointV2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ @@ -882,18 +765,10 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMES org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping stats before and after optimize org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#data skipping stats before and after optimize - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ @@ -902,16 +777,8 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suit org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch100Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ @@ -920,16 +787,8 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite# org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch1Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Data skipping handles aliasing for _metadata fields org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Data skipping handles aliasing for _metadata fields - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Test file pruning metrics with data skipping -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#Test file pruning metrics with data skipping - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping flags org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping flags - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ @@ -938,14 +797,7 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite# org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ near Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping shouldn't use expressions involving a subquery -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping shouldn't use expressions involving a subquery - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#loading data from Delta to parquet should skip data -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#loading data from Delta to parquet should skip data - old behavior with DataFrame schema -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#support case insensitivity for partitioning filters -org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#support case insensitivity for partitioning filters - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.PartitionLikeDataSkippingColumnMappingSuite#partition-like data skipping for expression COALESCE: COALESCE(TO_DATE(S.b), c) = '1976-07-03' - column mapping id mode -org.apache.spark.sql.delta.stats.StatsCollectionSuite#gather stats org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute stats multiple columns and files org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute variant stats org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening BIGINT -> DECIMAL(20,0), partitioned=true diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 8da1b660ad7..9feab22f7ca 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -115,6 +115,31 @@ echo "--- diff vs. upstream ---" git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true echo "::endgroup::" +# Delta's tests collect file-source scans by matching the concrete +# `FileSourceScanExec` case class; Gluten offloads the scan to +# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss +# (`scala.MatchError: List()`, empty partition filters, broken column-pruning / +# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the +# matches to the shared `FileSourceScanLike` interface that both the vanilla and +# Gluten scans implement (behavior-preserving for vanilla). Both are merged +# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once +# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go. +# +# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to +# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change +# without requiring a committer identity. +cherry_pick_delta_fix() { + local sha="$1" pr="$2" + echo "Cherry-picking delta-io/delta${pr}" + git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha" + git -C "$DELTA_DIR" cherry-pick -n "$sha" +} + +echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes" +cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)" +cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)" +echo "::endgroup::" + echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests" # Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table. # Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of @@ -128,6 +153,10 @@ echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests" # until the native memory blow-up is fixed, at which point this patch should be # removed. NOTE: making the suite complete also un-skips the rest of the shard's # suite queue, so the known-failures baseline must be refreshed after this. +# +# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits +# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work +# tree has uncommitted edits to a file it touches. DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" if [ ! -f "$DVS" ]; then echo "Expected file not found in Delta clone: $DVS" >&2 From 40a887f28d706f98a6884f30a138b8dc55bf39f0 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Sun, 28 Jun 2026 01:08:01 +0000 Subject: [PATCH 04/28] [CI] Reuse velox_backend_x86's native build for Delta Spark UT Make delta_spark_ut.yml a reusable workflow (on: workflow_call) and call it from velox_backend_x86.yml so the Delta tests reuse the native lib + arrow jars that workflow already builds, instead of duplicating the build-native-lib-centos-7 job. GitHub artifacts cannot be shared across workflows, so the only way to reuse the artifact is to run the Delta jobs in the same workflow run. delta_spark_ut.yml keeps a workflow_dispatch trigger for standalone manual runs (its build-native-lib-centos-7 job is gated to that case and skipped when called); the pull_request trigger is removed so the suite no longer double-runs. velox_backend_x86.yml gains an arrow-jars upload on its native build and a delta-spark-ut job that calls the reusable workflow. That job runs on every velox trigger like the other spark-test jobs, since core/velox/substrait/cpp changes can affect Delta query offload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 192 +++++++++++++----------- .github/workflows/velox_backend_x86.yml | 27 ++++ 2 files changed, 131 insertions(+), 88 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index a96c95df940..dbc210afedf 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -32,6 +32,46 @@ name: Delta Spark UT (Gluten) on: + # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant + # changes) and passes the native-lib + arrow-jars artifacts it already built, + # so the expensive native C++ build is NOT duplicated. Those artifacts live in + # the CALLER's run (a called workflow runs as part of the caller run), so the + # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`. + # + # NOTE: the `pull_request` trigger was removed so this no longer runs as its own + # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml + # is now the single PR entry point; `workflow_dispatch` keeps manual standalone + # runs working (those build the native lib themselves -- see build-native-lib). + workflow_call: + inputs: + native_lib_artifact: + description: 'Name of the cpp/build artifact uploaded by the caller' + type: string + required: true + arrow_jars_artifact: + description: 'Name of the org.apache.arrow jars artifact uploaded by the caller' + type: string + required: true + delta_ref: + type: string + required: false + default: 'v4.2.0' + spark_version: + type: string + required: false + default: '4.1' + test_parallelism: + type: string + required: false + default: '4' + update_baseline: + type: boolean + required: false + default: false + fail_on_fixed: + type: boolean + required: false + default: true workflow_dispatch: inputs: delta_ref: @@ -56,12 +96,6 @@ on: type: boolean required: false default: true - pull_request: - paths: - - '.github/workflows/delta_spark_ut.yml' - - '.github/workflows/util/delta-spark-ut/**' - - 'gluten-delta/**' - - 'backends-velox/src-delta40/**/DeltaSQLCommandTest.scala' env: ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true @@ -75,37 +109,32 @@ env: GLUTEN_JAVA_PROFILE: 'java-17' GLUTEN_BUNDLE_SPARK_VERSION: '4.1' GLUTEN_BUNDLE_SCALA_VERSION: '2.13' - # Default values used when the workflow is triggered by pull_request - # (where `inputs.*` is empty). Keep these in sync with the workflow_dispatch - # defaults above. - DELTA_REF_DEFAULT: 'v4.2.0' - DELTA_SPARK_VERSION_DEFAULT: '4.1' - DELTA_TEST_PARALLELISM_DEFAULT: '4' - # Default mode for pull_request runs (where inputs.* is empty): enforce the - # committed baseline and fail when a baseline test starts passing. Override - # via the workflow_dispatch inputs above. - DELTA_UPDATE_BASELINE_DEFAULT: 'false' - DELTA_FAIL_ON_FIXED_DEFAULT: 'true' DELTA_SCALA_VERSION: '2.13.16' # Number of shards in the delta-spark-test matrix. Must equal the length of # the `shard` matrix below. # - # EXPERIMENT: 4 shards x TEST_PARALLELISM_COUNT=4 (vs production 16 shards x 1). - # Both give ~16-way parallelism, but this packs it into 4 runner jobs (4 forks - # each) instead of 16 single-fork jobs -- fewer concurrent runners for the same - # throughput. Sharding is by SUITE; total work (~1250 shard-minutes) is fixed. - # RISK: each forked test JVM uses ~4G (2G heap + 2G off-heap), so 4 forks atop - # the sbt launcher push the ~16G runner to its limit and may OOM on heavy suites - # -- which is why production uses TEST_PARALLELISM_COUNT=1. Measuring whether it - # fits now that the worst memory hog (DeletionVectorsSuite 2B-row) is force-failed. + # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4 + # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent + # runners for the same throughput. Sharding is by SUITE; total work + # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G + # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit; + # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is + # force-failed in setup-delta.sh. DELTA_NUM_SHARDS: '4' -concurrency: - group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} - cancel-in-progress: true +# No `concurrency:` here on purpose. As a reusable workflow this runs inside the +# caller's run, where `github.workflow` resolves to the CALLER's name -- a group +# keyed on it would collide with the caller's own group and, with +# cancel-in-progress, could cancel the parent run. The caller's concurrency +# already governs cancellation. (A standalone workflow_dispatch run just won't +# auto-cancel, which is fine for infrequent manual runs.) jobs: build-native-lib-centos-7: + # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml + # the caller already built the native lib + arrow jars and passes them as + # inputs, so this job is skipped and the duplicate native build is avoided. + if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 @@ -152,6 +181,9 @@ jobs: build-gluten-bundle: needs: build-native-lib-centos-7 + # Run whether the native lib was built here (dispatch -> success) or provided + # by the caller (workflow_call -> build-native-lib-centos-7 skipped). + if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }} runs-on: ubuntu-22.04 container: apache/gluten:centos-9-jdk17 steps: @@ -159,7 +191,7 @@ jobs: - name: Download native artifacts uses: actions/download-artifact@v4 with: - name: delta-spark-ut-native-lib-centos-7-${{github.sha}} + name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }} path: ./cpp/build/ - name: Cache Maven repository uses: actions/cache@v4 @@ -172,7 +204,7 @@ jobs: - name: Download Arrow jars uses: actions/download-artifact@v4 with: - name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} + name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }} path: /root/.m2/repository/org/apache/arrow/ - name: Build Gluten Velox + Delta bundle run: | @@ -216,10 +248,15 @@ jobs: delta-spark-test: needs: build-gluten-bundle + # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need + # is skipped on workflow_call), so this job needs an explicit condition too -- + # otherwise GitHub's transitive skip propagation, seeing the skipped + # build-native-lib-centos-7 ancestor, would skip the whole shard matrix. + if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }} runs-on: ubuntu-22.04 container: apache/gluten:centos-9-jdk17 - # EXPERIMENT (4 shards x 4 forks): back to 350 -- per-shard suites now run - # 4-at-a-time, so each shard should finish well under the cap again. + # 350-min safety cap. With 4 forks per shard the per-shard suites run + # 4-at-a-time, so a shard finishes well under this. timeout-minutes: 350 strategy: fail-fast: false @@ -235,24 +272,22 @@ jobs: - name: Resolve workflow inputs id: resolve + # Every input has a default (workflow_call + workflow_dispatch), so they + # are always set; just surface them as step outputs for the steps below. + env: + DELTA_REF: ${{ inputs.delta_ref }} + SPARK_VERSION: ${{ inputs.spark_version }} + TEST_PARALLELISM: ${{ inputs.test_parallelism }} + UPDATE_BASELINE: ${{ inputs.update_baseline }} + FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }} run: | set -euo pipefail - delta_ref='${{ github.event.inputs.delta_ref }}' - spark_version='${{ github.event.inputs.spark_version }}' - test_parallelism='${{ github.event.inputs.test_parallelism }}' - update_baseline='${{ github.event.inputs.update_baseline }}' - fail_on_fixed='${{ github.event.inputs.fail_on_fixed }}' - : "${delta_ref:=${DELTA_REF_DEFAULT}}" - : "${spark_version:=${DELTA_SPARK_VERSION_DEFAULT}}" - : "${test_parallelism:=${DELTA_TEST_PARALLELISM_DEFAULT}}" - : "${update_baseline:=${DELTA_UPDATE_BASELINE_DEFAULT}}" - : "${fail_on_fixed:=${DELTA_FAIL_ON_FIXED_DEFAULT}}" { - echo "delta_ref=${delta_ref}" - echo "spark_version=${spark_version}" - echo "test_parallelism=${test_parallelism}" - echo "update_baseline=${update_baseline}" - echo "fail_on_fixed=${fail_on_fixed}" + echo "delta_ref=${DELTA_REF}" + echo "spark_version=${SPARK_VERSION}" + echo "test_parallelism=${TEST_PARALLELISM}" + echo "update_baseline=${UPDATE_BASELINE}" + echo "fail_on_fixed=${FAIL_ON_FIXED}" } | tee -a "$GITHUB_OUTPUT" - name: Download Gluten bundle jar @@ -371,45 +406,26 @@ jobs: # `sparkGroup` aggregates many other projects (sparkV2, contribs, # sharing, connect*, ...) that are out of scope for this pipeline. # - # JVM heap layout (16 GB ubuntu-22.04 runner, TEST_PARALLELISM=1): - # * sbt launcher JVM: -J-Xmx4G, BUT made to RETURN idle memory (see the - # G1 periodic-GC flags below). The per-minute MEM profiler in the - # watchdog (run #18) DISPROVED the old "launcher RSS is well under 4G" - # assumption: the launcher grew to a ROCK-STEADY 5.3G (RSS) during the - # test-compile and then HELD it for the entire run -- ~3.8G of pure - # idle waste during the (long) test phase, where it only relays the - # fork's test events. That fixed 5.3G + the fork's native spike in a - # heavy suite is what pushed the cgroup to the ~16G OOM-kill. G1 does - # NOT uncommit on its own here because the idle launcher never GCs. - # FIX (behaviour-neutral -- touches NO Gluten/Spark runtime config, so - # it cannot pollute the measured pass/fail signal): keep -Xmx4G for the - # compile headroom but force a periodic GC every 10s when idle - # (G1PeriodicGCInterval) with the system-load gate disabled - # (G1PeriodicGCSystemLoadThreshold=0, since the busy fork would - # otherwise suppress it) as a full STW collection - # (-G1PeriodicGCInvokesConcurrent) that uncommits down to a tight free - # ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low -Xms512m - # floor. The idle launcher then drops from ~5.3G back to ~1-2G during - # the test phase, cutting the cgroup peak by ~3.8G (~15.9G -> ~12G) -- - # real headroom under the OOM threshold -- with zero compile-OOM risk. - # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command - # below. Delta v4.2.0 caps its test fork at `-Xmx1024m` in build.sbt; - # Gluten OFFLOADS data to Velox off-heap (capped at 2g via - # spark.memory.offHeap.size in the patched DeltaSQLCommandTest), so the - # fork's JVM HEAP need is modest -- 2G is generous. - # HISTORY: briefly bumped to 8G then 4G to absorb a DV+CDC merge suite's - # giant RoaringBitmapArray heap allocation, but that was a Gluten bug - # (garbage native _metadata.row_index) FIXED UPSTREAM by #12269 -- so the - # large heap is no longer needed. Worse, on the ~16G runner the cgroup - # memory.peak hit 15.97G with a 4G fork heap and the kernel OOM-killed - # the fork mid-shard (sudden death, no hs_err / no heap dump), which - # wedged sbt's main process forever in ScalaTestRunner.done -> Thread - # .join (the chronic "shard 2 hang"). The JVM heaps -- not off-heap -- - # drive that peak, so 2G fork heap (+ the unchanged 4G sbt launcher, - # which needs its heap to COMPILE the tests) brings the peak to ~13G, - # leaving real headroom. The `++=` appends to Delta's own Test/javaOptions - # seq so our `-Xmx2G` comes AFTER `-Xmx1024m` and wins (last `-Xmx` - # wins). Keep heap-dump-on-OOM so a genuine >2G heap OOM is analyzable. + # JVM heap layout -- two memory consumers on the ~16G runner: + # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to + # return idle memory during the (long) test phase via G1 periodic GC + # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the + # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent + # forces each periodic GC to a full STW collection) that uncommits to a + # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low + # -Xms512m floor. + # Without this the idle launcher holds ~5.3G for the whole run; with + # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime + # config, so they cannot affect the measured pass/fail signal. + # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command + # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends + # so our -Xmx2G comes last and wins. Gluten offloads data to Velox + # off-heap (capped at 2g via spark.memory.offHeap.size in the patched + # DeltaSQLCommandTest), so the fork's heap need is modest. A larger + # fork heap pushed the cgroup peak past the ~16G OOM threshold and the + # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G + # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is + # analyzable. # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so # every suite writes per-test results. Delta itself only configures # the console reporter (-oDF), so without this we'd have no machine- @@ -440,8 +456,8 @@ jobs: # subshell inherits. Without `set +e` here, ANY non-zero command -- # e.g. fork detection finding no match, or `kill`/`jps` returning # non-zero -- silently kills this watchdog. That errexit kill (plus a - # /proc detection miss) is why the watchdog captured ZERO dumps in - # runs #12 and #13. A diagnostic must never abort on a failed probe. + # /proc detection miss) once made the watchdog capture ZERO dumps. A + # diagnostic must never abort on a failed probe. set +e +o pipefail JSTACK="${JAVA_HOME}/bin/jstack" JPS="${JAVA_HOME}/bin/jps" diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index 2e34f3084b5..edb8173d416 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -19,6 +19,11 @@ on: pull_request: paths: - '.github/workflows/velox_backend_x86.yml' + # Delta Spark UT runs here too (reusable delta_spark_ut.yml). These extra + # paths make Delta-CI-only changes trigger this workflow; Delta also runs on + # the velox paths below since core/velox changes can affect Delta offload. + - '.github/workflows/delta_spark_ut.yml' + - '.github/workflows/util/delta-spark-ut/**' - '.github/workflows/util/install-spark-deps.sh' #TODO remove after image update - '.github/workflows/util/install-spark-resources.sh' #TODO remove after image update - 'pom.xml' @@ -86,6 +91,10 @@ jobs: ccache -sz bash dev/ci-velox-buildstatic-centos-7.sh ccache -s + # Stage the custom-built org.apache.arrow jars so the reusable Delta + # workflow's bundle build can consume them (see delta-spark-ut job). + mkdir -p /work/.m2/repository/org/apache/arrow/ + cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/ " - name: "Save ccache" @@ -100,6 +109,24 @@ jobs: name: velox-native-lib-centos-7-${{github.sha}} path: ./cpp/build/ if-no-files-found: error + # Consumed by the reusable Delta workflow (delta-spark-ut job). + - uses: actions/upload-artifact@v4 + with: + name: velox-arrow-jars-centos-7-${{github.sha}} + path: .m2/repository/org/apache/arrow/ + if-no-files-found: error + + # Delta Spark UT, run via the reusable workflow so it reuses the native lib + + # arrow jars built above instead of duplicating the native build. Not gated on + # Delta-only paths: core/velox/substrait/cpp/shims changes can affect Delta + # query offload, so this runs on every trigger like the other spark-test jobs. + delta-spark-ut: + needs: build-native-lib-centos-7 + uses: ./.github/workflows/delta_spark_ut.yml + with: + native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }} + arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }} + secrets: inherit tpc-test-ubuntu: needs: build-native-lib-centos-7 From 0cec09cfc066db8f144598224c836b7362f8db0b Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 30 Jun 2026 05:42:35 +0000 Subject: [PATCH 05/28] [CI] Harden Delta UT setup clone and enforce-mode baseline check Address PR review feedback: - setup-delta.sh: replace the shallow-clone + full-clone fallback (which ran a destructive `rm -rf "$DELTA_DIR"`) with a single `git init` + shallow `fetch --depth 1 origin "$DELTA_REF"` + `checkout FETCH_HEAD`. This resolves a tag, branch, or commit SHA uniformly (`git clone --branch` rejects SHAs), drops the dead fallback branch, and removes the unguarded recursive delete. - compare-test-results.py: in enforce mode, a missing/typoed --known-failures path made load_entries() return an empty set, silently degrading to seed mode and passing the gate without enforcing regressions. Treat a missing baseline file as a configuration error (exit 2); an existing-but-empty file is still allowed and legitimately seeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../delta-spark-ut/compare-test-results.py | 21 +++++++++++++++++-- .../util/delta-spark-ut/setup-delta.sh | 14 ++++++------- 2 files changed, 26 insertions(+), 9 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index bed6d18712e..7a0868ef64a 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -36,8 +36,10 @@ fail the build (``--fail-on-fixed true``) so the baseline stays honest and contributors remove entries as they fix them. - If the baseline is empty (not yet bootstrapped) the mode automatically - degrades to ``seed`` so the first run is never spuriously red. + If the baseline file exists but is empty (not yet bootstrapped) the mode + automatically degrades to ``seed`` so the first run is never spuriously red. + A *missing* ``--known-failures`` file is treated as a configuration error + (the gate fails) so a mis-referenced path can't silently pass. ``seed`` (bootstrap / ``update_baseline``) Never fails. Just writes the current shard's failing tests so the baseline @@ -257,6 +259,21 @@ def _print_block(write, title, entries, limit=50): # Modes # --------------------------------------------------------------------------- # def run_enforce(args): + # In enforce mode a missing baseline file would make load_entries() return an + # empty set, silently degrading to seed mode and passing the gate without + # enforcing anything. Treat a missing path as a configuration error; an + # existing-but-empty file is still allowed (it legitimately seeds). + if args.mode == "enforce" and ( + not args.known_failures or not os.path.exists(args.known_failures) + ): + eprint( + "ERROR: --known-failures '{}' does not exist. In enforce mode the " + "baseline file must exist (an existing-but-empty file is allowed and " + "triggers seed mode). Refusing to silently pass.".format( + args.known_failures + ) + ) + return 2 baseline = load_entries(args.known_failures) passed, failed, skipped = parse_reports(args.reports_dir) diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 9feab22f7ca..684df319d70 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -59,13 +59,13 @@ if [ ! -f "$PATCH_SOURCE" ]; then fi echo "::group::Cloning delta-io/delta @ ${DELTA_REF}" -# Shallow clone the requested tag/branch. Fall back to full clone when the ref is a SHA. -if ! git clone --depth 1 --branch "$DELTA_REF" https://github.com/delta-io/delta.git "$DELTA_DIR"; then - echo "Shallow clone of ref '${DELTA_REF}' failed, falling back to full clone." - rm -rf "$DELTA_DIR" - git clone https://github.com/delta-io/delta.git "$DELTA_DIR" - git -C "$DELTA_DIR" checkout "$DELTA_REF" -fi +# init + shallow fetch resolves a tag, branch OR commit SHA in a single path +# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the +# destructive `rm -rf "$DELTA_DIR"` it required. +git init -q "$DELTA_DIR" +git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git +git -C "$DELTA_DIR" fetch -q --depth 1 origin "$DELTA_REF" +git -C "$DELTA_DIR" checkout -q FETCH_HEAD git -C "$DELTA_DIR" --no-pager log -1 --oneline echo "::endgroup::" From 866e05731384c55b4fe18aa7b3611315c5392748 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 30 Jun 2026 23:55:22 +0000 Subject: [PATCH 06/28] [CI] Harden Delta UT gate and setup against silent failures Address PR review feedback with four robustness fixes: - compare-test-results.py (enforce/seed): raise NoReportsError and exit 2 when no JUnit elements are parsed, instead of warning and returning empty sets. Otherwise a misconfiguration (wrong reports dir, broken reporter, suites crashing before writing XML) yields zero failures -> zero regressions -> a silent green gate. - compare-test-results.py (aggregate): exit 2 before writing baseline-out when no per-shard failures-*.txt / ran-*.txt inputs are found. The gate-list download is continue-on-error and aggregate runs with if: always(), so missing artifacts would otherwise produce an empty baseline that could be committed, wiping known-failures.txt. - setup-delta.sh: pass the Delta ref after `--` in git fetch so a ref starting with `-` can't be misread as a git option (the script is workflow_dispatch- runnable with a user-supplied ref). - velox_backend_x86.yml: drop secrets: inherit from the reusable Delta UT call. delta_spark_ut.yml references no secrets, so inheriting them needlessly forwards all caller secrets to a workflow that clones and runs external code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../delta-spark-ut/compare-test-results.py | 31 +++++++++++++++++-- .../util/delta-spark-ut/setup-delta.sh | 6 ++-- .github/workflows/velox_backend_x86.yml | 1 - 3 files changed, 32 insertions(+), 6 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index 7a0868ef64a..8b5ad35a7ad 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -76,6 +76,11 @@ # would record zero failing testcases and the regression would be missed. SUITE_ABORTED = "" + +class NoReportsError(RuntimeError): + """Raised when no JUnit elements are found under reports_dir.""" + + SEP = "#" @@ -212,8 +217,12 @@ def parse_reports(reports_dir): failed.add((suite_name, SUITE_ABORTED)) if not parsed_any: - eprint( - "WARNING: no JUnit elements found under {}".format(reports_dir) + raise NoReportsError( + "No JUnit elements found under {}. The test reports are " + "missing or in an unexpected format -- refusing to evaluate the gate " + "on an empty result set (this would otherwise pass silently).".format( + reports_dir + ) ) # A test can't be both passed and failed; failure wins. Skipped only counts @@ -275,7 +284,11 @@ def run_enforce(args): ) return 2 baseline = load_entries(args.known_failures) - passed, failed, skipped = parse_reports(args.reports_dir) + try: + passed, failed, skipped = parse_reports(args.reports_dir) + except NoReportsError as exc: + eprint("ERROR: {}".format(exc)) + return 2 # Always emit this shard's artifacts for the aggregation job. if args.failures_out: @@ -375,6 +388,18 @@ def run_aggregate(args): glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True) ) + # No per-shard gate lists means the artifacts were never produced or the + # download failed (the workflow's download step is continue-on-error). Bail + # out before writing an empty baseline-out, which could otherwise be committed + # and wipe the entire known-failures.txt. + if not failure_files and not ran_files: + eprint( + "ERROR: no per-shard failures-*.txt / ran-*.txt files found under " + "{}. Refusing to aggregate an empty baseline (gate-list artifacts are " + "missing or were not downloaded).".format(args.inputs_dir) + ) + return 2 + union_failed = set() for f in failure_files: union_failed |= load_entries(f) diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 684df319d70..1d9dcf7f954 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -61,10 +61,12 @@ fi echo "::group::Cloning delta-io/delta @ ${DELTA_REF}" # init + shallow fetch resolves a tag, branch OR commit SHA in a single path # (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the -# destructive `rm -rf "$DELTA_DIR"` it required. +# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a +# DELTA_REF starting with `-` can't be misread as a git flag (this script is +# workflow_dispatch-runnable with a user-supplied ref). git init -q "$DELTA_DIR" git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git -git -C "$DELTA_DIR" fetch -q --depth 1 origin "$DELTA_REF" +git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF" git -C "$DELTA_DIR" checkout -q FETCH_HEAD git -C "$DELTA_DIR" --no-pager log -1 --oneline echo "::endgroup::" diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index edb8173d416..a0c3cc8db42 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -126,7 +126,6 @@ jobs: with: native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }} arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }} - secrets: inherit tpc-test-ubuntu: needs: build-native-lib-centos-7 From f2a4f0582c633f2a2aec0943808c2a23893c4e56 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Fri, 3 Jul 2026 20:52:13 +0000 Subject: [PATCH 07/28] [CI] Quarantine flaky Delta DV-merge tests in the UT gate Some Delta-on-Gluten MERGE tests that write deletion vectors fail non-deterministically: the same bundle passes them on one CI run and fails them on the next (native RoaringBitmapArray addSafe aborting on an invalid Long.MAX_VALUE row index). Such tests cannot live in known-failures.txt -- baselining them reds the gate on every run where they pass, and leaving them out reds it on every run where they fail. Add a flaky-tests.txt quarantine list read by the gate. A quarantined test is neutral: it never counts as a regression when it fails nor as now-passing when it passes, and is excluded from the regenerated baseline (aggregate mode). The suite portion of each entry is an fnmatch glob so one line covers a root-cause family across generated suite variants (e.g. *DVs*Suite); the test name is matched exactly. Seed the list with the DV-merge family behind the native row-index bug. This is an interim measure -- entries should be removed once that bug is fixed in the native backend so the tests are enforced again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 2 + .../workflows/util/delta-spark-ut/README.md | 37 +++++++- .../delta-spark-ut/compare-test-results.py | 94 ++++++++++++++++++- .../util/delta-spark-ut/flaky-tests.txt | 32 +++++++ 4 files changed, 155 insertions(+), 10 deletions(-) create mode 100644 .github/workflows/util/delta-spark-ut/flaky-tests.txt diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index dbc210afedf..ce6a2513991 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -590,6 +590,7 @@ jobs: --mode "${GATE_MODE}" \ --reports-dir "$GITHUB_WORKSPACE/delta" \ --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \ + --flaky-tests "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/flaky-tests.txt" \ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \ --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}" @@ -673,6 +674,7 @@ jobs: --mode aggregate \ --inputs-dir gate-lists \ --known-failures .github/workflows/util/delta-spark-ut/known-failures.txt \ + --flaky-tests .github/workflows/util/delta-spark-ut/flaky-tests.txt \ --baseline-out aggregated/known-failures.txt - name: Upload refreshed baseline if: always() diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index ea2cc6af190..148ab5ecd20 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -33,6 +33,7 @@ starts failing** (a regression). | File | Purpose | |---|---| | `known-failures.txt` | Committed baseline: the tests currently expected to fail. One `#` per line. | +| `flaky-tests.txt` | Quarantine list: tests whose pass/fail is non-deterministic. Ignored by the gate whether they pass or fail. `#` per line. | | `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and gates / seeds / aggregates against the baseline. Standard-library only. | | `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches `DeltaSQLCommandTest`. | @@ -48,6 +49,8 @@ Each test shard: - **expected** — failed and in the baseline → ignored. - **now-passing** — in the baseline but passed this run → fails the shard (so the baseline is kept honest), unless `fail_on_fixed=false`. + - **quarantined** — matches an entry in `flaky-tests.txt` → always ignored, + whether it passed or failed (see [Flaky tests](#flaky-tests) below). A final `aggregate` job merges every shard's results into a single, sorted, ready-to-commit `known-failures.txt` artifact and reports **stale** baseline @@ -89,13 +92,36 @@ same way as bootstrapping: run the workflow with `update_baseline=true`, downloa the `delta-spark-ut-known-failures` artifact, and commit it. The aggregate job also lists **stale** entries you can prune. +## Flaky tests + +Some tests are genuinely non-deterministic (e.g. the Delta MERGE-with-deletion-vector +suites that intermittently hit a native row-index bug). Such a test would otherwise +red the gate as a **regression** when it flakes to a failure, or as **now-passing** +when it flakes to a pass — noise either way. + +List these in **`flaky-tests.txt`** to **quarantine** them: the gate ignores a +quarantined test whether it passes or fails, and never writes it into the +regenerated baseline. Format is one `#` per line, `#`-comments +and blank lines allowed: + +``` +# suite portion is an fnmatch glob; test portion is matched exactly. +*DVs*Suite#matched only merge - enabled - with update and delete - isPartitioned: true +``` + +- The **suite** portion is an `fnmatch` glob, so `*DVs*Suite` covers every + generated deletion-vector merge variant in one line. Use the narrowest glob + that still covers the root-cause family. +- The **test** portion is matched **exactly** (test names are freeform and may + contain glob metacharacters), so a same-named test in a non-matching suite is + still gated normally. + +Quarantining is an **interim** measure — it hides a real bug from CI. Each entry +should reference the tracking issue, and be removed once the underlying bug is +fixed so the test is enforced again. + ## Caveats -- **Flaky tests.** A flaky test that usually passes will be flagged as a - regression when it flakes; one that usually fails (and is in the baseline) - may be flagged as now-passing when it happens to pass. Re-run, or set - `fail_on_fixed=false` for that run, and keep genuinely flaky tests out of the - enforced set. - **Known failures still execute** (and fail) — they are gated *after* the run, not skipped — so they still consume CI time. This keeps us decoupled from Delta's sources; skipping them at runtime would require patching Delta. @@ -108,5 +134,6 @@ python3 .github/workflows/util/delta-spark-ut/compare-test-results.py \ --mode enforce \ --reports-dir delta \ --known-failures .github/workflows/util/delta-spark-ut/known-failures.txt \ + --flaky-tests .github/workflows/util/delta-spark-ut/flaky-tests.txt \ --failures-out /tmp/failures.txt --ran-out /tmp/ran.txt ``` diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index 8b5ad35a7ad..d30018a50bb 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -50,6 +50,17 @@ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline entries (tests no longer present in any shard). +Flaky quarantine (``--flaky-tests``) + Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that + only triggers on certain runtime plans), so they are neither a stable pass + nor a stable failure and cannot live in the baseline: baselining them turns + the gate red on every run where they pass, and leaving them out turns it red + on every run where they fail. ``flaky-tests.txt`` quarantines them -- a + quarantined test never counts as a regression (when it fails) nor as + now-passing (when it passes), and is excluded from the regenerated baseline. + Its SUITE is an fnmatch glob (so one line covers a root-cause family across + generated suite variants); its TEST name is matched exactly. + Baseline file format (``known-failures.txt``):: # comment lines start with '#' @@ -65,6 +76,7 @@ """ import argparse +import fnmatch import glob import os import sys @@ -120,6 +132,36 @@ def load_entries(path): return entries +def make_is_flaky(flaky_entries): + """Build a predicate that matches a (suite, test) tuple against flaky entries. + + A flaky entry quarantines a test whose failure is known to be non-deterministic + (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a + single line can cover a root-cause family across generated suite variants + (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches + any suite); the TEST name is matched exactly (test names are freeform and may + contain glob metacharacters, so they are never globbed). + """ + exact = set() + globbed = [] + for suite, test in flaky_entries: + if any(ch in suite for ch in "*?["): + globbed.append((suite, test)) + else: + exact.add((suite, test)) + + def is_flaky(entry): + if entry in exact: + return True + suite, test = entry + for glob_suite, glob_test in globbed: + if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite): + return True + return False + + return is_flaky + + def write_entries(path, entries, header=None): """Write a sorted set of (suite, test) tuples to a file.""" os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) @@ -284,6 +326,7 @@ def run_enforce(args): ) return 2 baseline = load_entries(args.known_failures) + flaky_is = make_is_flaky(load_entries(args.flaky_tests)) try: passed, failed, skipped = parse_reports(args.reports_dir) except NoReportsError as exc: @@ -330,8 +373,13 @@ def run_enforce(args): ) return 0 - regressions = failed - baseline - fixed = baseline & passed + regressions = {e for e in (failed - baseline) if not flaky_is(e)} + quarantined = {e for e in (failed - baseline) if flaky_is(e)} + # `- flaky` on fixed is defensive: flaky tests are excluded from the + # regenerated baseline (aggregate mode), so a flaky test should never be in + # `baseline` in the first place -- but if one slips in, don't let its + # non-deterministic pass trip the now-passing gate. + fixed = {e for e in (baseline & passed) if not flaky_is(e)} expected = failed & baseline write("") @@ -340,6 +388,7 @@ def run_enforce(args): write("| Expected failures (in baseline) | {} |".format(len(expected))) write("| **Regressions (new failures)** | {} |".format(len(regressions))) write("| Now-passing (remove from baseline) | {} |".format(len(fixed))) + write("| Quarantined flaky failures (ignored) | {} |".format(len(quarantined))) _print_block( write, "Regressions -- new failures NOT in the baseline", regressions @@ -352,6 +401,12 @@ def run_enforce(args): "failure) add the lines above to `known-failures.txt`." ) + _print_block( + write, + "Quarantined flaky failures -- ignored (see flaky-tests.txt)", + quarantined, + ) + if args.fail_on_fixed: _print_block( write, "Now-passing -- delete these lines from the baseline", fixed @@ -407,6 +462,13 @@ def run_aggregate(args): for f in ran_files: union_ran |= load_entries(f) + flaky_is = make_is_flaky(load_entries(args.flaky_tests)) + # Exclude quarantined flaky tests from the regenerated baseline: a flaky test + # that happened to fail this run must never be baked into known-failures.txt + # (otherwise it would trip the now-passing gate on the next run where it + # passes). Flaky failures are tracked in flaky-tests.txt, not the baseline. + baseline_body = {e for e in union_failed if not flaky_is(e)} + header = ( "# Known Delta-on-Gluten unit test failures.\n" "#\n" @@ -418,7 +480,7 @@ def run_aggregate(args): "# update_baseline=true and committing the produced artifact.\n" ) if args.baseline_out: - write_entries(args.baseline_out, union_failed, header=header) + write_entries(args.baseline_out, baseline_body, header=header) write, handle = _summary_sink() try: @@ -434,15 +496,30 @@ def run_aggregate(args): if args.known_failures and os.path.exists(args.known_failures): baseline = load_entries(args.known_failures) if baseline: - regressions = union_failed - baseline - fixed = baseline & (union_ran - union_failed) + regressions = {e for e in (union_failed - baseline) if not flaky_is(e)} + quarantined = {e for e in (union_failed - baseline) if flaky_is(e)} + fixed = { + e + for e in (baseline & (union_ran - union_failed)) + if not flaky_is(e) + } stale = baseline - union_ran write("| Baseline entries | {} |".format(len(baseline))) write("| Regressions (global) | {} |".format(len(regressions))) write("| Now-passing (global) | {} |".format(len(fixed))) + write( + "| Quarantined flaky failures (ignored) | {} |".format( + len(quarantined) + ) + ) write("| Stale (not seen this run) | {} |".format(len(stale))) _print_block(write, "Regressions (global)", regressions) _print_block(write, "Now-passing (global)", fixed) + _print_block( + write, + "Quarantined flaky failures -- ignored (see flaky-tests.txt)", + quarantined, + ) _print_block(write, "Stale baseline entries (suite/test gone)", stale) if args.fail_on_regression and regressions: exit_code = 1 @@ -469,6 +546,13 @@ def main(argv=None): parser.add_argument( "--known-failures", help="Path to the committed known-failures.txt baseline." ) + parser.add_argument( + "--flaky-tests", + help="Path to flaky-tests.txt: tests quarantined as non-deterministic. A " + "flaky test is neither counted as a regression when it fails nor as " + "now-passing when it passes, and is excluded from the regenerated baseline " + "(aggregate mode). Optional; omitting it disables quarantining.", + ) parser.add_argument( "--reports-dir", help="Root dir to search for JUnit XML (enforce/seed)." ) diff --git a/.github/workflows/util/delta-spark-ut/flaky-tests.txt b/.github/workflows/util/delta-spark-ut/flaky-tests.txt new file mode 100644 index 00000000000..b56bea43947 --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/flaky-tests.txt @@ -0,0 +1,32 @@ +# Quarantined flaky Delta-on-Gluten tests. +# +# These tests fail NON-DETERMINISTICALLY under the Gluten Velox bundle: the same +# byte-for-byte bundle passes them on one CI run and fails them on the next. They +# therefore cannot live in known-failures.txt -- baselining them turns the gate +# red on every run where they PASS (fail-on-fixed), while leaving them out turns +# it red on every run where they FAIL. The Delta Spark UT (Gluten) gate treats a +# quarantined test as NEUTRAL: it never counts as a regression when it fails, nor +# as now-passing when it passes, and it is excluded from the regenerated baseline. +# +# Format: #. The SUITE part is an fnmatch glob so a +# single line covers a root-cause family across generated suite variants (e.g. +# `*DVs*Suite` = every deletion-vector merge suite); the TEST name is matched +# exactly. Lines starting with '#' are comments. +# +# Prefer fixing the underlying bug and REMOVING the entry over growing this list. +# Every entry should reference a tracking issue for the root cause. +# +# --------------------------------------------------------------------------- +# Root cause: native Delta DV bitmap aggregator aborts on an invalid Long.MAX_VALUE +# (9223372036854775807 > kMaxRepresentableValue) row index during MERGE that writes +# deletion vectors -- RoaringBitmapArray.cpp addSafe, INVALID_STATE VeloxRuntimeError. +# Intermittent (depends on runtime plan/scan/scheduling), so it hits a different +# `*DVs*Suite` MERGE test on each run. Tracked upstream (DV bitmap Long.MAX_VALUE). +# Remove these once the row-index materialization is fixed in the native backend. +# --------------------------------------------------------------------------- +*DVs*Suite#basic case - merge to Delta table by path, isPartitioned: true +*DVs*Suite#extended syntax - conditional update + conditional delete + conditional insert - isPartitioned: true +*DVs*Suite#extended syntax - update + conditional insert - isPartitioned: true +*DVs*Suite#matched only merge - disabled - with update and delete - isPartitioned: true +*DVs*Suite#matched only merge - enabled - with update and delete - isPartitioned: true +*DVs*Suite#single file, isPartitioned: true From b14e1ebc0f485dc288ca8d35be6ea03474d5aad3 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 6 Jul 2026 18:53:03 +0000 Subject: [PATCH 08/28] [CI] Address Delta UT review: single-source spark_version, corrupt-report fail-fast, baseline count Fixes three open review comments on the Delta Spark UT pipeline: 1. spark_version was redundant with the hard-coded GLUTEN_BUNDLE_SPARK_VERSION / GLUTEN_SPARK_PROFILE, and a workflow_dispatch run could set them out of sync so the Delta tests ran against a mismatched Gluten bundle. Make spark_version the single source of truth: the Gluten bundle profile (-Pspark-), the bundle jar name and Delta's -DsparkVersion are all derived from it, so a mismatch is now impossible (no guard needed). Scala 2.13 / JDK 17 stay pinned. 2. Corrupt/truncated JUnit reports (compare-test-results.py). parse_reports previously warned and skipped any XML that failed to parse, so a report truncated by a killed/OOM'd fork could silently drop a suite's failures and let the gate go green on partial data. A TEST-*.xml that fails to parse now raises CorruptReportError (exit 2); other XML matched by the broad target/** glob is still skipped. 3. Hard-coded failure count in the baseline header (known-failures.txt). The header stated a fixed total that drifts every time the baseline is refreshed. Drop the number and point at the entry count / delta-spark-aggregate summary as the authoritative source instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 16 +++++++++++----- .../delta-spark-ut/compare-test-results.py | 19 +++++++++++++++++++ .../util/delta-spark-ut/known-failures.txt | 4 +++- 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index ce6a2513991..689312d2b69 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -57,6 +57,7 @@ on: required: false default: 'v4.2.0' spark_version: + description: 'Spark version driving both the Gluten bundle profile (-Pspark-) and Delta -DsparkVersion.' type: string required: false default: '4.1' @@ -79,7 +80,7 @@ on: required: true default: 'v4.2.0' spark_version: - description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)' + description: 'Spark version: drives the Gluten bundle profile (-Pspark-) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.' required: true default: '4.1' test_parallelism: @@ -102,12 +103,17 @@ env: MVN_CMD: 'build/mvn -ntp' CCACHE_DIR: "${{ github.workspace }}/.ccache" # Gluten profile / bundle naming for the build-gluten-bundle and - # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's - # default Spark version (4.1.0) from project/CrossSparkVersions.scala. - GLUTEN_SPARK_PROFILE: 'spark-4.1' + # delta-spark-test jobs. `spark_version` is the single source of truth for the + # Spark version: it drives the Gluten bundle profile (-Pspark-), the bundle + # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle + # built for the same Spark version (no separate value to keep in sync). Scala + # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from + # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a + # compatible delta_ref. + GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }} GLUTEN_SCALA_PROFILE: 'scala-2.13' GLUTEN_JAVA_PROFILE: 'java-17' - GLUTEN_BUNDLE_SPARK_VERSION: '4.1' + GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }} GLUTEN_BUNDLE_SCALA_VERSION: '2.13' DELTA_SCALA_VERSION: '2.13.16' # Number of shards in the delta-spark-test matrix. Must equal the length of diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index d30018a50bb..0247aaabf62 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -93,6 +93,15 @@ class NoReportsError(RuntimeError): """Raised when no JUnit elements are found under reports_dir.""" +class CorruptReportError(NoReportsError): + """Raised when an expected JUnit report file (TEST-*.xml) fails to parse. + + Subclasses NoReportsError so the enforce/seed handler treats a truncated + report as a hard data error (exit 2) instead of silently dropping the + suite's results and letting the gate pass on partial data. + """ + + SEP = "#" @@ -220,6 +229,16 @@ def parse_reports(reports_dir): try: tree = ET.parse(xml_file) except ET.ParseError as exc: + # A TEST-*.xml that fails to parse is almost always a report truncated + # when a forked test JVM was killed mid-write (e.g. OOM). Silently + # skipping it drops that suite's results and could let the gate go + # green on partial data, so fail hard for report files. Other XML that + # merely matched the broad `target/**` glob is still skipped. + if os.path.basename(xml_file).startswith("TEST-"): + raise CorruptReportError( + "corrupt or truncated JUnit report {}: {}. Refusing to " + "evaluate the gate on partial data.".format(xml_file, exc) + ) eprint("WARNING: could not parse {}: {}".format(xml_file, exc)) continue root = tree.getroot() diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index 18487b7e75e..624e0c5cf46 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -9,7 +9,9 @@ # --------------------------------------------------------------------------- # Baseline for the committed 4-shard x 4-fork config. Originally seeded from run # 27490052632; refreshed from run 28318129710 (4 x 4) after the FileSourceScanLike -# test fixes (delta-io/delta #7104 + #7105). 810 known failures total. +# test fixes (delta-io/delta #7104 + #7105). The authoritative failure count is +# the number of entries below (and the delta-spark-aggregate job summary); it is +# deliberately not restated here so it can't drift as the baseline is refreshed. # # IMPORTANT: regenerate this baseline under the SAME NUM_SHARDS x # TEST_PARALLELISM_COUNT the gate runs (here 4 x 4). ~34 of these failures are From 20a9e7540d1f1e3837e04181efecaed15b0e40c2 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 7 Jul 2026 01:48:17 +0000 Subject: [PATCH 09/28] [CI] Fail Delta UT aggregate when a shard's gate lists are missing The aggregate job downloads per-shard gate lists with continue-on-error and the aggregator only hard-failed when *no* inputs were found. If a shard died before writing its gate lists (OOM / watchdog) or its artifact failed to download, the job still regenerated a baseline that silently omitted that shard's failures, shrinking known-failures.txt and reddening the next run. Add --expected-shards to compare-test-results.py: in aggregate mode it counts shards that produced a complete failures-*/ran-* pair and refuses to aggregate (exit 2, before writing --baseline-out) when fewer than expected are present. A shard counts only when BOTH files exist, so a partial download is caught too. The workflow passes --expected-shards ${{ env.DELTA_NUM_SHARDS }}; omitting the flag (default 0) disables the check, preserving prior behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 1 + .../workflows/util/delta-spark-ut/README.md | 6 ++ .../delta-spark-ut/compare-test-results.py | 56 ++++++++++++++++++- 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 689312d2b69..0ffcdca9ed9 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -679,6 +679,7 @@ jobs: python3 .github/workflows/util/delta-spark-ut/compare-test-results.py \ --mode aggregate \ --inputs-dir gate-lists \ + --expected-shards "${{ env.DELTA_NUM_SHARDS }}" \ --known-failures .github/workflows/util/delta-spark-ut/known-failures.txt \ --flaky-tests .github/workflows/util/delta-spark-ut/flaky-tests.txt \ --baseline-out aggregated/known-failures.txt diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 148ab5ecd20..5b4d65e9c63 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -92,6 +92,12 @@ same way as bootstrapping: run the workflow with `update_baseline=true`, downloa the `delta-spark-ut-known-failures` artifact, and commit it. The aggregate job also lists **stale** entries you can prune. +The aggregate job passes `--expected-shards` (the shard count), so if a shard +dies before writing its gate lists (or its artifact fails to download) the +aggregate **fails** instead of emitting a baseline that silently omits that +shard's failures — which would otherwise shrink `known-failures.txt` and red the +next run. Re-run the workflow if this happens. + ## Flaky tests Some tests are genuinely non-deterministic (e.g. the Delta MERGE-with-deletion-vector diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index 0247aaabf62..d1086dca7d9 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -48,7 +48,9 @@ ``aggregate`` (final job) Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single, sorted, ready-to-commit ``known-failures.txt`` and report stale baseline - entries (tests no longer present in any shard). + entries (tests no longer present in any shard). Pass ``--expected-shards N`` + to fail when fewer than ``N`` shards contributed gate lists (a shard that + died before writing them), so an incomplete baseline is never produced. Flaky quarantine (``--flaky-tests``) Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that @@ -79,6 +81,7 @@ import fnmatch import glob import os +import re import sys import xml.etree.ElementTree as ET @@ -454,6 +457,23 @@ def run_enforce(args): handle.close() +def _shard_ids(files, prefix): + """Return the set of shard ids from gate-list filenames. + + Gate lists are named ``.txt`` (e.g. ``failures-shard-0.txt``, + ``ran-shard-0.txt``); this extracts the ```` token so aggregate mode + can count how many shards contributed. Matching is on the basename, so nested + download dirs are fine. + """ + ids = set() + pat = re.compile(r"^" + re.escape(prefix) + r"(.+)\.txt$") + for f in files: + m = pat.match(os.path.basename(f)) + if m: + ids.add(m.group(1)) + return ids + + def run_aggregate(args): failure_files = sorted( glob.glob(os.path.join(args.inputs_dir, "**", "failures-*.txt"), recursive=True) @@ -474,6 +494,31 @@ def run_aggregate(args): ) return 2 + # Completeness guard: each shard writes its failures-.txt and ran-.txt + # together (see run_enforce), so a shard that died before the gate step -- or + # whose artifact failed to download -- contributes neither. The empty-inputs + # check above only catches losing *all* shards; without this a partial set + # would silently regenerate a baseline missing that shard's failures, wrongly + # shrinking known-failures.txt and reddening the next run. A shard counts as + # complete only when BOTH its files are present (robust to partial downloads). + if args.expected_shards: + complete = _shard_ids(failure_files, "failures-") & _shard_ids( + ran_files, "ran-" + ) + if len(complete) != args.expected_shards: + eprint( + "ERROR: expected {} shard gate-list set(s) but found {} complete " + "(shards: {}). A shard's failures-*/ran-* artifact is missing -- it " + "likely died before writing its gate lists or its artifact failed " + "to download -- so the regenerated baseline would be incomplete and " + "could wrongly shrink known-failures.txt. Refusing to aggregate.".format( + args.expected_shards, + len(complete), + ", ".join(sorted(complete)) or "none", + ) + ) + return 2 + union_failed = set() for f in failure_files: union_failed |= load_entries(f) @@ -590,6 +635,15 @@ def main(argv=None): parser.add_argument( "--inputs-dir", help="Dir with per-shard failures-*/ran-* files (aggregate)." ) + parser.add_argument( + "--expected-shards", + type=int, + default=0, + help="In aggregate mode, the number of shards expected to contribute gate " + "lists. When >0, fail if fewer complete shard gate-list pairs " + "(failures-*/ran-*) are found -- e.g. a shard died before writing them -- so " + "an incomplete baseline is never produced. 0 (default) disables the check.", + ) parser.add_argument( "--baseline-out", help="Write the merged baseline here (aggregate)." ) From 412555f15e62335817d1b5771eaf4e64e72f80b7 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 9 Jul 2026 07:13:42 +0000 Subject: [PATCH 10/28] [CI] Slim Delta UT pipeline: drop redundant Arrow jars, extract shard body Two readability/cleanup changes to the Delta Spark UT pipeline: 1. Drop redundant Arrow jar handling. Since apache/gluten#12244 (2026-06-09) Gluten depends on vanilla Apache Arrow instead of the custom 15.0.0-gluten rename, so the Arrow jars resolve from Maven Central. Remove the arrow-jar staging in both native builds, the velox-arrow-jars / delta-spark-ut-arrow-jars uploads, the caller's arrow_jars_artifact input, and the bundle job's "Download Arrow jars" step. The bundle build's Maven cache still holds Arrow across runs and resolves it from Central on a miss. 2. Extract the shard test body into run-delta-tests.sh. The "Run Delta spark module tests" step embedded ~190 lines of shell (hang watchdog, sbt invocation with JVM/heap tuning, cgroup memory forensics, report check, known-failures gate). Move that whole body into util/delta-spark-ut/run-delta-tests.sh so the workflow step is just an `env:` block plus a one-line script call, shrinking delta_spark_ut.yml by ~180 lines. The move is faithful: the script body is the previous inline block with only the GitHub `${{ }}` expressions replaced by env vars (matrix.shard -> SHARD_ID which is already job-level env; spark_version/update_baseline/fail_on_fixed added to the step env). Verified by a textual diff against the old body and by end-to-end runs with a stubbed sbt (baseline-only -> green, regression -> red, seed -> green). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 230 +---------------- .../workflows/util/delta-spark-ut/README.md | 1 + .../util/delta-spark-ut/run-delta-tests.sh | 234 ++++++++++++++++++ .github/workflows/velox_backend_x86.yml | 19 +- 4 files changed, 251 insertions(+), 233 deletions(-) create mode 100755 .github/workflows/util/delta-spark-ut/run-delta-tests.sh diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 0ffcdca9ed9..32919cb785c 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -33,10 +33,10 @@ name: Delta Spark UT (Gluten) on: # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant - # changes) and passes the native-lib + arrow-jars artifacts it already built, - # so the expensive native C++ build is NOT duplicated. Those artifacts live in - # the CALLER's run (a called workflow runs as part of the caller run), so the - # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`. + # changes) and passes the native-lib artifact it already built, so the expensive + # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a + # called workflow runs as part of the caller run), so the jobs below download it + # by name. See velox_backend_x86.yml `delta-spark-ut`. # # NOTE: the `pull_request` trigger was removed so this no longer runs as its own # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml @@ -48,10 +48,6 @@ on: description: 'Name of the cpp/build artifact uploaded by the caller' type: string required: true - arrow_jars_artifact: - description: 'Name of the org.apache.arrow jars artifact uploaded by the caller' - type: string - required: true delta_ref: type: string required: false @@ -138,8 +134,8 @@ env: jobs: build-native-lib-centos-7: # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml - # the caller already built the native lib + arrow jars and passes them as - # inputs, so this job is skipped and the duplicate native build is avoided. + # the caller already built the native lib and passes it as an input, so this + # job is skipped and the duplicate native build is avoided. if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-22.04 steps: @@ -165,8 +161,6 @@ jobs: ccache -sz bash dev/ci-velox-buildstatic-centos-7.sh ccache -s - mkdir -p /work/.m2/repository/org/apache/arrow/ - cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/ " - name: Save Ccache if: always() @@ -179,11 +173,6 @@ jobs: name: delta-spark-ut-native-lib-centos-7-${{github.sha}} path: ./cpp/build/ if-no-files-found: error - - uses: actions/upload-artifact@v4 - with: - name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}} - path: .m2/repository/org/apache/arrow/ - if-no-files-found: error build-gluten-bundle: needs: build-native-lib-centos-7 @@ -207,11 +196,6 @@ jobs: restore-keys: | m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}- m2-delta-spark-ut-bundle- - - name: Download Arrow jars - uses: actions/download-artifact@v4 - with: - name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }} - path: /root/.m2/repository/org/apache/arrow/ - name: Build Gluten Velox + Delta bundle run: | set -euo pipefail @@ -356,6 +340,9 @@ jobs: env: NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }} TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }} + SPARK_VERSION: ${{ steps.resolve.outputs.spark_version }} + UPDATE_BASELINE: ${{ steps.resolve.outputs.update_baseline }} + FAIL_ON_FIXED: ${{ steps.resolve.outputs.fail_on_fixed }} # Required by Delta to enable testing-only code paths # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1"). DELTA_TESTING: '1' @@ -404,202 +391,9 @@ jobs: -Dfile.encoding=UTF-8 run: | set -euo pipefail - export JAVA_HOME=/usr/lib/jvm/java-17-openjdk - export PATH=$JAVA_HOME/bin:$PATH - cd "$GITHUB_WORKSPACE/delta" - chmod +x build/sbt - # Only run the unified `spark` sbt project, NOT `sparkGroup/test` -- - # `sparkGroup` aggregates many other projects (sparkV2, contribs, - # sharing, connect*, ...) that are out of scope for this pipeline. - # - # JVM heap layout -- two memory consumers on the ~16G runner: - # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to - # return idle memory during the (long) test phase via G1 periodic GC - # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the - # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent - # forces each periodic GC to a full STW collection) that uncommits to a - # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low - # -Xms512m floor. - # Without this the idle launcher holds ~5.3G for the whole run; with - # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime - # config, so they cannot affect the measured pass/fail signal. - # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command - # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends - # so our -Xmx2G comes last and wins. Gluten offloads data to Velox - # off-heap (capped at 2g via spark.memory.offHeap.size in the patched - # DeltaSQLCommandTest), so the fork's heap need is modest. A larger - # fork heap pushed the cgroup peak past the ~16G OOM threshold and the - # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G - # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is - # analyzable. - # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so - # every suite writes per-test results. Delta itself only configures - # the console reporter (-oDF), so without this we'd have no machine- - # readable results to gate on. The path is relative to the forked - # test JVM's working dir (Test / baseDirectory = spark/), i.e. - # delta/spark/target/test-reports/TEST-*.xml. - # - # We deliberately do NOT let an sbt non-zero exit (which fires on the - # MANY expected Delta-on-Gluten failures) fail this step directly. - # Instead the known-failures gate below decides pass/fail: the build - # is green when the only failures are ones already recorded in the - # baseline, and red on a genuine regression. - set +e - # --- hang watchdog --------------------------------------------------- - # Shard 2 (and occasionally others) hangs indefinitely after a suite's - # last test with no further output. ScalaTest's failAfter only wraps - # individual test BODIES, so a wedge in suite teardown/afterAll -- or in - # a non-interruptible native Velox/JNI call that ignores - # Thread.interrupt() -- has no timeout and stalls until the 350-min job - # limit with zero diagnostics. This watchdog dumps the forked test JVM's - # threads (to the job log, and to a file for the artifact) once the test - # output has been silent for too long, so the deadlock is diagnosable. - SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log" - : > "$SBT_LOG" - rm -f /tmp/sbt-done - ( - # CRITICAL: the step shell runs with `bash -eo pipefail`, which the - # subshell inherits. Without `set +e` here, ANY non-zero command -- - # e.g. fork detection finding no match, or `kill`/`jps` returning - # non-zero -- silently kills this watchdog. That errexit kill (plus a - # /proc detection miss) once made the watchdog capture ZERO dumps. A - # diagnostic must never abort on a failed probe. - set +e +o pipefail - JSTACK="${JAVA_HOME}/bin/jstack" - JPS="${JAVA_HOME}/bin/jps" - silent_limit=900 # 15 min with no new test output => treat as hung - dumps=0 - fork_pids() { - # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads - # the main class from hsperfdata, robust to sbt's @argfile launch); - # fall back to scanning /proc cmdline + @argfile. - "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}' - local p cl arg - for p in /proc/[0-9]*; do - [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue - cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)" - case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac - arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)" - [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \ - && echo "${p##*/}" - done - } - all_java_pids() { - "$JPS" -q 2>/dev/null - local p - for p in /proc/[0-9]*; do - [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}" - done - } - echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence" - hb=0 - while [ ! -f /tmp/sbt-done ]; do - sleep 60 - [ -f "$SBT_LOG" ] || continue - now=$(date +%s) - mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now") - silent=$(( now - mtime )) - # Per-minute memory profile: heap tuning proved the ~16G OOM peak is - # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows - # toward it -- the last lines before a hang reveal the real hog to cut. - # Read /proc directly (no `ps` dependency in the minimal container). - memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null) - jvmrss="" - for mp in $(all_java_pids 2>/dev/null | sort -un); do - r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null) - [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)" - done - echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]" - hb=$(( hb + 1 )) - # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how - # long the test has been silent) without waiting for a hang. - [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago" - if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then - dumps=$(( dumps + 1 )) - pids="$(fork_pids | sort -un)" - # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM. - [ -n "$pids" ] || pids="$(all_java_pids | sort -un)" - echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))" - [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump" - for pid in $pids; do - # SIGQUIT makes the JVM print a full thread dump to its OWN stderr, - # which sbt relays into the test log via the SAME stream as test - # output -- so it lands in the job log even when a separately - # spawned jstack child's output would be buffered/lost. Also write - # jstack to a file for the per-shard artifact. - echo "----- SIGQUIT + jstack pid ${pid} -----" - kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}" - timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \ - || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}" - done - echo "::endgroup::" - # The dump is now captured (job log via SIGQUIT + artifact via - # jstack file). A hung JVM otherwise stalls the whole shard until - # the 350-min job timeout AND keeps the job log frozen so the dump - # never becomes reachable. So KILL the wedged JVM(s): the suite - # fails fast (acceptable -- errors are expected; only an - # unrecoverable hang blocks CI), the job proceeds/ends, and the log - # + artifacts flush. Give SIGQUIT a moment to print first. - sleep 20 - echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)" - for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done - fi - done - ) & - WATCHDOG_PID=$! - - ./build/sbt \ - -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \ - -v \ - -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \ - -J-XX:G1PeriodicGCInterval=10000 \ - -J-XX:G1PeriodicGCSystemLoadThreshold=0 \ - -J-XX:-G1PeriodicGCInvokesConcurrent \ - -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \ - "++ ${DELTA_SCALA_VERSION}" \ - 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \ - 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \ - "spark/test" 2>&1 | tee "$SBT_LOG" - SBT_EXIT=${PIPESTATUS[0]} - touch /tmp/sbt-done - kill "$WATCHDOG_PID" 2>/dev/null || true - set -e - echo "sbt spark/test exited with ${SBT_EXIT}" - - # Memory forensics: a sudden forked-JVM death with no hs_err and no heap - # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM - # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill - # count so we can confirm/measure it (cgroup v2 paths; best-effort). - ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ===" - for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \ - /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do - [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; } - done ) || true - - # A compile/launch failure leaves no reports at all. In that case the - # gate would see zero failures and pass spuriously, so fail loudly. - REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true) - echo "Found ${REPORT_COUNT} JUnit XML report file(s)." - if [ "${REPORT_COUNT}" -eq 0 ]; then - echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures." - exit 1 - fi - - # update_baseline=true -> SEED mode (record failures, never fail) so the - # baseline can be (re)generated. Otherwise ENFORCE against the baseline. - GATE_MODE=enforce - if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then - GATE_MODE=seed - fi - mkdir -p "$GITHUB_WORKSPACE/gate-out" - python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \ - --mode "${GATE_MODE}" \ - --reports-dir "$GITHUB_WORKSPACE/delta" \ - --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \ - --flaky-tests "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/flaky-tests.txt" \ - --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \ - --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \ - --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}" + # Run the shard's Delta tests + hang watchdog + memory forensics, then + # gate against the baseline. See util/delta-spark-ut/run-delta-tests.sh. + bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/run-delta-tests.sh" - name: Compress heap dumps (if any) if: ${{ failure() }} diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 5b4d65e9c63..7a40c610486 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -35,6 +35,7 @@ starts failing** (a regression). | `known-failures.txt` | Committed baseline: the tests currently expected to fail. One `#` per line. | | `flaky-tests.txt` | Quarantine list: tests whose pass/fail is non-deterministic. Ignored by the gate whether they pass or fail. `#` per line. | | `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and gates / seeds / aggregates against the baseline. Standard-library only. | +| `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the results against the baseline via `compare-test-results.py`. | | `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches `DeltaSQLCommandTest`. | ## How the gate works diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh new file mode 100755 index 00000000000..9d3e8981032 --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -0,0 +1,234 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Runs the Delta `spark` module tests for one shard under the Gluten bundle: +# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt +# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and +# then gates the results against the baseline (compare-test-results.py). Extracted +# from delta_spark_ut.yml so the workflow step stays readable. +# +# Driven by environment (set by the workflow step / job): +# SHARD_ID - this shard's id (matrix.shard) +# SPARK_VERSION - Delta -DsparkVersion value +# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce +# FAIL_ON_FIXED - passed through to the gate +# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING, +# JAVA_TOOL_OPTIONS - test env (see the workflow step's `env:` block) +# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts) +# + +set -euo pipefail +export JAVA_HOME=/usr/lib/jvm/java-17-openjdk +export PATH=$JAVA_HOME/bin:$PATH +cd "$GITHUB_WORKSPACE/delta" +chmod +x build/sbt +# Only run the unified `spark` sbt project, NOT `sparkGroup/test` -- +# `sparkGroup` aggregates many other projects (sparkV2, contribs, +# sharing, connect*, ...) that are out of scope for this pipeline. +# +# JVM heap layout -- two memory consumers on the ~16G runner: +# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to +# return idle memory during the (long) test phase via G1 periodic GC +# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the +# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent +# forces each periodic GC to a full STW collection) that uncommits to a +# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low +# -Xms512m floor. +# Without this the idle launcher holds ~5.3G for the whole run; with +# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime +# config, so they cannot affect the measured pass/fail signal. +# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command +# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends +# so our -Xmx2G comes last and wins. Gluten offloads data to Velox +# off-heap (capped at 2g via spark.memory.offHeap.size in the patched +# DeltaSQLCommandTest), so the fork's heap need is modest. A larger +# fork heap pushed the cgroup peak past the ~16G OOM threshold and the +# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G +# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is +# analyzable. +# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so +# every suite writes per-test results. Delta itself only configures +# the console reporter (-oDF), so without this we'd have no machine- +# readable results to gate on. The path is relative to the forked +# test JVM's working dir (Test / baseDirectory = spark/), i.e. +# delta/spark/target/test-reports/TEST-*.xml. +# +# We deliberately do NOT let an sbt non-zero exit (which fires on the +# MANY expected Delta-on-Gluten failures) fail this step directly. +# Instead the known-failures gate below decides pass/fail: the build +# is green when the only failures are ones already recorded in the +# baseline, and red on a genuine regression. +set +e +# --- hang watchdog --------------------------------------------------- +# Shard 2 (and occasionally others) hangs indefinitely after a suite's +# last test with no further output. ScalaTest's failAfter only wraps +# individual test BODIES, so a wedge in suite teardown/afterAll -- or in +# a non-interruptible native Velox/JNI call that ignores +# Thread.interrupt() -- has no timeout and stalls until the 350-min job +# limit with zero diagnostics. This watchdog dumps the forked test JVM's +# threads (to the job log, and to a file for the artifact) once the test +# output has been silent for too long, so the deadlock is diagnosable. +SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log" +: > "$SBT_LOG" +rm -f /tmp/sbt-done +( + # CRITICAL: the step shell runs with `bash -eo pipefail`, which the + # subshell inherits. Without `set +e` here, ANY non-zero command -- + # e.g. fork detection finding no match, or `kill`/`jps` returning + # non-zero -- silently kills this watchdog. That errexit kill (plus a + # /proc detection miss) once made the watchdog capture ZERO dumps. A + # diagnostic must never abort on a failed probe. + set +e +o pipefail + JSTACK="${JAVA_HOME}/bin/jstack" + JPS="${JAVA_HOME}/bin/jps" + silent_limit=900 # 15 min with no new test output => treat as hung + dumps=0 + fork_pids() { + # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads + # the main class from hsperfdata, robust to sbt's @argfile launch); + # fall back to scanning /proc cmdline + @argfile. + "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}' + local p cl arg + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue + cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)" + case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac + arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)" + [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \ + && echo "${p##*/}" + done + } + all_java_pids() { + "$JPS" -q 2>/dev/null + local p + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}" + done + } + echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence" + hb=0 + while [ ! -f /tmp/sbt-done ]; do + sleep 60 + [ -f "$SBT_LOG" ] || continue + now=$(date +%s) + mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now") + silent=$(( now - mtime )) + # Per-minute memory profile: heap tuning proved the ~16G OOM peak is + # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows + # toward it -- the last lines before a hang reveal the real hog to cut. + # Read /proc directly (no `ps` dependency in the minimal container). + memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null) + jvmrss="" + for mp in $(all_java_pids 2>/dev/null | sort -un); do + r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null) + [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)" + done + echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]" + hb=$(( hb + 1 )) + # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how + # long the test has been silent) without waiting for a hang. + [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago" + if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then + dumps=$(( dumps + 1 )) + pids="$(fork_pids | sort -un)" + # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM. + [ -n "$pids" ] || pids="$(all_java_pids | sort -un)" + echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))" + [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump" + for pid in $pids; do + # SIGQUIT makes the JVM print a full thread dump to its OWN stderr, + # which sbt relays into the test log via the SAME stream as test + # output -- so it lands in the job log even when a separately + # spawned jstack child's output would be buffered/lost. Also write + # jstack to a file for the per-shard artifact. + echo "----- SIGQUIT + jstack pid ${pid} -----" + kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}" + timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${SHARD_ID}-${dumps}-${pid}.txt" 2>&1 \ + || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}" + done + echo "::endgroup::" + # The dump is now captured (job log via SIGQUIT + artifact via + # jstack file). A hung JVM otherwise stalls the whole shard until + # the 350-min job timeout AND keeps the job log frozen so the dump + # never becomes reachable. So KILL the wedged JVM(s): the suite + # fails fast (acceptable -- errors are expected; only an + # unrecoverable hang blocks CI), the job proceeds/ends, and the log + # + artifacts flush. Give SIGQUIT a moment to print first. + sleep 20 + echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)" + for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done + fi + done +) & +WATCHDOG_PID=$! + +./build/sbt \ + -DsparkVersion=${SPARK_VERSION} \ + -v \ + -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \ + -J-XX:G1PeriodicGCInterval=10000 \ + -J-XX:G1PeriodicGCSystemLoadThreshold=0 \ + -J-XX:-G1PeriodicGCInvokesConcurrent \ + -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \ + "++ ${DELTA_SCALA_VERSION}" \ + 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \ + 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \ + "spark/test" 2>&1 | tee "$SBT_LOG" +SBT_EXIT=${PIPESTATUS[0]} +touch /tmp/sbt-done +kill "$WATCHDOG_PID" 2>/dev/null || true +set -e +echo "sbt spark/test exited with ${SBT_EXIT}" + +# Memory forensics: a sudden forked-JVM death with no hs_err and no heap +# dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM +# heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill +# count so we can confirm/measure it (cgroup v2 paths; best-effort). +( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ===" + for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \ + /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do + [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; } + done ) || true + +# A compile/launch failure leaves no reports at all. In that case the +# gate would see zero failures and pass spuriously, so fail loudly. +REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true) +echo "Found ${REPORT_COUNT} JUnit XML report file(s)." +if [ "${REPORT_COUNT}" -eq 0 ]; then + echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures." + exit 1 +fi + +# Classify this shard's results against the baseline: seed mode when +# UPDATE_BASELINE=true (record failures, never fail) so the baseline can be +# (re)generated; otherwise enforce against it. Writes this shard's gate-out/*.txt +# for the aggregate job. +UTIL_DIR="$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut" +GATE_MODE=enforce +if [ "${UPDATE_BASELINE}" = "true" ]; then + GATE_MODE=seed +fi +mkdir -p "$GITHUB_WORKSPACE/gate-out" +python3 "$UTIL_DIR/compare-test-results.py" \ + --mode "$GATE_MODE" \ + --reports-dir "$GITHUB_WORKSPACE/delta" \ + --known-failures "$UTIL_DIR/known-failures.txt" \ + --flaky-tests "$UTIL_DIR/flaky-tests.txt" \ + --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${SHARD_ID}.txt" \ + --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${SHARD_ID}.txt" \ + --fail-on-fixed "${FAIL_ON_FIXED}" diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index a0c3cc8db42..de701db37f3 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -91,10 +91,6 @@ jobs: ccache -sz bash dev/ci-velox-buildstatic-centos-7.sh ccache -s - # Stage the custom-built org.apache.arrow jars so the reusable Delta - # workflow's bundle build can consume them (see delta-spark-ut job). - mkdir -p /work/.m2/repository/org/apache/arrow/ - cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/ " - name: "Save ccache" @@ -109,23 +105,16 @@ jobs: name: velox-native-lib-centos-7-${{github.sha}} path: ./cpp/build/ if-no-files-found: error - # Consumed by the reusable Delta workflow (delta-spark-ut job). - - uses: actions/upload-artifact@v4 - with: - name: velox-arrow-jars-centos-7-${{github.sha}} - path: .m2/repository/org/apache/arrow/ - if-no-files-found: error - # Delta Spark UT, run via the reusable workflow so it reuses the native lib + - # arrow jars built above instead of duplicating the native build. Not gated on - # Delta-only paths: core/velox/substrait/cpp/shims changes can affect Delta - # query offload, so this runs on every trigger like the other spark-test jobs. + # Delta Spark UT, run via the reusable workflow so it reuses the native lib + # built above instead of duplicating the native build. Not gated on Delta-only + # paths: core/velox/substrait/cpp/shims changes can affect Delta query offload, + # so this runs on every trigger like the other spark-test jobs. delta-spark-ut: needs: build-native-lib-centos-7 uses: ./.github/workflows/delta_spark_ut.yml with: native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }} - arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }} tpc-test-ubuntu: needs: build-native-lib-centos-7 From 0455656aaf9f1e591202e58e28cba0356f92910a Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 9 Jul 2026 07:33:43 +0000 Subject: [PATCH 11/28] [CI] Share Delta test JVM flags via java-test-args.sh for local runs The Gluten/JDK17 test JVM flags (--add-opens + the Netty reflection property, mirroring root pom.xml's extraJavaTestArgs) lived only in the workflow step's env: block, so a developer running the Delta suite locally had no shared source for them. Move them into util/delta-spark-ut/java-test-args.sh, which exports JAVA_TOOL_OPTIONS. run-delta-tests.sh sources it (via a BASH_SOURCE- relative path so it also works for local runs), and the workflow env block no longer sets JAVA_TOOL_OPTIONS. A developer can source the same file before `sbt spark/test` to get identical flags; README documents it. Faithful: the sourced value is byte-identical to the previous env-block value (verified), and an end-to-end run with a stubbed sbt confirms the forked test JVM still receives all 18 flags with JAVA_TOOL_OPTIONS unset in the environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/delta_spark_ut.yml | 46 +------------- .../workflows/util/delta-spark-ut/README.md | 17 ++++++ .../util/delta-spark-ut/java-test-args.sh | 60 +++++++++++++++++++ .../util/delta-spark-ut/run-delta-tests.sh | 9 ++- 4 files changed, 87 insertions(+), 45 deletions(-) create mode 100755 .github/workflows/util/delta-spark-ut/java-test-args.sh diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 32919cb785c..ad5e74d860f 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -346,49 +346,9 @@ jobs: # Required by Delta to enable testing-only code paths # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1"). DELTA_TESTING: '1' - # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the - # `io.netty.tryReflectionSetAccessible` system property; otherwise - # the forked test JVM fails with - # java.lang.UnsupportedOperationException: sun.misc.Unsafe or - # java.nio.DirectByteBuffer.(long, int) not available - # as soon as Gluten's bundled Arrow allocator initializes Netty - # direct buffers. Delta's own `Test / javaOptions` (see - # project/CrossSparkVersions.scala `java17TestSettings`) sets the - # base add-opens but NOT the Netty property -- Delta's own tests - # don't load Arrow/Netty buffers in a way that triggers it. - # - # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt - # launcher JVM and the forked test JVM (sbt forks tests and the - # child inherits the parent's env). The set below mirrors - # `extraJavaTestArgs` from Gluten's own root pom.xml (the - # canonical Gluten test JVM flag set). - # - # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS - # is processed BEFORE the JVM command line, so Delta's explicit - # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still - # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via - # an sbt `set spark / Test / javaOptions ++= ...` command below, - # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER - # `-Xmx1024m` and wins. - JAVA_TOOL_OPTIONS: >- - -XX:+IgnoreUnrecognizedVMOptions - --add-opens=java.base/java.lang=ALL-UNNAMED - --add-opens=java.base/java.lang.invoke=ALL-UNNAMED - --add-opens=java.base/java.lang.reflect=ALL-UNNAMED - --add-opens=java.base/java.io=ALL-UNNAMED - --add-opens=java.base/java.net=ALL-UNNAMED - --add-opens=java.base/java.nio=ALL-UNNAMED - --add-opens=java.base/java.util=ALL-UNNAMED - --add-opens=java.base/java.util.concurrent=ALL-UNNAMED - --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED - --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED - --add-opens=java.base/sun.nio.ch=ALL-UNNAMED - --add-opens=java.base/sun.nio.cs=ALL-UNNAMED - --add-opens=java.base/sun.security.action=ALL-UNNAMED - --add-opens=java.base/sun.util.calendar=ALL-UNNAMED - -Djdk.reflect.useDirectMethodHandle=false - -Dio.netty.tryReflectionSetAccessible=true - -Dfile.encoding=UTF-8 + # NOTE: the Gluten/JDK17 test JVM flags (JAVA_TOOL_OPTIONS) live in + # util/delta-spark-ut/java-test-args.sh, which run-delta-tests.sh sources, + # so CI and local dev runs share one definition. run: | set -euo pipefail # Run the shard's Delta tests + hang watchdog + memory forensics, then diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 7a40c610486..dc962c75ba4 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -36,6 +36,7 @@ starts failing** (a regression). | `flaky-tests.txt` | Quarantine list: tests whose pass/fail is non-deterministic. Ignored by the gate whether they pass or fail. `#` per line. | | `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and gates / seeds / aggregates against the baseline. Standard-library only. | | `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the results against the baseline via `compare-test-results.py`. | +| `java-test-args.sh` | Shared JVM flags (`--add-opens` + Netty property) needed to run the suite on JDK 17 with the Gluten bundle. Sourced by `run-delta-tests.sh` and by local runs. | | `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches `DeltaSQLCommandTest`. | ## How the gate works @@ -144,3 +145,19 @@ python3 .github/workflows/util/delta-spark-ut/compare-test-results.py \ --flaky-tests .github/workflows/util/delta-spark-ut/flaky-tests.txt \ --failures-out /tmp/failures.txt --ran-out /tmp/ran.txt ``` + +## Running the suite locally + +`sbt spark/test` needs extra JDK-17 JVM flags to run the Delta suite against the +Gluten bundle (`--add-opens` + the Netty reflection property). CI and local runs +share one definition in `java-test-args.sh` — `source` it before invoking sbt so +the flags reach the sbt launcher and the forked test JVM: + +```bash +# from the Delta clone prepared by setup-delta.sh (which has the Gluten bundle): +source /.github/workflows/util/delta-spark-ut/java-test-args.sh +./build/sbt "++ 2.13.16" spark/test # or a single suite via testOnly +``` + +`run-delta-tests.sh` sources the same file, so CI and local runs use identical +flags. diff --git a/.github/workflows/util/delta-spark-ut/java-test-args.sh b/.github/workflows/util/delta-spark-ut/java-test-args.sh new file mode 100755 index 00000000000..94b3246a9fa --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/java-test-args.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Shared JVM options for running delta-io/delta's `spark` ScalaTest suite against +# the Gluten Velox bundle. SOURCE this file (don't execute it) before invoking +# `sbt spark/test`, in CI (run-delta-tests.sh) and locally: +# +# source .github/workflows/util/delta-spark-ut/java-test-args.sh +# ./build/sbt ... spark/test # in the Delta clone +# +# Why these are needed: JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and +# the `io.netty.tryReflectionSetAccessible` property; otherwise the forked test JVM +# fails with "sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not +# available" as soon as Gluten's bundled Arrow allocator initializes Netty direct +# buffers. Delta's own `Test / javaOptions` (project/CrossSparkVersions.scala +# `java17TestSettings`) sets the base add-opens but NOT the Netty property. This set +# mirrors `extraJavaTestArgs` in Gluten's root pom.xml. +# +# Exported via JAVA_TOOL_OPTIONS (not sbt's .jvmopts/.sbtopts, which only configure +# the sbt LAUNCHER JVM) so the flags reach BOTH the launcher and the forked test JVM +# -- the forked child inherits the parent env. +# +# NOTE: no -Xmx here on purpose. JAVA_TOOL_OPTIONS is processed BEFORE the JVM +# command line, so Delta's own -Xmx (build.sbt) would win; run-delta-tests.sh bumps +# the forked-test-JVM heap via `set spark / Test / javaOptions ++= ...` instead. + +export JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS:+${JAVA_TOOL_OPTIONS} }\ +-XX:+IgnoreUnrecognizedVMOptions \ +--add-opens=java.base/java.lang=ALL-UNNAMED \ +--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \ +--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ +--add-opens=java.base/java.io=ALL-UNNAMED \ +--add-opens=java.base/java.net=ALL-UNNAMED \ +--add-opens=java.base/java.nio=ALL-UNNAMED \ +--add-opens=java.base/java.util=ALL-UNNAMED \ +--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \ +--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED \ +--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED \ +--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \ +--add-opens=java.base/sun.nio.cs=ALL-UNNAMED \ +--add-opens=java.base/sun.security.action=ALL-UNNAMED \ +--add-opens=java.base/sun.util.calendar=ALL-UNNAMED \ +-Djdk.reflect.useDirectMethodHandle=false \ +-Dio.netty.tryReflectionSetAccessible=true \ +-Dfile.encoding=UTF-8" diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh index 9d3e8981032..72f39ec01cc 100755 --- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -27,14 +27,19 @@ # SPARK_VERSION - Delta -DsparkVersion value # UPDATE_BASELINE - 'true' -> gate seed mode; else enforce # FAIL_ON_FIXED - passed through to the gate -# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING, -# JAVA_TOOL_OPTIONS - test env (see the workflow step's `env:` block) +# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING +# - test env (see the workflow step's `env:` block) # GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts) # +# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller. set -euo pipefail export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export PATH=$JAVA_HOME/bin:$PATH +# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local +# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs. +# shellcheck source=./java-test-args.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh" cd "$GITHUB_WORKSPACE/delta" chmod +x build/sbt # Only run the unified `spark` sbt project, NOT `sparkGroup/test` -- From 20460a2f2d6cfe4c5aa3cbde6b9f37cf586886f7 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 9 Jul 2026 18:38:10 +0000 Subject: [PATCH 12/28] [CI] Quarantine flaky Delta failures by error signature, not test name The native Delta DV bitmap row-index bug (RoaringBitmapArray aborting on a Long.MAX_VALUE row index during a MERGE that writes deletion vectors) is intermittent and lands on a DIFFERENT *DVs*Suite MERGE test each run, so listing every test in flaky-tests.txt was whack-a-mole (6 entries and counting). Add error-signature quarantine: the gate now records each failed test's JUnit / text, and flaky-error-patterns.txt holds regexes matched against it. A failure matching a pattern is treated as flaky regardless of which test it hit -- neither counted as a regression nor written to the shard's failures list (so it can't leak into the regenerated baseline). Seeded with the RoaringBitmapArray signature. This is more precise than a name glob: a different real failure in the same DV suite is still caught, because only failures carrying the signature are ignored. The six *DVs*Suite name entries are removed from flaky-tests.txt (superseded); the name mechanism stays for flakes without a distinctive error signature. Verified against a real failing shard report: the DV failure is a regression without the patterns file and quarantined (gate green) with it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../workflows/util/delta-spark-ut/README.md | 28 ++++- .../delta-spark-ut/compare-test-results.py | 105 ++++++++++++++++-- .../delta-spark-ut/flaky-error-patterns.txt | 28 +++++ .../util/delta-spark-ut/flaky-tests.txt | 20 +--- .../util/delta-spark-ut/run-delta-tests.sh | 1 + 5 files changed, 152 insertions(+), 30 deletions(-) create mode 100644 .github/workflows/util/delta-spark-ut/flaky-error-patterns.txt diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index dc962c75ba4..54581dbdf64 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -33,7 +33,8 @@ starts failing** (a regression). | File | Purpose | |---|---| | `known-failures.txt` | Committed baseline: the tests currently expected to fail. One `#` per line. | -| `flaky-tests.txt` | Quarantine list: tests whose pass/fail is non-deterministic. Ignored by the gate whether they pass or fail. `#` per line. | +| `flaky-tests.txt` | Quarantine list by test name: tests whose pass/fail is non-deterministic. Ignored by the gate whether they pass or fail. `#` per line. | +| `flaky-error-patterns.txt` | Quarantine list by error signature: regex patterns matched against a failure's text, for bugs that surface on a different test each run (e.g. the native DV bitmap row-index error). | | `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and gates / seeds / aggregates against the baseline. Standard-library only. | | `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the results against the baseline via `compare-test-results.py`. | | `java-test-args.sh` | Shared JVM flags (`--add-opens` + Netty property) needed to run the suite on JDK 17 with the Gluten bundle. Sourced by `run-delta-tests.sh` and by local runs. | @@ -124,9 +125,28 @@ and blank lines allowed: contain glob metacharacters), so a same-named test in a non-matching suite is still gated normally. -Quarantining is an **interim** measure — it hides a real bug from CI. Each entry -should reference the tracking issue, and be removed once the underlying bug is -fixed so the test is enforced again. +### Quarantine by error signature + +Some bugs surface on a **different test each run** — for example the native Delta +DV bitmap row-index error (`RoaringBitmapArray ... exceeds max representable +value`, a `Long.MAX_VALUE` written during a MERGE that writes deletion vectors) +lands on a different `*DVs*Suite` MERGE test every time. Chasing those by name is +whack-a-mole, so quarantine them by **root cause** in **`flaky-error-patterns.txt`** +instead: each line is a regex matched against a failed test's ``/`` +text. Any failure that matches is treated as flaky regardless of which test it hit +(and is dropped from the shard's failures list so it can't leak into the baseline): + +``` +# regex matched against the failure message + stack (enforce mode). +Delta RoaringBitmapArray row index \d+ exceeds max representable value +``` + +This is more precise than a name glob: a *different* real failure in the same +suite is still caught, because only failures carrying the signature are ignored. + +Quarantining (either kind) is an **interim** measure — it hides a real bug from +CI. Each entry should reference the tracking issue, and be removed once the +underlying bug is fixed so the test is enforced again. ## Caveats diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index d1086dca7d9..959226a90c0 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -63,6 +63,14 @@ Its SUITE is an fnmatch glob (so one line covers a root-cause family across generated suite variants); its TEST name is matched exactly. +Flaky quarantine by error signature (``--flaky-error-patterns``) + When a failure is caused by a known nondeterministic bug that surfaces on a + *different test each run* (e.g. the native Delta DV bitmap row-index error), + matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead + quarantines by root cause: each line is a regex matched against a failed + test's / text, and any failure that matches is treated as + flaky regardless of which test it landed on. + Baseline file format (``known-failures.txt``):: # comment lines start with '#' @@ -174,6 +182,38 @@ def is_flaky(entry): return is_flaky +def load_patterns(path): + """Load flaky-error regex patterns from a file (one per line). + + Blank lines and ``#`` comments are ignored. Each remaining line is compiled + as a case-sensitive regex. These match the FAILURE TEXT of a failed test + (its JUnit / message + stack), so a test that fails with a + known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug) + can be quarantined by root cause instead of by exact test name. + """ + patterns = [] + if not path or not os.path.exists(path): + return patterns + with open(path, encoding="utf-8") as fh: + for line in fh: + line = line.rstrip("\n") + if not line.strip() or line.lstrip().startswith("#"): + continue + patterns.append(re.compile(line)) + return patterns + + +def make_signature_matcher(patterns): + """Return a predicate matching a failure text against any flaky-error pattern.""" + + def matches(text): + if not text: + return False + return any(p.search(text) for p in patterns) + + return matches + + def write_entries(path, entries, header=None): """Write a sorted set of (suite, test) tuples to a file.""" os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) @@ -206,16 +246,33 @@ def _child_local_tags(elem): return {c.tag.split("}")[-1] for c in elem} +def _failure_text(tc): + """Concatenate the message attribute + body text of a testcase's + / children, for error-signature matching.""" + parts = [] + for c in tc: + if c.tag.split("}")[-1] in ("failure", "error"): + msg = c.get("message") + if msg: + parts.append(msg) + if c.text: + parts.append(c.text) + return "\n".join(parts) + + def parse_reports(reports_dir): """Walk reports_dir for JUnit XML and classify every test. - Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is - 'failed' if its has a or child, 'skipped' if - it has a child, otherwise 'passed'. Suite-level aborts (a - reporting errors/failures with no failing ) are - recorded as a synthetic (suite, SUITE_ABORTED) failure. + Returns (passed, failed, skipped, fail_texts). The first three are sets of + (suite, test) tuples; fail_texts maps each failed (suite, test) to its + combined / message + stack text (used for error-signature + quarantine). A test is 'failed' if its has a or + child, 'skipped' if it has a child, otherwise 'passed'. Suite-level + aborts (a reporting errors/failures with no failing ) + are recorded as a synthetic (suite, SUITE_ABORTED) failure. """ passed, failed, skipped = set(), set(), set() + fail_texts = {} xml_files = [] # ScalaTest's -u reporter and Maven surefire both write `TEST-.xml` @@ -263,6 +320,7 @@ def parse_reports(reports_dir): if "failure" in tags or "error" in tags: failed.add(key) suite_has_failing_tc = True + fail_texts[key] = _failure_text(tc) elif "skipped" in tags: skipped.add(key) else: @@ -294,7 +352,7 @@ def parse_reports(reports_dir): passed -= failed skipped -= failed skipped -= passed - return passed, failed, skipped + return passed, failed, skipped, fail_texts # --------------------------------------------------------------------------- # @@ -348,16 +406,30 @@ def run_enforce(args): ) return 2 baseline = load_entries(args.known_failures) - flaky_is = make_is_flaky(load_entries(args.flaky_tests)) + name_flaky = make_is_flaky(load_entries(args.flaky_tests)) + sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns)) try: - passed, failed, skipped = parse_reports(args.reports_dir) + passed, failed, skipped, fail_texts = parse_reports(args.reports_dir) except NoReportsError as exc: eprint("ERROR: {}".format(exc)) return 2 - # Always emit this shard's artifacts for the aggregation job. + # A test is quarantined if its NAME is in flaky-tests.txt, or its failure + # TEXT matches a flaky-error signature (e.g. the native DV bitmap row-index + # bug that hits a different DV-merge test each run). + def sig_flaky(e): + return e in fail_texts and sig_matches(fail_texts[e]) + + def flaky_is(e): + return name_flaky(e) or sig_flaky(e) + + # Always emit this shard's artifacts for the aggregation job. Signature-flaky + # failures are dropped from failures-out: the aggregate job works off these + # text-less lists and cannot re-derive the signature match, so excluding them + # here keeps the regenerated baseline from absorbing a flaky failure. Name- + # flaky entries stay (the aggregate re-filters them via flaky-tests.txt). if args.failures_out: - write_entries(args.failures_out, failed) + write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)}) if args.ran_out: write_entries(args.ran_out, passed | failed) @@ -425,7 +497,7 @@ def run_enforce(args): _print_block( write, - "Quarantined flaky failures -- ignored (see flaky-tests.txt)", + "Quarantined flaky failures -- ignored (flaky-tests.txt + flaky-error-patterns.txt)", quarantined, ) @@ -581,7 +653,7 @@ def run_aggregate(args): _print_block(write, "Now-passing (global)", fixed) _print_block( write, - "Quarantined flaky failures -- ignored (see flaky-tests.txt)", + "Quarantined flaky failures -- ignored (flaky-tests.txt + flaky-error-patterns.txt)", quarantined, ) _print_block(write, "Stale baseline entries (suite/test gone)", stale) @@ -617,6 +689,15 @@ def main(argv=None): "now-passing when it passes, and is excluded from the regenerated baseline " "(aggregate mode). Optional; omitting it disables quarantining.", ) + parser.add_argument( + "--flaky-error-patterns", + help="Path to flaky-error-patterns.txt: regex patterns matched against a " + "failed test's / text (enforce mode). A failure that " + "matches is quarantined by root cause -- neither a regression nor written " + "to this shard's failures list -- so a nondeterministic native error (e.g. " + "the DV bitmap row-index bug) is ignored on whichever test it lands. " + "Optional.", + ) parser.add_argument( "--reports-dir", help="Root dir to search for JUnit XML (enforce/seed)." ) diff --git a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt new file mode 100644 index 00000000000..917f62079d4 --- /dev/null +++ b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt @@ -0,0 +1,28 @@ +# Flaky-error signatures for the Delta Spark UT (Gluten) gate. +# +# Each non-comment line is a Python regex matched (re.search, case-sensitive) +# against a failed test's JUnit / text (message + stack). A +# failure that matches is QUARANTINED by root cause: it never counts as a +# regression, and is dropped from the shard's failures list so it can't leak +# into the regenerated baseline -- regardless of WHICH test it landed on. +# +# Use this (instead of flaky-tests.txt) when a known nondeterministic bug +# surfaces on a different test each run, so matching by test name is +# whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry. +# +# --------------------------------------------------------------------------- +# Native Delta DV bitmap aggregator aborts on an invalid Long.MAX_VALUE +# (9223372036854775807) row index during a MERGE that writes deletion vectors: +# +# VeloxRuntimeError ... INVALID_STATE +# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max +# representable value 9223372030412324864 +# Expression: value <= kMaxRepresentableValue +# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp +# +# Intermittent (depends on the runtime plan/scan/scheduling), so it hits a +# different *DVs*Suite MERGE test on each run. Remove this once the row-index +# materialization is fixed in the native backend. Tracked upstream (DV bitmap +# Long.MAX_VALUE row index). +# --------------------------------------------------------------------------- +Delta RoaringBitmapArray row index \d+ exceeds max representable value diff --git a/.github/workflows/util/delta-spark-ut/flaky-tests.txt b/.github/workflows/util/delta-spark-ut/flaky-tests.txt index b56bea43947..e42348f0b0d 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-tests.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-tests.txt @@ -16,17 +16,9 @@ # Prefer fixing the underlying bug and REMOVING the entry over growing this list. # Every entry should reference a tracking issue for the root cause. # -# --------------------------------------------------------------------------- -# Root cause: native Delta DV bitmap aggregator aborts on an invalid Long.MAX_VALUE -# (9223372036854775807 > kMaxRepresentableValue) row index during MERGE that writes -# deletion vectors -- RoaringBitmapArray.cpp addSafe, INVALID_STATE VeloxRuntimeError. -# Intermittent (depends on runtime plan/scan/scheduling), so it hits a different -# `*DVs*Suite` MERGE test on each run. Tracked upstream (DV bitmap Long.MAX_VALUE). -# Remove these once the row-index materialization is fixed in the native backend. -# --------------------------------------------------------------------------- -*DVs*Suite#basic case - merge to Delta table by path, isPartitioned: true -*DVs*Suite#extended syntax - conditional update + conditional delete + conditional insert - isPartitioned: true -*DVs*Suite#extended syntax - update + conditional insert - isPartitioned: true -*DVs*Suite#matched only merge - disabled - with update and delete - isPartitioned: true -*DVs*Suite#matched only merge - enabled - with update and delete - isPartitioned: true -*DVs*Suite#single file, isPartitioned: true +# NOTE: when a bug surfaces on a DIFFERENT test each run (so matching by name is +# whack-a-mole), quarantine it by ERROR SIGNATURE in flaky-error-patterns.txt +# instead. The native Delta DV bitmap row-index bug (RoaringBitmapArray +# Long.MAX_VALUE) is handled there, which is why no `*DVs*Suite` MERGE entries +# are listed below. + diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh index 72f39ec01cc..09b0d0bf04d 100755 --- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -234,6 +234,7 @@ python3 "$UTIL_DIR/compare-test-results.py" \ --reports-dir "$GITHUB_WORKSPACE/delta" \ --known-failures "$UTIL_DIR/known-failures.txt" \ --flaky-tests "$UTIL_DIR/flaky-tests.txt" \ + --flaky-error-patterns "$UTIL_DIR/flaky-error-patterns.txt" \ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${SHARD_ID}.txt" \ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${SHARD_ID}.txt" \ --fail-on-fixed "${FAIL_ON_FIXED}" From 69333144db91f8e967b15a57b05ff8cb0b5aef11 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 9 Jul 2026 22:51:48 +0000 Subject: [PATCH 13/28] [CI] Quarantine the negative-index DV-bitmap flaky variant Run 29042495519 regressed on another DV-merge test whose failure was NOT the RoaringBitmapArray "exceeds max representable value" signature but a second, distinct native error from the same DV bitmap aggregation -- Reason: Delta bitmap row index cannot be negative: -6254810385378525259 Expression: value >= 0 Function: addRowIndex File: .../delta/DeltaBitmapAggregator.cc:44 vs the previously-seen too-large variant (value <= kMaxRepresentableValue, RoaringBitmapArray.cpp addSafe). Same root cause (garbage row index into the DV bitmap aggregator), different bounds check and different message. Add a second explicit pattern for it rather than broadening the existing one -- the two are distinct native error strings, so keeping each pattern tightly bound to its message avoids masking an unrelated failure that merely mentions a bitmap row index. Verified against the real failing report: the negative-index failure is a regression without the pattern and quarantined with it; the suite's five other (non-bitmap) baseline failures and benign controls are not matched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../workflows/util/delta-spark-ut/README.md | 18 +++++---- .../delta-spark-ut/flaky-error-patterns.txt | 37 +++++++++++++------ 2 files changed, 37 insertions(+), 18 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 54581dbdf64..85d1798d7cd 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -128,17 +128,21 @@ and blank lines allowed: ### Quarantine by error signature Some bugs surface on a **different test each run** — for example the native Delta -DV bitmap row-index error (`RoaringBitmapArray ... exceeds max representable -value`, a `Long.MAX_VALUE` written during a MERGE that writes deletion vectors) -lands on a different `*DVs*Suite` MERGE test every time. Chasing those by name is -whack-a-mole, so quarantine them by **root cause** in **`flaky-error-patterns.txt`** -instead: each line is a regex matched against a failed test's ``/`` -text. Any failure that matches is treated as flaky regardless of which test it hit -(and is dropped from the shard's failures list so it can't leak into the baseline): +DV bitmap row-index error (the aggregator gets a garbage row index during a MERGE +that writes deletion vectors and aborts, e.g. `Delta RoaringBitmapArray row index +... exceeds max representable value` or `Delta bitmap row index cannot be +negative: ...`) lands on a different `*DVs*Suite` MERGE test every time. Chasing +those by name is whack-a-mole, so quarantine them by **root cause** in +**`flaky-error-patterns.txt`** instead: each line is a regex matched against a +failed test's ``/`` text. Any failure that matches is treated as +flaky regardless of which test it hit (and is dropped from the shard's failures +list so it can't leak into the baseline): ``` # regex matched against the failure message + stack (enforce mode). +# one explicit pattern per known error, deliberately specific. Delta RoaringBitmapArray row index \d+ exceeds max representable value +Delta bitmap row index cannot be negative: -?\d+ ``` This is more precise than a name glob: a *different* real failure in the same diff --git a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt index 917f62079d4..23cf641aa99 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt @@ -11,18 +11,33 @@ # whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry. # # --------------------------------------------------------------------------- -# Native Delta DV bitmap aggregator aborts on an invalid Long.MAX_VALUE -# (9223372036854775807) row index during a MERGE that writes deletion vectors: +# Native Delta bitmap aggregator receives an INVALID row index during a MERGE +# that writes deletion vectors, and aborts. The garbage row index trips one of +# two bounds checks, so the same root cause shows up with two messages: # -# VeloxRuntimeError ... INVALID_STATE -# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max -# representable value 9223372030412324864 -# Expression: value <= kMaxRepresentableValue -# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp +# too large (Long.MAX_VALUE): +# VeloxRuntimeError INVALID_STATE +# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max +# representable value 9223372030412324864 +# Expression: value <= kMaxRepresentableValue +# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92 # -# Intermittent (depends on the runtime plan/scan/scheduling), so it hits a -# different *DVs*Suite MERGE test on each run. Remove this once the row-index -# materialization is fixed in the native backend. Tracked upstream (DV bitmap -# Long.MAX_VALUE row index). +# negative (garbage): +# VeloxRuntimeError INVALID_STATE +# Reason: Delta bitmap row index cannot be negative: -6254810385378525259 +# Expression: value >= 0 +# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44 +# +# Both are the same root cause (garbage row index into the DV bitmap aggregator) +# but distinct native errors, so each has its own explicit pattern below -- +# deliberately specific (bound to the exact error string) rather than a broad +# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a +# new line if a further bounds-check variant appears. Intermittent (depends on +# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test +# on each run. Remove these once the row-index materialization is fixed in the +# native backend. Tracked upstream (DV bitmap invalid row index). # --------------------------------------------------------------------------- +# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue Delta RoaringBitmapArray row index \d+ exceeds max representable value +# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0 +Delta bitmap row index cannot be negative: -?\d+ From 319995d706aadc6ccc328797cba511db63997da7 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 13 Jul 2026 20:36:14 +0000 Subject: [PATCH 14/28] [CI] Remove now-passing Variant/UDT tests from Delta baseline The Velox Variant/UDT type-offload gap was fixed upstream, so 74 Delta UT tests (65 Variant + 9 user-defined-type) that were expected failures now pass and were tripping the fail-on-fixed gate as "now-passing". Remove them from known-failures.txt (810 -> 736). The 6 remaining `variant auto compact` entries are a different root cause (auto-compact metrics) and correctly stay in the baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../util/delta-spark-ut/known-failures.txt | 74 ------------------- 1 file changed, 74 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index 624e0c5cf46..3cf71910846 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -51,25 +51,9 @@ org.apache.spark.sql.delta.AutoCompactExecutionSuite#auto-compact-enabled-proper org.apache.spark.sql.delta.AutoCompactExecutionSuite#auto-compact-enabled-property: auto compact should not kick in when session config is off org.apache.spark.sql.delta.AutoCompactExecutionSuite#variant auto compact kicks in when enabled - session config org.apache.spark.sql.delta.AutoCompactExecutionSuite#variant auto compact kicks in when enabled - table config -org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled -org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsSuite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled org.apache.spark.sql.delta.CheckpointsSuite#SC-86940: writing a GCS checkpoint should happen in a new thread -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch100Suite#SC-86940: writing a GCS checkpoint should happen in a new thread -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch1Suite#SC-86940: writing a GCS checkpoint should happen in a new thread -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs corrupts variant stats when collectVariantDataSkippingStats is disabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves nested variant stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves variant and struct stats when collectVariantDataSkippingStats is enabled -org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#DML with DVs preserves variant stats when collectVariantDataSkippingStats is enabled org.apache.spark.sql.delta.CheckpointsWithCatalogOwnedBatch2Suite#SC-86940: writing a GCS checkpoint should happen in a new thread org.apache.spark.sql.delta.CloneTableSQLSuite#shallow clone across file systems org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch100Suite#shallow clone across file systems @@ -156,27 +140,6 @@ org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#convert to delta with partiti org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#partitioned convert to delta with schema change org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - delete tombstones org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - update/merge -org.apache.spark.sql.delta.DeltaVariantShreddingSuite#Infer schema for Delta table -org.apache.spark.sql.delta.DeltaVariantSuite#DISABLE_VARIANT_TABLE_FEATURE_FOR_SPARK_40 - config disabled does not block -org.apache.spark.sql.delta.DeltaVariantSuite#DISABLE_VARIANT_TABLE_FEATURE_FOR_SPARK_40 - no-op on Spark 4.1+ -org.apache.spark.sql.delta.DeltaVariantSuite#Existing table with variant type can enable CDF -org.apache.spark.sql.delta.DeltaVariantSuite#Table with variant type can use CDF -org.apache.spark.sql.delta.DeltaVariantSuite#Variant can be used as a source for generated columns -org.apache.spark.sql.delta.DeltaVariantSuite#Variant can have default value set -org.apache.spark.sql.delta.DeltaVariantSuite#Variant cannot be created as a generated column -org.apache.spark.sql.delta.DeltaVariantSuite#Variant respects Delta table CHECK constraints -org.apache.spark.sql.delta.DeltaVariantSuite#Variant respects Delta table IS NOT NULL constraints -org.apache.spark.sql.delta.DeltaVariantSuite#Zorder is not supported for Variant -org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - id - false -org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - id - true -org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - name - false -org.apache.spark.sql.delta.DeltaVariantSuite#column mapping works - name - true -org.apache.spark.sql.delta.DeltaVariantSuite#optimize variant -org.apache.spark.sql.delta.DeltaVariantSuite#shallow cloning table with variant -org.apache.spark.sql.delta.DeltaVariantSuite#streaming variant delta table -org.apache.spark.sql.delta.DeltaVariantSuite#time travel with variant column works -org.apache.spark.sql.delta.DeltaVariantSuite#variant works with schema evolution for INSERT -org.apache.spark.sql.delta.DeltaVariantSuite#variant works with schema evolution for MERGE org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skip deleted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skipping deleted file still throws on corrupted file org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#deleted files cause failure by default @@ -495,16 +458,8 @@ org.apache.spark.sql.delta.coordinatedcommits.CoordinatedCommitsSuite#Incomplete org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#DELETE with DVs with column mapping mode=id org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#huge table: delete a small number of rows from tables of 2B rows with DVs org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#huge table: read from tables of 2B rows with existing DV of many zeros -org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#variant types DELETE with DVs with column mapping mode=id -org.apache.spark.sql.delta.deletionvectors.DeletionVectorsSuite#variant types DELETE with DVs with column mapping mode=name org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite#(It is not a test it is a sbt.testing.SuiteSelector) org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite# -org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLNameBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedCDCOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedDVPredPushOffSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedDVPredPushOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.DeleteBaseSQLPathBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.DeleteBaseScalaSuite#Variant type org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLNameBasedSuite#test delete on temp view - nontrivial projection - Dataset TempView org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLNameBasedSuite#test delete on temp view - nontrivial projection - SQL TempView org.apache.spark.sql.delta.generatedsuites.DeleteTempViewSQLPathBasedCDCOnSuite#test delete on temp view - nontrivial projection - Dataset TempView @@ -611,53 +566,26 @@ org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumn org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedCDCOnSuite#schema evolution - extra nested column in source - update - single target partition org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnSQLPathBasedSuite#schema evolution - extra nested column in source - update - single target partition org.apache.spark.sql.delta.generatedsuites.MergeIntoSchemaEvolutionBaseNewColumnScalaSuite#schema evolution - extra nested column in source - update - single target partition -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLNameBasedSuite#merge with repartition - insert only merge -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOffSuite#merge with repartition - partition on multiple columns -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnDVsPredPushOnSuite#merge with repartition - partition on multiple columns -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedCDCOnSuite#merge with repartition - insert only merge -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOffSuite#merge with repartition - partition on multiple columns -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - insert only merge org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedDVsPredPushOnSuite#merge with repartition - partition on multiple columns -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscSQLPathBasedSuite#merge with repartition - insert only merge -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#UDT Data Types - simple and nested -org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#Variant type org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#data skipping with matched predicates - with insert clause org.apache.spark.sql.delta.generatedsuites.MergeIntoSuiteBaseMiscScalaSuite#merge with repartition - insert only merge -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLNameBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnDVSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnRowTrackingOffSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnRowTrackingOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedCDCOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedDVPredPushOffSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedDVPredPushOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedRowTrackingOffSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedRowTrackingOnSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscSQLPathBasedSuite#Variant type -org.apache.spark.sql.delta.generatedsuites.UpdateBaseMiscScalaSuite#Variant type org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLNameBasedSuite#test update on temp view - nontrivial projection - Dataset TempView org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLNameBasedSuite#test update on temp view - nontrivial projection - SQL TempView org.apache.spark.sql.delta.generatedsuites.UpdateBaseTempViewSQLPathBasedCDCOnDVSuite#test update on temp view - nontrivial projection - Dataset TempView @@ -801,14 +729,12 @@ org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite# org.apache.spark.sql.delta.stats.DataSkippingDeltaV1WithCatalogOwnedBatch2Suite#data skipping on TIMESTAMP_NTZ with Long.MaxValue - old behavior with DataFrame schema org.apache.spark.sql.delta.stats.PartitionLikeDataSkippingColumnMappingSuite#partition-like data skipping for expression COALESCE: COALESCE(TO_DATE(S.b), c) = '1976-07-03' - column mapping id mode org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute stats multiple columns and files -org.apache.spark.sql.delta.stats.StatsCollectionSuite#recompute variant stats org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening BIGINT -> DECIMAL(20,0), partitioned=true org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DATE -> TIMESTAMP_NTZ, partitioned=false org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DATE -> TIMESTAMP_NTZ, partitioned=true org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening DECIMAL(9,2) -> DECIMAL(19,3), partitioned=true org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening FLOAT -> DOUBLE, partitioned=true org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening INT -> DOUBLE, partitioned=true -org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type widening with user-defined type in table org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#unsupported type changes DOUBLE -> FLOAT, partitioned=true org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#unsupported type changes TIMESTAMP_NTZ -> DATE, partitioned=false org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - always automatic type widening DATE -> TIMESTAMP_NTZ From 3f3076f873fc19a9429caf84308fe997560de64e Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 14 Jul 2026 02:55:57 +0000 Subject: [PATCH 15/28] [CI] Remove now-passing DV-tombstone test from Delta baseline DeltaFastDropFeatureSuite "We do not create redundant DV tombstones after cloning isShallowClone: true" now passes consistently (green on 2026-07-13 and 2026-07-14 runs); the earlier FileNotFoundException on a deletion-vector .bin during the DROP FEATURE parallel OPTIMIZE no longer reproduces. Keeping it in the baseline trips the fail-on-fixed gate on every run, so remove it (736 -> 735). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/util/delta-spark-ut/known-failures.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index 3cf71910846..0999a1a389c 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -90,7 +90,6 @@ org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#SC-86916: read/write org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#all operations should propagate Hadoop file system options org.apache.spark.sql.delta.DeltaDataFrameHadoopOptionsSuite#operations without Hadoop options should fail for fake:// filesystem org.apache.spark.sql.delta.DeltaFastDropFeatureSuite#Vacuum does not delete deletion vector files.generateDVTombstones: false -org.apache.spark.sql.delta.DeltaFastDropFeatureSuite#We do not create redundant DV tombstones after cloning isShallowClone: true org.apache.spark.sql.delta.DeltaGenerateSymlinkManifestSuite#incremental manifest: failure to generate manifest throws exception org.apache.spark.sql.delta.DeltaGenerateSymlinkManifestSuite#special partition column values org.apache.spark.sql.delta.DeltaHistoryManagerSuite#data skipping still works with time travel From b5e1f03ae67b4d59d1edd576f6dc9c299eab5517 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 20 Jul 2026 20:31:53 +0000 Subject: [PATCH 16/28] [CI] Fail Delta shard on watchdog kill; kill only forks; normalize gate keys Addresses three review comments on the Delta Spark UT pipeline: * Hang watchdog: when it kills a wedged test fork, the running suite plus every suite queued behind it in that JVM never run and never write a report; since we ignore sbt's exit code, the gate only judged the suites that reported and the shard could go green. The watchdog now touches a marker on kill and the shard fails afterwards if it exists. * Hang watchdog: only KILL matched sbt.ForkMain fork(s), never the sbt launcher. Before any fork exists (dependency resolution / cold-cache compile) sbt can be silent for >15 min; killing the launcher then wasted the slot on a confusing "compile/launch failure". All JVMs are still dumped for diagnostics. The per-episode dump/kill budget resets when output resumes so a transient pre-fork stall can't starve a later fork hang. * Gate: normalize (suite, test) keys parsed from JUnit XML the same way baseline/flaky entries are (write_entries collapses CR/LF; parse_entry strips the line), so a test name with a trailing newline or surrounding whitespace is suppressible by a line pasted from the gate's REGRESSION output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../delta-spark-ut/compare-test-results.py | 26 +++++++- .../util/delta-spark-ut/run-delta-tests.sh | 66 ++++++++++++++----- 2 files changed, 74 insertions(+), 18 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index 959226a90c0..e3b6b9b2404 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -139,6 +139,28 @@ def parse_entry(line): return (stripped[:idx], stripped[idx + len(SEP) :]) +def normalize_key(suite, test): + """Normalize a (suite, test) key parsed from JUnit XML to match baseline keys. + + Baseline/flaky entries round-trip through write_entries (which collapses CR/LF + in the test name to spaces) and parse_entry (which strips the whole + ``suite#test`` line). A raw XML name carrying a trailing newline or + surrounding whitespace would therefore never equal its normalized baseline + entry: the gate would keep reporting it as a REGRESSION, and the + copy-pasteable line it prints could never suppress it (load strips it back). + Delta test names are freeform, so a version bump could introduce exactly that. + Applying the identical format+parse round-trip here keeps the two sides in + sync (and is a no-op for the normal, whitespace-free names). + """ + safe_test = (test or "").replace("\r", " ").replace("\n", " ") + normalized = parse_entry(format_entry(suite or "", safe_test)) + # parse_entry only returns None for a blank/comment line, which a real + # testcase key is not; fall back to a bare strip to keep this total. + if normalized is None: + return ((suite or "").strip(), safe_test.strip()) + return normalized + + def load_entries(path): """Load a set of (suite, test) tuples from a baseline/shard-list file.""" entries = set() @@ -315,7 +337,7 @@ def parse_reports(reports_dir): continue suite = tc.get("classname") or suite_name name = tc.get("name") or "" - key = (suite, name) + key = normalize_key(suite, name) tags = _child_local_tags(tc) if "failure" in tags or "error" in tags: failed.add(key) @@ -336,7 +358,7 @@ def parse_reports(reports_dir): except ValueError: errors = failures = 0 if (errors + failures) > 0 and not suite_has_failing_tc: - failed.add((suite_name, SUITE_ABORTED)) + failed.add(normalize_key(suite_name, SUITE_ABORTED)) if not parsed_any: raise NoReportsError( diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh index 09b0d0bf04d..79238505a43 100755 --- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -89,8 +89,13 @@ set +e # threads (to the job log, and to a file for the artifact) once the test # output has been silent for too long, so the deadlock is diagnosable. SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log" +# Marker the watchdog touches when it KILLS a wedged test fork. The killed fork's +# running suite plus every suite queued behind it never run and never write a +# report, and since we ignore sbt's exit code the gate would only judge the +# suites that DID report -- so the main flow fails the shard when this exists. +WATCHDOG_KILL_MARKER="/tmp/sbt-watchdog-killed-shard-${SHARD_ID}" : > "$SBT_LOG" -rm -f /tmp/sbt-done +rm -f /tmp/sbt-done "$WATCHDOG_KILL_MARKER" ( # CRITICAL: the step shell runs with `bash -eo pipefail`, which the # subshell inherits. Without `set +e` here, ANY non-zero command -- @@ -148,14 +153,22 @@ rm -f /tmp/sbt-done # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how # long the test has been silent) without waiting for a hang. [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago" + # The dump/kill budget below is PER silent-episode: reset it whenever output + # is flowing again. Otherwise a transient pre-fork/compile stall that we dump + # but (correctly) don't kill could exhaust the budget and leave a later real + # fork hang un-dumped and un-killed for the rest of the run. + [ "$silent" -lt "$silent_limit" ] && dumps=0 if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then dumps=$(( dumps + 1 )) - pids="$(fork_pids | sort -un)" - # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM. - [ -n "$pids" ] || pids="$(all_java_pids | sort -un)" - echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))" - [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump" - for pid in $pids; do + fork_matched="$(fork_pids | sort -un)" + # Dump set: the sbt.ForkMain test fork(s) if we can pinpoint them; if not, + # dump EVERY JVM so a hang is still diagnosable. Diagnostics are harmless on + # any JVM, so the broad fallback stays here. + dump_pids="$fork_matched" + [ -n "$dump_pids" ] || dump_pids="$(all_java_pids | sort -un)" + echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $dump_pids))" + [ -n "$dump_pids" ] || echo "HANG WATCHDOG: no java process found to dump" + for pid in $dump_pids; do # SIGQUIT makes the JVM print a full thread dump to its OWN stderr, # which sbt relays into the test log via the SAME stream as test # output -- so it lands in the job log even when a separately @@ -167,16 +180,27 @@ rm -f /tmp/sbt-done || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}" done echo "::endgroup::" - # The dump is now captured (job log via SIGQUIT + artifact via - # jstack file). A hung JVM otherwise stalls the whole shard until - # the 350-min job timeout AND keeps the job log frozen so the dump - # never becomes reachable. So KILL the wedged JVM(s): the suite - # fails fast (acceptable -- errors are expected; only an - # unrecoverable hang blocks CI), the job proceeds/ends, and the log - # + artifacts flush. Give SIGQUIT a moment to print first. + # The dump is now captured (job log via SIGQUIT + artifact via jstack + # file). A hung fork otherwise stalls the whole shard until the 350-min job + # timeout AND keeps the job log frozen so the dump never becomes reachable. + # So KILL the wedged fork(s): the suite fails fast (acceptable -- errors are + # expected; only an unrecoverable hang blocks CI), the job proceeds/ends, + # and the log + artifacts flush. Give SIGQUIT a moment to print first. sleep 20 - echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)" - for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done + # Kill ONLY the matched sbt.ForkMain fork(s) -- never the sbt launcher. + # Before any fork exists (dependency resolution or a cold-cache compile of + # the big spark test module) sbt can legitimately go silent for >15 min; + # killing the launcher then would kill the job with a confusing "compile or + # launch failure" and waste the whole slot. A pre-fork hang is left running + # (rare; still bounded by the 350-min job timeout). Touch the marker so the + # main flow fails the shard: the killed fork's queued suites never ran. + if [ -n "$fork_matched" ]; then + echo "HANG WATCHDOG: killing wedged fork JVM(s) to unblock the shard:$(printf ' %s' $fork_matched)" + touch "$WATCHDOG_KILL_MARKER" + for pid in $fork_matched; do kill -KILL "$pid" 2>/dev/null; done + else + echo "HANG WATCHDOG: no sbt.ForkMain fork matched -- dumped all JVMs but leaving sbt running (likely a pre-fork resolve/compile stall, not a wedged test)." + fi fi done ) & @@ -210,6 +234,16 @@ echo "sbt spark/test exited with ${SBT_EXIT}" [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; } done ) || true +# If the hang watchdog killed a wedged test fork, the suite it was running plus +# every suite QUEUED BEHIND it in that fork never ran and never wrote a report. +# Because we intentionally ignore sbt's exit code, the gate would only judge the +# suites that DID report and could go green with those tests silently unrun. A +# watchdog kill is an abnormal run, so fail the shard outright (re-run to retry). +if [ -f "$WATCHDOG_KILL_MARKER" ]; then + echo "::error::hang watchdog killed a wedged test fork on shard ${SHARD_ID}; suites queued behind it never ran, so results are incomplete. Failing the shard." + exit 1 +fi + # A compile/launch failure leaves no reports at all. In that case the # gate would see zero failures and pass spuriously, so fail loudly. REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true) From f25a9b701f1a3b198fd2378d35bfbcf76c38650d Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 20 Jul 2026 21:40:01 +0000 Subject: [PATCH 17/28] [CI] Gate Delta UT per-PR to Delta paths + label; add nightly full run Reduce GitHub Actions usage for the Delta Spark UT suite (per review feedback on the pipeline PR): * Per PR (velox_backend_x86.yml): a new `delta-changes` job runs the suite only when the PR touches high-signal Delta paths -- the Delta integration code (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own files -- or carries the `run-delta-ci` opt-in label. Changes to general Velox/core/native code (touched on most PRs) skip it; this drops the per-PR trigger rate from ~60% to ~17% of recent commits. * Nightly (delta_spark_ut.yml): a `schedule` (05:00 UTC) runs the full suite against the latest default branch, so regressions from the skipped-per-PR paths are still caught daily. It builds its own native lib (no caller) and uses fail_on_fixed=true, so baseline drift surfaces as a red nightly -- the signal to refresh known-failures.txt. * Docs: README "When it runs" section documents the per-PR path gate, the opt-in label, and the nightly run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/delta_spark_ut.yml | 39 ++++++++----- .../workflows/util/delta-spark-ut/README.md | 21 +++++++ .github/workflows/velox_backend_x86.yml | 56 ++++++++++++++++++- 3 files changed, 102 insertions(+), 14 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index ad5e74d860f..580cb95fbd9 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -93,6 +93,14 @@ on: type: boolean required: false default: true + # Nightly full run against the latest default branch. The per-PR entry point + # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches + # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this + # scheduled run keeps full coverage once a day so rarer regressions are still + # caught. It builds its own native lib (build-native-lib-centos-7 below) since + # there is no caller to provide one, and uses the workflow's default inputs. + schedule: + - cron: '0 5 * * *' env: ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true @@ -106,10 +114,10 @@ env: # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a # compatible delta_ref. - GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }} + GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version || '4.1' }} GLUTEN_SCALA_PROFILE: 'scala-2.13' GLUTEN_JAVA_PROFILE: 'java-17' - GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }} + GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version || '4.1' }} GLUTEN_BUNDLE_SCALA_VERSION: '2.13' DELTA_SCALA_VERSION: '2.13.16' # Number of shards in the delta-spark-test matrix. Must equal the length of @@ -133,10 +141,10 @@ env: jobs: build-native-lib-centos-7: - # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml - # the caller already built the native lib and passes it as an input, so this - # job is skipped and the duplicate native build is avoided. - if: github.event_name == 'workflow_dispatch' + # Standalone runs (workflow_dispatch + nightly schedule) build the native lib + # here. When called by velox_backend_x86.yml the caller already built it and + # passes it as an input, so this job is skipped and the duplicate build avoided. + if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 @@ -262,14 +270,19 @@ jobs: - name: Resolve workflow inputs id: resolve - # Every input has a default (workflow_call + workflow_dispatch), so they - # are always set; just surface them as step outputs for the steps below. + # Surface the inputs as step outputs. workflow_call / workflow_dispatch + # supply them (with defaults); the nightly `schedule` event supplies NONE, + # so fall back to the same defaults here. The boolean inputs are rendered + # as explicit 'true'/'false' strings (never empty) via the `&&/||` form so + # a schedule run resolves cleanly: update_baseline=false (enforce) and + # fail_on_fixed=true (so a now-passing baseline test turns the nightly red + # -- our signal that the committed baseline needs refreshing). env: - DELTA_REF: ${{ inputs.delta_ref }} - SPARK_VERSION: ${{ inputs.spark_version }} - TEST_PARALLELISM: ${{ inputs.test_parallelism }} - UPDATE_BASELINE: ${{ inputs.update_baseline }} - FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }} + DELTA_REF: ${{ inputs.delta_ref || 'v4.2.0' }} + SPARK_VERSION: ${{ inputs.spark_version || '4.1' }} + TEST_PARALLELISM: ${{ inputs.test_parallelism || '4' }} + UPDATE_BASELINE: ${{ inputs.update_baseline && 'true' || 'false' }} + FAIL_ON_FIXED: ${{ github.event_name == 'schedule' && 'true' || (inputs.fail_on_fixed && 'true' || 'false') }} run: | set -euo pipefail { diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 85d1798d7cd..3111050f687 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -63,6 +63,27 @@ Because Delta shards **by suite**, every suite (and therefore every test) runs in exactly one shard, so per-shard enforcement sees complete suites and never double-counts. +## When it runs + +To keep GitHub Actions usage in check, the suite does **not** run on every PR: + +- **Per PR** — `velox_backend_x86.yml` runs the Delta suite only when the PR + touches a **high-signal Delta path**: the Delta integration code + (`backends-velox/src-delta*`), the `gluten-delta` module, or this pipeline's + own files (`delta_spark_ut.yml`, `util/delta-spark-ut/**`, + `velox_backend_x86.yml`). Changes to general Velox/core/native code can also + affect Delta offload, but they're touched on most PRs, so per-PR they skip the + suite — the nightly run and the opt-in label are the safety nets. Add the + **`run-delta-ci`** label to force the suite on any PR (the label is read from + the triggering event, so apply it before/with a push). +- **Nightly** — `delta_spark_ut.yml` runs the **full** suite against the latest + default branch on a `schedule` (05:00 UTC), so rarer regressions are still + caught daily. The nightly run enforces the baseline **and** fails on + now-passing tests (`fail_on_fixed=true`), so baseline drift surfaces as a red + nightly — the signal to refresh `known-failures.txt`. +- **Manually** — **Actions → Delta Spark UT (Gluten) → Run workflow** + (`workflow_dispatch`), e.g. to refresh the baseline (see below). + ## Bootstrapping the baseline (first time) While `known-failures.txt` has no entries the gate auto-runs in **seed mode** diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index de701db37f3..82527a8cbf4 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -110,8 +110,62 @@ jobs: # built above instead of duplicating the native build. Not gated on Delta-only # paths: core/velox/substrait/cpp/shims changes can affect Delta query offload, # so this runs on every trigger like the other spark-test jobs. + # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR + # touches high-signal Delta paths -- the Delta integration code + # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own + # files -- or carries the `run-delta-ci` opt-in label. Changes to general + # Velox/core/native code can also affect Delta offload but are touched + # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml + # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage + # down. NOTE: the label is read from the event that triggered this run, so add + # it before/with a push; labeling an already-finished run needs a new push. + delta-changes: + runs-on: ubuntu-22.04 + outputs: + run_delta: ${{ steps.filter.outputs.run_delta }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Detect Delta-relevant changes / opt-in label + id: filter + env: + HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: | + set -euo pipefail + # Opt-in label forces the suite even with no Delta-relevant path change. + if [ "$HAS_LABEL" = "true" ]; then + echo "run-delta-ci label present -> running Delta suite" + echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 + fi + # Fail open if we can't determine the PR range (e.g. a non-PR trigger): + # never silently skip coverage. + if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then + echo "no PR base/head sha -> running Delta suite (fail-open)" + echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 + fi + BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA") + echo "diff base=$BASE head=$HEAD_SHA" + # High-signal Delta paths only: the Delta integration code + # (backends-velox/src-delta*), the Delta module, and this pipeline's own + # files. A change to general Velox/core/native code can also affect Delta + # offload, but those are touched constantly; per-PR we skip them (the + # nightly full run + the `run-delta-ci` label are the safety nets) to + # keep GHA usage down. + if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq \ + '^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)'; then + echo "Delta-relevant paths changed -> running Delta suite" + echo "run_delta=true" >> "$GITHUB_OUTPUT" + else + echo "No Delta-relevant paths changed and no opt-in label -> skipping Delta suite" + echo "run_delta=false" >> "$GITHUB_OUTPUT" + fi + delta-spark-ut: - needs: build-native-lib-centos-7 + needs: [build-native-lib-centos-7, delta-changes] + if: ${{ needs.delta-changes.outputs.run_delta == 'true' }} uses: ./.github/workflows/delta_spark_ut.yml with: native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }} From 30fe8c1f4712212a68d2cb2399c980c41be33181 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 20 Jul 2026 22:13:45 +0000 Subject: [PATCH 18/28] [CI] Fix stale Delta-job comment in velox_backend_x86.yml The comment above the Delta Spark UT job still said it "runs on every trigger like the other spark-test jobs", which contradicted the per-PR gating added by the delta-changes job. Remove the stale block; keep the accurate gating comment and add a concise native-lib-reuse note on the delta-spark-ut job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/velox_backend_x86.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index 82527a8cbf4..cd53a4fc886 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -106,10 +106,6 @@ jobs: path: ./cpp/build/ if-no-files-found: error - # Delta Spark UT, run via the reusable workflow so it reuses the native lib - # built above instead of duplicating the native build. Not gated on Delta-only - # paths: core/velox/substrait/cpp/shims changes can affect Delta query offload, - # so this runs on every trigger like the other spark-test jobs. # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR # touches high-signal Delta paths -- the Delta integration code # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own @@ -163,6 +159,10 @@ jobs: echo "run_delta=false" >> "$GITHUB_OUTPUT" fi + # Run the Delta Spark UT via the reusable workflow, passing the native lib + # built above so it is not rebuilt. Gated by `delta-changes` (Delta-relevant + # paths or the `run-delta-ci` label); the nightly full run lives in + # delta_spark_ut.yml's `schedule` trigger. delta-spark-ut: needs: [build-native-lib-centos-7, delta-changes] if: ${{ needs.delta-changes.outputs.run_delta == 'true' }} From b6b3b08e661f7924d1e750206032aa3f6f976275 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 27 Jul 2026 18:08:54 +0000 Subject: [PATCH 19/28] [CI] Baseline new ImplicitStreamingMergeCasting BIGINT->DECIMAL overflow failure After the latest rebase, ImplicitStreamingMergeCastingSuite "Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY" fails deterministically: Velox raises `VeloxUserError INVALID_ARGUMENT: Cannot cast BIGINT '9223372036854775807' to DECIMAL(7,2)` (rescaleInt, DecimalUtil.h) where vanilla Spark handles the overflow per the ANSI/LEGACY policy. Its followAnsiEnabled:true siblings are already baselined; add this variant so the gate stays green (735 -> 736 entries, still sorted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/util/delta-spark-ut/known-failures.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index 0999a1a389c..dbf2108cf0d 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -361,6 +361,7 @@ org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATC org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: Struct, targetType: Struct followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY +org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow sourceType: BIGINT, targetType: INT followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI From e40e346eff443f846ef556b4482f36ec1a890f8d Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 27 Jul 2026 20:02:10 +0000 Subject: [PATCH 20/28] [CI] Skip already-applied Delta cherry-picks in setup-delta.sh setup-delta.sh runs under `set -euo pipefail`, so cherry-picking a Delta FileSourceScanLike fix that the pinned DELTA_REF already contains exits non-zero (empty/conflicting patch) and aborts the whole setup. This is a latent break the moment DELTA_REF is bumped past delta-io/delta#7104/#7105. Make cherry_pick_delta_fix attempt the cherry-pick and, on failure, recover only the paths that fix touches (git diff-tree -> per-file reset + checkout, then cherry-pick --quit) and continue -- leaving the DeltaSQLCommandTest patch and bundle jar intact. It is self-correcting: a genuinely missing fix resurfaces as gate regressions instead of a hard abort. Ancestry can't distinguish "already contained" from a real conflict here because the Delta clone is shallow (depth 1) and merge-base --is-ancestor can't see past the graft; a reverse-apply-check is fragile when the newer ref carries the fix plus adjacent edits. Recover-on-failure avoids both. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../util/delta-spark-ut/setup-delta.sh | 28 +++++++++++++++++-- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 1d9dcf7f954..5cc216b79f8 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -125,16 +125,38 @@ echo "::endgroup::" # matches to the shared `FileSourceScanLike` interface that both the vanilla and # Gluten scans implement (behavior-preserving for vanilla). Both are merged # upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once -# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go. +# DELTA_REF includes a fix, cherry_pick_delta_fix detects it and skips (see below). # # Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to # diff against (a depth-1 fetch grafts the parent away); `-n` stages the change # without requiring a committer identity. cherry_pick_delta_fix() { local sha="$1" pr="$2" - echo "Cherry-picking delta-io/delta${pr}" git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha" - git -C "$DELTA_DIR" cherry-pick -n "$sha" + echo "Cherry-picking delta-io/delta${pr}" + if git -C "$DELTA_DIR" cherry-pick -n "$sha"; then + return 0 + fi + # The cherry-pick did not apply. The usual cause is that the pinned DELTA_REF + # already contains this fix (e.g. after a version bump), which makes the patch + # empty/conflicting and would -- under `set -e` -- abort the whole setup. We + # can't use ancestry to tell "already contained" from a genuine conflict here + # (the clone is shallow, so `merge-base --is-ancestor` can't see past the graft), + # so recover the exact paths this fix touches -- leaving other setup such as the + # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting: if + # the fix is genuinely still needed, the FileSourceScanLike failures it prevents + # resurface as gate regressions rather than being hidden by a hard abort here. + echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \ + "(most likely already contained in ${DELTA_REF}); skipping it." + local f + while IFS= read -r f; do + [ -n "$f" ] || continue + git -C "$DELTA_DIR" reset -q -- "$f" 2>/dev/null || true + git -C "$DELTA_DIR" checkout -q -- "$f" 2>/dev/null || true + done < <(git -C "$DELTA_DIR" diff-tree --no-commit-id --name-only -r "$sha") + # Clear any leftover sequencer state (harmless if none exists). + git -C "$DELTA_DIR" cherry-pick --quit 2>/dev/null || true + return 0 } echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes" From 408637d82dfa6c2840f9488ede346f34883e0acc Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 27 Jul 2026 20:36:45 +0000 Subject: [PATCH 21/28] [CI] Track skipped Delta tests separately and make the clone step idempotent Two fixes from review feedback on the Delta gate. 1. compare-test-results.py treated a skipped test as "not seen this run", so a baseline entry that merely got skipped was reported under "Stale baseline entries (suite/test gone)" and silently dropped from the regenerated baseline -- only to return as a regression the next time it executed. Each shard now writes a --skipped-out list alongside --ran-out, and the aggregate job reports those entries as "Skipped this run" instead of stale and carries them over into the regenerated baseline. Genuinely removed tests appear in neither list and are still reported as stale. Skips are deliberately NOT folded into --ran-out: "now-passing" is derived from `ran - failed`, so counting a skip as a run would report it as fixed and, under fail_on_fixed, demand its removal from the baseline. Missing skipped-*.txt (older artifacts) degrades to the previous behaviour, and the shard-completeness guard still keys off failures-*/ran-* only. 2. setup-delta.sh could not be re-run over an existing DELTA_DIR: `git remote add origin` exits 3 when origin already exists, which aborts the script under `set -euo pipefail` and forces manual cleanup. Drop the remote first and force the checkout so a partial previous run is recovered rather than fatal. No `rm -rf` is reintroduced; the bundle jar and source patches are applied after this block, so nothing worth keeping is discarded. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../workflows/util/delta-spark-ut/README.md | 7 +++ .../delta-spark-ut/compare-test-results.py | 63 +++++++++++++++++-- .../util/delta-spark-ut/run-delta-tests.sh | 1 + .../util/delta-spark-ut/setup-delta.sh | 10 ++- 4 files changed, 74 insertions(+), 7 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 3111050f687..3f3d1a52bd4 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -59,6 +59,13 @@ A final `aggregate` job merges every shard's results into a single, sorted, ready-to-commit `known-failures.txt` artifact and reports **stale** baseline entries (tests no longer present in any shard, e.g. after a Delta version bump). +Tests reported as `` are tracked separately from tests that actually +ran, so a baseline entry that was merely skipped this run is **not** mistaken +for either a fix or a stale entry: it is listed under *"Skipped this run"* and +carried over into the regenerated baseline. (Folding skips into the run set +would report them as now-passing and — under `fail_on_fixed` — demand their +removal, only for them to return as regressions the next time they execute.) + Because Delta shards **by suite**, every suite (and therefore every test) runs in exactly one shard, so per-shard enforcement sees complete suites and never double-counts. diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index e3b6b9b2404..0e1036a946b 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -46,9 +46,12 @@ can be (re)generated from a real run. ``aggregate`` (final job) - Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single, - sorted, ready-to-commit ``known-failures.txt`` and report stale baseline - entries (tests no longer present in any shard). Pass ``--expected-shards N`` + Merge every shard's ``--failures-out`` / ``--ran-out`` / ``--skipped-out`` + file into a single, sorted, ready-to-commit ``known-failures.txt`` and report + stale baseline entries (tests no longer present in any shard). Skipped tests + are tracked apart from run tests so "stale" means *truly absent* rather than + merely skipped this run; they are also kept out of the now-passing set, since + a skipped test is not evidence of a fix. Pass ``--expected-shards N`` to fail when fewer than ``N`` shards contributed gate lists (a shard that died before writing them), so an incomplete baseline is never produced. @@ -454,6 +457,14 @@ def flaky_is(e): write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)}) if args.ran_out: write_entries(args.ran_out, passed | failed) + # Skipped tests are written separately rather than folded into --ran-out: the + # aggregate job derives "now-passing" from `ran - failed`, so counting a + # skipped test as "ran" would report it as fixed and (under fail_on_fixed) + # demand its removal from the baseline -- only for it to come back as a + # regression the next time it actually executes. Kept apart, skipped tests can + # still be excluded from the "stale" set, which is what they are not. + if args.skipped_out: + write_entries(args.skipped_out, skipped) write, handle = _summary_sink() try: @@ -575,6 +586,13 @@ def run_aggregate(args): ran_files = sorted( glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True) ) + # Optional: older gate-list artifacts predate --skipped-out, so a missing set + # of skipped-*.txt just degrades to the previous behaviour (no skip tracking) + # rather than failing the aggregation. Deliberately left out of the + # completeness guard below for the same reason. + skipped_files = sorted( + glob.glob(os.path.join(args.inputs_dir, "**", "skipped-*.txt"), recursive=True) + ) # No per-shard gate lists means the artifacts were never produced or the # download failed (the workflow's download step is continue-on-error). Bail @@ -619,13 +637,29 @@ def run_aggregate(args): union_ran = set() for f in ran_files: union_ran |= load_entries(f) + union_skipped = set() + for f in skipped_files: + union_skipped |= load_entries(f) + # A test that ran in any shard is not "skipped" overall. + union_skipped -= union_ran flaky_is = make_is_flaky(load_entries(args.flaky_tests)) + prev_baseline = ( + load_entries(args.known_failures) + if args.known_failures and os.path.exists(args.known_failures) + else set() + ) + # A known failure that was merely *skipped* this run produced no failure + # record, so regenerating purely from union_failed would silently drop it -- + # and it would come back as a regression the next time it actually executes. + # Carry those entries over; genuinely removed tests appear in neither + # union_ran nor union_skipped and are still dropped (reported as stale). + carried_skipped = prev_baseline & union_skipped # Exclude quarantined flaky tests from the regenerated baseline: a flaky test # that happened to fail this run must never be baked into known-failures.txt # (otherwise it would trip the now-passing gate on the next run where it # passes). Flaky failures are tracked in flaky-tests.txt, not the baseline. - baseline_body = {e for e in union_failed if not flaky_is(e)} + baseline_body = {e for e in (union_failed | carried_skipped) if not flaky_is(e)} header = ( "# Known Delta-on-Gluten unit test failures.\n" @@ -652,7 +686,7 @@ def run_aggregate(args): exit_code = 0 if args.known_failures and os.path.exists(args.known_failures): - baseline = load_entries(args.known_failures) + baseline = prev_baseline if baseline: regressions = {e for e in (union_failed - baseline) if not flaky_is(e)} quarantined = {e for e in (union_failed - baseline) if flaky_is(e)} @@ -661,7 +695,8 @@ def run_aggregate(args): for e in (baseline & (union_ran - union_failed)) if not flaky_is(e) } - stale = baseline - union_ran + stale = baseline - union_ran - union_skipped + skipped_baseline = baseline & union_skipped write("| Baseline entries | {} |".format(len(baseline))) write("| Regressions (global) | {} |".format(len(regressions))) write("| Now-passing (global) | {} |".format(len(fixed))) @@ -670,6 +705,11 @@ def run_aggregate(args): len(quarantined) ) ) + write( + "| Skipped this run (kept in baseline) | {} |".format( + len(skipped_baseline) + ) + ) write("| Stale (not seen this run) | {} |".format(len(stale))) _print_block(write, "Regressions (global)", regressions) _print_block(write, "Now-passing (global)", fixed) @@ -678,6 +718,11 @@ def run_aggregate(args): "Quarantined flaky failures -- ignored (flaky-tests.txt + flaky-error-patterns.txt)", quarantined, ) + _print_block( + write, + "Skipped this run -- NOT stale, leave them in the baseline", + skipped_baseline, + ) _print_block(write, "Stale baseline entries (suite/test gone)", stale) if args.fail_on_regression and regressions: exit_code = 1 @@ -729,6 +774,12 @@ def main(argv=None): parser.add_argument( "--ran-out", help="Write this shard's run tests (pass+fail) here." ) + parser.add_argument( + "--skipped-out", + help="Write this shard's skipped tests here. Kept separate from --ran-out " + "so the aggregate job can tell 'skipped this run' from 'gone' without " + "mistaking a skip for a fix.", + ) parser.add_argument( "--fail-on-fixed", type=str2bool, diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh index 79238505a43..d42b4151a31 100755 --- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -271,4 +271,5 @@ python3 "$UTIL_DIR/compare-test-results.py" \ --flaky-error-patterns "$UTIL_DIR/flaky-error-patterns.txt" \ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${SHARD_ID}.txt" \ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${SHARD_ID}.txt" \ + --skipped-out "$GITHUB_WORKSPACE/gate-out/skipped-shard-${SHARD_ID}.txt" \ --fail-on-fixed "${FAIL_ON_FIXED}" diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 5cc216b79f8..73deec8d6ad 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -64,10 +64,18 @@ echo "::group::Cloning delta-io/delta @ ${DELTA_REF}" # destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a # DELTA_REF starting with `-` can't be misread as a git flag (this script is # workflow_dispatch-runnable with a user-supplied ref). +# +# Every step here is idempotent so a local re-run (or a CI re-run on a runner +# that kept the workspace) resumes instead of dying: `git init` re-initializes +# an existing repo harmlessly, but `remote add` errors out when `origin` already +# exists, so drop it first; and `checkout -f` discards leftovers from a previous +# partial run. Nothing worth keeping exists here yet -- the bundle jar and the +# source patches below are applied *after* this block. git init -q "$DELTA_DIR" +git -C "$DELTA_DIR" remote remove origin 2>/dev/null || true git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF" -git -C "$DELTA_DIR" checkout -q FETCH_HEAD +git -C "$DELTA_DIR" checkout -qf FETCH_HEAD git -C "$DELTA_DIR" --no-pager log -1 --oneline echo "::endgroup::" From 63c46918ddf4e4b261786b281af37212fbaa626e Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 27 Jul 2026 21:39:25 +0000 Subject: [PATCH 22/28] [CI] Report invalid flaky-error regexes clearly and tighten the DV pattern Follow-ups from review feedback on the Delta gate. - load_patterns() compiled each line of flaky-error-patterns.txt with no error handling, so a typo in that hand-edited file surfaced as a bare re.error traceback naming neither the file nor the offending line. Raise BadPatternError with path, line number and pattern, and exit 2 from the gate: ":4: invalid regex '[unclosed': unterminated character set at position 0". - The negative-row-index signature used `-?\d+`, which would also match a positive index. The native check is `value >= 0`, so the reported index is always negative; match the sign explicitly, per this file's own rule of binding patterns to the exact error string so a quarantine can't swallow an unrelated failure. - Note in the DeletionVectorsSuite block that the sed depends on the clone step's `checkout -f`. The sed appends after the test-declaration line, so without that per-run reset a re-run injects duplicate `fail` lines and trips the INJECTED != 2 check (verified: 2, 4, 6 without `-f`; 2, 2, 2 with it). Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../delta-spark-ut/compare-test-results.py | 27 ++++++++++++++++--- .../delta-spark-ut/flaky-error-patterns.txt | 5 +++- .../util/delta-spark-ut/setup-delta.sh | 5 +++- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/compare-test-results.py b/.github/workflows/util/delta-spark-ut/compare-test-results.py index 0e1036a946b..438029ebae8 100644 --- a/.github/workflows/util/delta-spark-ut/compare-test-results.py +++ b/.github/workflows/util/delta-spark-ut/compare-test-results.py @@ -116,6 +116,14 @@ class CorruptReportError(NoReportsError): """ +class BadPatternError(RuntimeError): + """Raised when flaky-error-patterns.txt contains an uncompilable regex. + + Hand-edited file, so a typo is plausible; without this the bare re.error + surfaces as a traceback that names neither the file nor the offending line. + """ + + SEP = "#" @@ -215,16 +223,25 @@ def load_patterns(path): (its JUnit / message + stack), so a test that fails with a known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug) can be quarantined by root cause instead of by exact test name. + + Raises BadPatternError (naming the file, line number and pattern) if a line + is not a valid regex, so a typo fails the gate with an actionable message + rather than an opaque traceback. """ patterns = [] if not path or not os.path.exists(path): return patterns with open(path, encoding="utf-8") as fh: - for line in fh: + for lineno, line in enumerate(fh, start=1): line = line.rstrip("\n") if not line.strip() or line.lstrip().startswith("#"): continue - patterns.append(re.compile(line)) + try: + patterns.append(re.compile(line)) + except re.error as exc: + raise BadPatternError( + "{}:{}: invalid regex {!r}: {}".format(path, lineno, line, exc) + ) return patterns @@ -432,7 +449,11 @@ def run_enforce(args): return 2 baseline = load_entries(args.known_failures) name_flaky = make_is_flaky(load_entries(args.flaky_tests)) - sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns)) + try: + sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns)) + except BadPatternError as exc: + eprint("ERROR: {}".format(exc)) + return 2 try: passed, failed, skipped, fail_texts = parse_reports(args.reports_dir) except NoReportsError as exc: diff --git a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt index 23cf641aa99..dcac80e3dc1 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt @@ -40,4 +40,7 @@ # too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue Delta RoaringBitmapArray row index \d+ exceeds max representable value # negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0 -Delta bitmap row index cannot be negative: -?\d+ +# The check is `value >= 0`, so the reported index is always negative: match the +# sign explicitly rather than `-?` so this can't quarantine an unrelated failure +# if the message format ever changes. +Delta bitmap row index cannot be negative: -\d+ diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh b/.github/workflows/util/delta-spark-ut/setup-delta.sh index 73deec8d6ad..4295c33420c 100755 --- a/.github/workflows/util/delta-spark-ut/setup-delta.sh +++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh @@ -188,7 +188,10 @@ echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests" # # ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits # DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work -# tree has uncommitted edits to a file it touches. +# tree has uncommitted edits to a file it touches. It also relies on the clone +# step's `checkout -f`: the sed appends after the declaration line, so without +# that per-run reset a re-run injects duplicate `fail` lines and trips the +# INJECTED != 2 check below. DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" if [ ! -f "$DVS" ]; then echo "Expected file not found in Delta clone: $DVS" >&2 From a87892f70a29a39a9bc3d89af3e762f607330b5b Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Mon, 27 Jul 2026 23:41:35 +0000 Subject: [PATCH 23/28] [CI] Make the Delta path gate fail open when git diff fails The delta-changes job intends to run the Delta suite whenever it cannot determine what the PR touched -- "never silently skip coverage". But the detection itself was fail-closed: if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq ''; then When git diff fails (missing objects after a force-push race, an unfetched fork head, or the `git merge-base ... || echo "$BASE_SHA"` fallback above handing it an unusable sha), grep gets empty input and exits 1 -- exactly as it does for "no Delta paths changed" -- so the else branch set run_delta=false and the suite was skipped. `set -euo pipefail` does not help here: a command used as an `if` condition is allowed to fail. Capture the diff first and treat a git failure as an explicit fail-open, then match against the captured output. Verified over 8 scenarios: bad base+head, valid base + bad head and empty shas now all yield run_delta=true, while gluten-delta/, backends-velox/src-delta40, docs-only, cpp/velox-only and no-change diffs are unchanged. The previous logic fails the first three. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/velox_backend_x86.yml | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index cd53a4fc886..c060f473448 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -144,13 +144,23 @@ jobs: fi BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA") echo "diff base=$BASE head=$HEAD_SHA" + # Fail open if the diff itself can't be computed (missing objects after a + # force-push race, an unfetched fork head, ...). Piping straight into + # `grep -q` inside an `if` would hide that: git's failure leaves grep with + # empty input, so the pipeline exits non-zero exactly as it does for "no + # match" -- and `set -e`/`pipefail` can't help, since a tested command is + # allowed to fail. Capture the diff first so the two cases stay distinct. + if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then + echo "git diff failed -> running Delta suite (fail-open)" + echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 + fi # High-signal Delta paths only: the Delta integration code # (backends-velox/src-delta*), the Delta module, and this pipeline's own # files. A change to general Velox/core/native code can also affect Delta # offload, but those are touched constantly; per-PR we skip them (the # nightly full run + the `run-delta-ci` label are the safety nets) to # keep GHA usage down. - if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq \ + if printf '%s\n' "$CHANGED" | grep -Eq \ '^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)'; then echo "Delta-relevant paths changed -> running Delta suite" echo "run_delta=true" >> "$GITHUB_OUTPUT" From 69fe6f922724a9bbbef5ce22879db6a23ed163ca Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 28 Jul 2026 17:40:10 +0000 Subject: [PATCH 24/28] [CI] Baseline new ImplicitMergeCasting BIGINT->DECIMAL overflow failure Shard 2 of run 30324192115 reports one regression: ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY Under storeAssignmentPolicy LEGACY the overflowing value is expected to be stored without error, but the offloaded Velox cast raises instead: VeloxUserError INVALID_ARGUMENT Reason: Cannot cast BIGINT '9223372036854775807' to DECIMAL(7, 2) Function: rescaleInt File: velox/type/DecimalUtil.h:218 This is the non-streaming sibling of the ImplicitStreamingMergeCastingSuite case for the same BIGINT -> DECIMAL(7,2) / LEGACY combination that was baselined earlier; both surfaced after the rebase. It went unnoticed one run longer because shard 2 of run 30186097028 died on a Docker pull before the gate ran, so the test never executed there. Baselined rather than quarantined in flaky-tests.txt: it failed in both runs where it actually executed, with an identical error, and the failure is a pure expression-evaluation overflow with no dependence on scheduling or runtime plan -- unlike the DV bitmap row-index bug that flaky-error-patterns.txt covers. Keeping it in the baseline means the gate will tell us to remove it once the LEGACY cast semantics are fixed, which a flaky entry would not. 736 -> 737 entries. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/util/delta-spark-ut/known-failures.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index dbf2108cf0d..883d7522548 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -324,6 +324,7 @@ org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATC org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: ARRAY, targetType: ARRAY followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: ANSI +org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: ANSI org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: true, ansiEnabled: true, storeAssignmentPolicy: LEGACY org.apache.spark.sql.delta.ImplicitMergeCastingSuite#MERGE overflow in WHEN MATCHED THEN UPDATE SET t.value = s.value sourceType: BIGINT, targetType: INT followAnsiEnabled: false, ansiEnabled: false, storeAssignmentPolicy: ANSI From 3fe660a79881eec9a02152699318fba36ab17bb0 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Tue, 28 Jul 2026 23:21:02 +0000 Subject: [PATCH 25/28] [CI] Make the Delta hang-watchdog done marker shard-specific run-delta-tests.sh is parameterized by SHARD_ID and already scopes its sbt log and watchdog kill marker per shard, but the marker that tells the watchdog sbt has finished was the global /tmp/sbt-done. With a shared /tmp -- parallel local runs of several shards -- the first shard to finish creates it and every other shard's watchdog exits its wait loop early, silently losing hang detection for the shards still running. Use /tmp/sbt-done-shard-${SHARD_ID} via a named SBT_DONE_MARKER, matching the existing SBT_LOG / WATCHDOG_KILL_MARKER convention. Verified with a reproduction of the arm/disarm handshake, running a fast shard alongside a slow one: with the global marker the slow shard's watchdog was disarmed after 3 ticks instead of its full 10; with the shard-scoped marker it stays armed for the whole run. CI is unaffected (each shard is its own container), so this only matters for local parallel runs. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .../workflows/util/delta-spark-ut/run-delta-tests.sh | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh index d42b4151a31..b5193e624a1 100755 --- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh +++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh @@ -94,8 +94,14 @@ SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log" # report, and since we ignore sbt's exit code the gate would only judge the # suites that DID report -- so the main flow fails the shard when this exists. WATCHDOG_KILL_MARKER="/tmp/sbt-watchdog-killed-shard-${SHARD_ID}" +# Marker the main flow touches when sbt returns, to stop this shard's watchdog. +# Shard-scoped like the log and kill marker above: shards get their own container +# in CI, but a shared /tmp (parallel local runs) would otherwise let the first +# shard to finish disarm every other shard's watchdog and silently lose hang +# detection for them. +SBT_DONE_MARKER="/tmp/sbt-done-shard-${SHARD_ID}" : > "$SBT_LOG" -rm -f /tmp/sbt-done "$WATCHDOG_KILL_MARKER" +rm -f "$SBT_DONE_MARKER" "$WATCHDOG_KILL_MARKER" ( # CRITICAL: the step shell runs with `bash -eo pipefail`, which the # subshell inherits. Without `set +e` here, ANY non-zero command -- @@ -132,7 +138,7 @@ rm -f /tmp/sbt-done "$WATCHDOG_KILL_MARKER" } echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence" hb=0 - while [ ! -f /tmp/sbt-done ]; do + while [ ! -f "$SBT_DONE_MARKER" ]; do sleep 60 [ -f "$SBT_LOG" ] || continue now=$(date +%s) @@ -219,7 +225,7 @@ WATCHDOG_PID=$! 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \ "spark/test" 2>&1 | tee "$SBT_LOG" SBT_EXIT=${PIPESTATUS[0]} -touch /tmp/sbt-done +touch "$SBT_DONE_MARKER" kill "$WATCHDOG_PID" 2>/dev/null || true set -e echo "sbt spark/test exited with ${SBT_EXIT}" From bc904eb1c03064f922828f90a46f3f63021e75ca Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 30 Jul 2026 18:58:29 +0000 Subject: [PATCH 26/28] [CI] Run the Delta Spark UT as its own workflow The Delta suite ran as a reusable workflow called by velox_backend_x86.yml, so its own pipeline files had to be in that workflow's `paths:` filter for a change to them to be tested. The side effect was that a Delta-CI-only change pulled in the entire Velox matrix: on this branch's last run, a commit touching a single shell script produced 65 jobs and 2319 runner-minutes, of which 1750 (75%) were TPC-H/DS and Spark UT jobs that the change could not affect. Make delta_spark_ut.yml standalone. It already built its own native library for the workflow_dispatch/schedule path, so this mostly means dropping `workflow_call` and the `native_lib_artifact` input, adding a `pull_request` trigger with a Delta `paths:` filter, and deleting the `delta-changes` gate and `delta-spark-ut` jobs from velox_backend_x86.yml along with the Delta entries in its `paths:`. The `paths:` filter now IS the per-PR gate, evaluated before the run is created, so an unrelated PR costs nothing at all -- this replaces the ~60-line `delta-changes` script. That also drops the `run-delta-ci` label opt-in, which cannot be expressed as a path filter: the label does not exist on apache/gluten and so was never functional, and `workflow_dispatch` covers the same need (a Velox/core author can run the suite against a branch, including on a fork). Tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` still match velox_backend_x86.yml's filter -- it builds with -Pdelta -- so a change there now runs both workflows and builds the native library twice (~10 min) instead of sharing it. That is small next to the case above, and sharing it again is a separate change. Two conditions had to move with the split: - `concurrency:` is now set. It was deliberately absent because, as a reusable workflow, `github.workflow` resolved to the caller and the group would have collided with the caller's own. Standalone it is needed, or every push to a Delta PR stacks another full run. - delta-spark-aggregate no longer uses `always()`. That evaluates true while a run is being cancelled, so with `cancel-in-progress` a superseded push would cancel the shards but still start the aggregation, which then fails on finding no gate lists; the same happened when a failed bundle build skipped the shards. `!cancelled() && needs.delta-spark-test.result != 'skipped'` keeps the intended behaviour of publishing a baseline when shards go red, without the spurious second failure. Also fix FAIL_ON_FIXED, which the split would otherwise have silently flipped: `inputs` now only exists for workflow_dispatch, so the old expression resolved to false on pull_request and schedule, where the removed workflow_call input had supplied `default: true`. It now defaults to true for those events and honours the input only on workflow_dispatch. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/delta_spark_ut.yml | 132 +++++++++--------- .../workflows/util/delta-spark-ut/README.md | 38 +++-- .github/workflows/velox_backend_x86.yml | 79 ----------- 3 files changed, 89 insertions(+), 160 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 580cb95fbd9..83c56594f78 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -32,43 +32,31 @@ name: Delta Spark UT (Gluten) on: - # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant - # changes) and passes the native-lib artifact it already built, so the expensive - # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a - # called workflow runs as part of the caller run), so the jobs below download it - # by name. See velox_backend_x86.yml `delta-spark-ut`. + # Self-contained: this workflow owns the whole Delta pipeline, including its own + # native build, and is NOT called by velox_backend_x86.yml. A change confined to + # this pipeline's own files no longer drags in that workflow's ~50-job + # TPC-H/DS + Spark-UT matrix (measured: ~1750 runner-minutes per run that such a + # change cannot affect), and a Velox/core change no longer carries the Delta + # suite. # - # NOTE: the `pull_request` trigger was removed so this no longer runs as its own - # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml - # is now the single PR entry point; `workflow_dispatch` keeps manual standalone - # runs working (those build the native lib themselves -- see build-native-lib). - workflow_call: - inputs: - native_lib_artifact: - description: 'Name of the cpp/build artifact uploaded by the caller' - type: string - required: true - delta_ref: - type: string - required: false - default: 'v4.2.0' - spark_version: - description: 'Spark version driving both the Gluten bundle profile (-Pspark-) and Delta -DsparkVersion.' - type: string - required: false - default: '4.1' - test_parallelism: - type: string - required: false - default: '4' - update_baseline: - type: boolean - required: false - default: false - fail_on_fixed: - type: boolean - required: false - default: true + # Note the tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` also + # match velox_backend_x86.yml's own filter (its spark-ut jobs build with + # -Pdelta), so a change there runs BOTH workflows and, now that the native lib + # is no longer shared between them, pays for the centos-7 build twice (~10 min). + # That is the price of decoupling, and it is small next to the case above. + # + # The `paths:` filter below IS the per-PR gate -- GitHub evaluates it before the + # run is created, so a non-Delta PR costs nothing at all. Changes to general + # Velox/core/native code can also affect Delta offload, but they are touched on + # most PRs; the nightly `schedule` run below is the safety net for those, and + # `workflow_dispatch` lets anyone run the suite against a branch on demand. + pull_request: + paths: + - '.github/workflows/delta_spark_ut.yml' + - '.github/workflows/util/delta-spark-ut/**' + - 'gluten-delta/**' + # Covers src-delta, src-delta33, src-delta40 and any future variant. + - 'backends-velox/src-delta*/**' workflow_dispatch: inputs: delta_ref: @@ -93,12 +81,11 @@ on: type: boolean required: false default: true - # Nightly full run against the latest default branch. The per-PR entry point - # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches - # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this - # scheduled run keeps full coverage once a day so rarer regressions are still - # caught. It builds its own native lib (build-native-lib-centos-7 below) since - # there is no caller to provide one, and uses the workflow's default inputs. + # Nightly full run against the latest default branch. Per-PR the `paths:` filter + # above runs the suite only for Delta-relevant changes, to save GHA minutes; + # this scheduled run keeps full coverage once a day so regressions from general + # Velox/core changes are still caught, and it enforces `fail_on_fixed` so the + # baseline stays honest. schedule: - cron: '0 5 * * *' @@ -132,19 +119,19 @@ env: # force-failed in setup-delta.sh. DELTA_NUM_SHARDS: '4' -# No `concurrency:` here on purpose. As a reusable workflow this runs inside the -# caller's run, where `github.workflow` resolves to the CALLER's name -- a group -# keyed on it would collide with the caller's own group and, with -# cancel-in-progress, could cancel the parent run. The caller's concurrency -# already governs cancellation. (A standalone workflow_dispatch run just won't -# auto-cancel, which is fine for infrequent manual runs.) +# Now that this is a standalone workflow (not called by velox_backend_x86.yml), +# `github.workflow` resolves to THIS workflow, so a concurrency group is both safe +# and necessary: without it every push to a PR branch stacks another full ~2.5 h +# Delta run instead of superseding the previous one. Keyed on the branch for PRs +# and the sha otherwise, matching velox_backend_x86.yml's group. +concurrency: + group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} + cancel-in-progress: true jobs: build-native-lib-centos-7: - # Standalone runs (workflow_dispatch + nightly schedule) build the native lib - # here. When called by velox_backend_x86.yml the caller already built it and - # passes it as an input, so this job is skipped and the duplicate build avoided. - if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' + # This workflow always builds its own native lib -- there is no caller to + # provide one, and the `paths:` filter already decided whether the run happens. runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 @@ -184,9 +171,6 @@ jobs: build-gluten-bundle: needs: build-native-lib-centos-7 - # Run whether the native lib was built here (dispatch -> success) or provided - # by the caller (workflow_call -> build-native-lib-centos-7 skipped). - if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }} runs-on: ubuntu-22.04 container: apache/gluten:centos-9-jdk17 steps: @@ -194,7 +178,7 @@ jobs: - name: Download native artifacts uses: actions/download-artifact@v4 with: - name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }} + name: delta-spark-ut-native-lib-centos-7-${{ github.sha }} path: ./cpp/build/ - name: Cache Maven repository uses: actions/cache@v4 @@ -246,10 +230,9 @@ jobs: delta-spark-test: needs: build-gluten-bundle - # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need - # is skipped on workflow_call), so this job needs an explicit condition too -- - # otherwise GitHub's transitive skip propagation, seeing the skipped - # build-native-lib-centos-7 ancestor, would skip the whole shard matrix. + # Explicit condition rather than relying on default skip propagation: it keeps + # the matrix bounded to a genuinely successful bundle, and stays correct if the + # upstream jobs' conditions change again. if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }} runs-on: ubuntu-22.04 container: apache/gluten:centos-9-jdk17 @@ -270,19 +253,24 @@ jobs: - name: Resolve workflow inputs id: resolve - # Surface the inputs as step outputs. workflow_call / workflow_dispatch - # supply them (with defaults); the nightly `schedule` event supplies NONE, - # so fall back to the same defaults here. The boolean inputs are rendered - # as explicit 'true'/'false' strings (never empty) via the `&&/||` form so - # a schedule run resolves cleanly: update_baseline=false (enforce) and + # Surface the inputs as step outputs. `workflow_dispatch` supplies them + # (with defaults); the `pull_request` and nightly `schedule` events supply + # NONE, so fall back to the same defaults here. The boolean inputs are + # rendered as explicit 'true'/'false' strings (never empty) via the `&&/||` + # form so those runs resolve cleanly: update_baseline=false (enforce) and # fail_on_fixed=true (so a now-passing baseline test turns the nightly red # -- our signal that the committed baseline needs refreshing). env: DELTA_REF: ${{ inputs.delta_ref || 'v4.2.0' }} SPARK_VERSION: ${{ inputs.spark_version || '4.1' }} TEST_PARALLELISM: ${{ inputs.test_parallelism || '4' }} - UPDATE_BASELINE: ${{ inputs.update_baseline && 'true' || 'false' }} - FAIL_ON_FIXED: ${{ github.event_name == 'schedule' && 'true' || (inputs.fail_on_fixed && 'true' || 'false') }} + UPDATE_BASELINE: ${{ github.event_name == 'workflow_dispatch' && (inputs.update_baseline && 'true' || 'false') || 'false' }} + # Defaults to true for pull_request and schedule: `inputs` only exists on + # workflow_dispatch, so a bare `inputs.fail_on_fixed` would silently + # resolve to false on those events and stop the now-passing check from + # being enforced -- the behaviour the removed workflow_call input used to + # provide via `default: true`. + FAIL_ON_FIXED: ${{ github.event_name == 'workflow_dispatch' && (inputs.fail_on_fixed && 'true' || 'false') || 'true' }} run: | set -euo pipefail { @@ -429,7 +417,15 @@ jobs: # to bootstrap or refresh the baseline (see util/delta-spark-ut/README.md). delta-spark-aggregate: needs: delta-spark-test - if: always() + # Runs even when shards go red -- a partially-failing run still produces a + # useful refreshed baseline. Deliberately NOT `always()`: that also evaluates + # true while a run is being cancelled, and with `cancel-in-progress` above a + # superseded push would cancel the shards but still start this job, which then + # fails hard (by design) on finding no per-shard gate lists. `!cancelled()` + # drops that case, and the `!= 'skipped'` check drops the other one -- shards + # skipped because the bundle build failed -- so a single upstream failure + # doesn't surface as a second, misleading red job. + if: ${{ !cancelled() && needs.delta-spark-test.result != 'skipped' }} runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 3f3d1a52bd4..c473c4aa9b8 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -72,24 +72,36 @@ double-counts. ## When it runs -To keep GitHub Actions usage in check, the suite does **not** run on every PR: - -- **Per PR** — `velox_backend_x86.yml` runs the Delta suite only when the PR +To keep GitHub Actions usage in check, the suite does **not** run on every PR. +It lives in its own workflow (`delta_spark_ut.yml`) rather than as part of +`velox_backend_x86.yml`, so a change confined to this pipeline's own files no +longer drags in that workflow's ~50-job TPC-H/DS + Spark-UT matrix (~1750 +runner-minutes per run that such a change cannot affect), and a Velox/core change +no longer carries the Delta suite. + +The tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` also match +`velox_backend_x86.yml`'s own filter (its spark-ut jobs build with `-Pdelta`), so +a change there runs **both** workflows — and because the native library is no +longer shared between them, those PRs pay for the centos-7 native build twice +(~10 min). That is the price of decoupling the two pipelines. + +- **Per PR** — the workflow's own `paths:` filter runs the suite only when the PR touches a **high-signal Delta path**: the Delta integration code (`backends-velox/src-delta*`), the `gluten-delta` module, or this pipeline's - own files (`delta_spark_ut.yml`, `util/delta-spark-ut/**`, - `velox_backend_x86.yml`). Changes to general Velox/core/native code can also - affect Delta offload, but they're touched on most PRs, so per-PR they skip the - suite — the nightly run and the opt-in label are the safety nets. Add the - **`run-delta-ci`** label to force the suite on any PR (the label is read from - the triggering event, so apply it before/with a push). -- **Nightly** — `delta_spark_ut.yml` runs the **full** suite against the latest - default branch on a `schedule` (05:00 UTC), so rarer regressions are still - caught daily. The nightly run enforces the baseline **and** fails on + own files (`delta_spark_ut.yml`, `util/delta-spark-ut/**`). GitHub evaluates + the filter before creating the run, so an unrelated PR costs nothing at all. + Changes to general Velox/core/native code can also affect Delta offload, but + they're touched on most PRs, so per-PR they skip the suite — the nightly run is + the safety net. +- **Nightly** — the **full** suite runs against the latest default branch on a + `schedule` (05:00 UTC), so regressions from general Velox/core changes are + still caught daily. The nightly run enforces the baseline **and** fails on now-passing tests (`fail_on_fixed=true`), so baseline drift surfaces as a red nightly — the signal to refresh `known-failures.txt`. - **Manually** — **Actions → Delta Spark UT (Gluten) → Run workflow** - (`workflow_dispatch`), e.g. to refresh the baseline (see below). + (`workflow_dispatch`), e.g. to refresh the baseline (see below). This is also + how you validate a Velox/core change against Delta before merging: run it on + your branch (on your fork if you don't have write access here). ## Bootstrapping the baseline (first time) diff --git a/.github/workflows/velox_backend_x86.yml b/.github/workflows/velox_backend_x86.yml index c060f473448..2e34f3084b5 100644 --- a/.github/workflows/velox_backend_x86.yml +++ b/.github/workflows/velox_backend_x86.yml @@ -19,11 +19,6 @@ on: pull_request: paths: - '.github/workflows/velox_backend_x86.yml' - # Delta Spark UT runs here too (reusable delta_spark_ut.yml). These extra - # paths make Delta-CI-only changes trigger this workflow; Delta also runs on - # the velox paths below since core/velox changes can affect Delta offload. - - '.github/workflows/delta_spark_ut.yml' - - '.github/workflows/util/delta-spark-ut/**' - '.github/workflows/util/install-spark-deps.sh' #TODO remove after image update - '.github/workflows/util/install-spark-resources.sh' #TODO remove after image update - 'pom.xml' @@ -106,80 +101,6 @@ jobs: path: ./cpp/build/ if-no-files-found: error - # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR - # touches high-signal Delta paths -- the Delta integration code - # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own - # files -- or carries the `run-delta-ci` opt-in label. Changes to general - # Velox/core/native code can also affect Delta offload but are touched - # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml - # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage - # down. NOTE: the label is read from the event that triggered this run, so add - # it before/with a push; labeling an already-finished run needs a new push. - delta-changes: - runs-on: ubuntu-22.04 - outputs: - run_delta: ${{ steps.filter.outputs.run_delta }} - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: Detect Delta-relevant changes / opt-in label - id: filter - env: - HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }} - BASE_SHA: ${{ github.event.pull_request.base.sha }} - HEAD_SHA: ${{ github.event.pull_request.head.sha }} - run: | - set -euo pipefail - # Opt-in label forces the suite even with no Delta-relevant path change. - if [ "$HAS_LABEL" = "true" ]; then - echo "run-delta-ci label present -> running Delta suite" - echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 - fi - # Fail open if we can't determine the PR range (e.g. a non-PR trigger): - # never silently skip coverage. - if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then - echo "no PR base/head sha -> running Delta suite (fail-open)" - echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 - fi - BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA") - echo "diff base=$BASE head=$HEAD_SHA" - # Fail open if the diff itself can't be computed (missing objects after a - # force-push race, an unfetched fork head, ...). Piping straight into - # `grep -q` inside an `if` would hide that: git's failure leaves grep with - # empty input, so the pipeline exits non-zero exactly as it does for "no - # match" -- and `set -e`/`pipefail` can't help, since a tested command is - # allowed to fail. Capture the diff first so the two cases stay distinct. - if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then - echo "git diff failed -> running Delta suite (fail-open)" - echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0 - fi - # High-signal Delta paths only: the Delta integration code - # (backends-velox/src-delta*), the Delta module, and this pipeline's own - # files. A change to general Velox/core/native code can also affect Delta - # offload, but those are touched constantly; per-PR we skip them (the - # nightly full run + the `run-delta-ci` label are the safety nets) to - # keep GHA usage down. - if printf '%s\n' "$CHANGED" | grep -Eq \ - '^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)'; then - echo "Delta-relevant paths changed -> running Delta suite" - echo "run_delta=true" >> "$GITHUB_OUTPUT" - else - echo "No Delta-relevant paths changed and no opt-in label -> skipping Delta suite" - echo "run_delta=false" >> "$GITHUB_OUTPUT" - fi - - # Run the Delta Spark UT via the reusable workflow, passing the native lib - # built above so it is not rebuilt. Gated by `delta-changes` (Delta-relevant - # paths or the `run-delta-ci` label); the nightly full run lives in - # delta_spark_ut.yml's `schedule` trigger. - delta-spark-ut: - needs: [build-native-lib-centos-7, delta-changes] - if: ${{ needs.delta-changes.outputs.run_delta == 'true' }} - uses: ./.github/workflows/delta_spark_ut.yml - with: - native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }} - tpc-test-ubuntu: needs: build-native-lib-centos-7 strategy: From 88ab5dfc1cf317d295511d9204b1b60b89126233 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 30 Jul 2026 19:37:01 +0000 Subject: [PATCH 27/28] [CI] Split the Delta test matrix into 8 shards Each Delta shard was taking ~2.5 hours, which sets the wall clock for the whole pipeline. Breaking down one shard of a 4-shard run: 9.1 min of fixed setup (clone Delta, apply the patches, compile the test sources) and 134.5 min actually running tests, so ~91% of a shard is work that shards away. Doubling to 8 shards therefore takes the slowest shard from ~148 min to ~74 min for about +7% total runner-minutes, since the fixed setup is paid 8 times instead of 4. This also matches delta-io/delta's own spark_test.yaml, which runs these same suites with NUM_SHARDS: 8, shard: [0..7] and TEST_PARALLELISM_COUNT=4. Memory is unaffected: each shard is a separate job, so it is still 4 forked test JVMs plus the sbt launcher against the runner's ~16G, and TEST_PARALLELISM_COUNT stays at 4. No baseline regeneration is needed. The gate compares (suite, test) sets against known-failures.txt -- regressions as `failed - baseline`, now-passing as `baseline & passed`, stale as `baseline - ran - skipped` -- so it does not depend on how suites are distributed across shards. Sharding further would hit a floor at the longest single suite, currently ~18 min (DeleteSQLSQLPathBasedDVPredPushOffSuite); 8 shards stays well clear of it, and with ~180 suites per shard there is enough granularity for the split to balance. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/delta_spark_ut.yml | 32 ++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/.github/workflows/delta_spark_ut.yml b/.github/workflows/delta_spark_ut.yml index 83c56594f78..2d9c4aa7090 100644 --- a/.github/workflows/delta_spark_ut.yml +++ b/.github/workflows/delta_spark_ut.yml @@ -110,14 +110,28 @@ env: # Number of shards in the delta-spark-test matrix. Must equal the length of # the `shard` matrix below. # - # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4 - # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent - # runners for the same throughput. Sharding is by SUITE; total work - # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G - # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit; - # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is - # force-failed in setup-delta.sh. - DELTA_NUM_SHARDS: '4' + # 8 shards x TEST_PARALLELISM_COUNT=4 gives ~32-way parallelism, which is also + # exactly what delta-io/delta's own spark_test.yaml uses (NUM_SHARDS: 8, + # shard: [0..7], TEST_PARALLELISM_COUNT=4), so we size the split the same way + # upstream does for the same suites. + # + # Sharding is by SUITE and the total work is fixed, so this trades runner count + # for wall clock: measured on a 4-shard run, a shard spent 9.1 min on fixed + # setup (clone Delta, apply patches, compile the test sources) and 134.5 min + # actually running tests, so ~91% of a shard shards away. Going 4 -> 8 takes the + # slowest shard from ~148 min to ~74 min for about +7% total runner-minutes (the + # fixed setup is paid 8 times instead of 4). + # + # Memory is per runner and therefore unaffected: each shard is its own job, so + # it is still 4 forks plus the sbt launcher against the ~16G limit, at ~4G per + # fork (2G heap + 2G off-heap). That fits because the worst memory hog + # (DeletionVectorsSuite 2B-row) is force-failed in setup-delta.sh. + # + # Further sharding has a floor: wall clock cannot drop below the longest single + # suite, which is ~18 min (DeleteSQLSQLPathBasedDVPredPushOffSuite). At 8 shards + # we are well above it, and with ~180 suites per shard there is ample + # granularity for the split to stay balanced. + DELTA_NUM_SHARDS: '8' # Now that this is a standalone workflow (not called by velox_backend_x86.yml), # `github.workflow` resolves to THIS workflow, so a concurrency group is both safe @@ -243,7 +257,7 @@ jobs: fail-fast: false matrix: # Length of this list MUST equal env.DELTA_NUM_SHARDS. - shard: [0, 1, 2, 3] + shard: [0, 1, 2, 3, 4, 5, 6, 7] env: # Mirror Delta's spark_test.yaml env vars used by run-tests.py / # TestParallelization.scala. From 24cb3718296fe525a0f824447eb2e8b33075c5e2 Mon Sep 17 00:00:00 2001 From: Felipe Fujiy Pessoto Date: Thu, 30 Jul 2026 22:21:32 +0000 Subject: [PATCH 28/28] [CI] Drop two now-passing DeltaUpdateCatalogSuite tests from the baseline Shard 3 of the first 8-shard run went red on the now-passing check, not on a regression: DeltaUpdateCatalogSuite's "convert to delta with partitioning change" and "partitioned convert to delta with schema change" are in the baseline but now pass. The aggregate job agrees globally -- 0 regressions, 2 now-passing, 0 stale. Both tests previously failed with IllegalStateException: TaskResourceRegistry is not initialized from ColumnarCachedBatchSerializer -> RowToVeloxColumnarExec -> Runtimes.contextInstance, i.e. a task-listener setup problem on the cached-batch path, which depends on what else is running in the same forked test JVM. Resharding from 4 to 8 changed how suites are grouped into forks, and the interleaving that triggered it no longer occurs. This is a baseline update rather than a flaky quarantine: both tests failed in each of the last two 4-shard runs and have been in the baseline since it was bootstrapped, so there is no evidence of nondeterminism within a given configuration -- the outcome tracked the shard count. Verified by re-running the gate against that run's own gate-list artifacts with this baseline: all 8 shards report 0 regressions and 0 now-passing, and the aggregate's "distinct failing tests" (735) now matches the baseline exactly. Generated-by: GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66a9e40f-3ac8-45be-8fee-a606a22fa098 --- .github/workflows/util/delta-spark-ut/known-failures.txt | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt b/.github/workflows/util/delta-spark-ut/known-failures.txt index 883d7522548..bc55624d493 100644 --- a/.github/workflows/util/delta-spark-ut/known-failures.txt +++ b/.github/workflows/util/delta-spark-ut/known-failures.txt @@ -135,8 +135,6 @@ org.apache.spark.sql.delta.DeltaSuite#deleted files cause failure by default org.apache.spark.sql.delta.DeltaSuite#invalid replaceWhere org.apache.spark.sql.delta.DeltaSuite#replaceArbitrary should enforce proper usage of backtick org.apache.spark.sql.delta.DeltaTableCreationSuite#Default column values: CONVERT TO DELTA keeps EXISTS_DEFAULT -org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#convert to delta with partitioning change -org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#partitioned convert to delta with schema change org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - delete tombstones org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - update/merge org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skip deleted file