[MINOR][CI] Use spark-4.0 instead of the nonexistent spark-4 profile in the m2 cache image

[LuciferYang]

What changes are proposed in this pull request?

dev/docker/ubuntu/Dockerfile.ubuntu22-m2-cache pre-populates the Maven cache with one dependency:go-offline per supported Spark version. The stanza commented # Spark 4.0 passes -Pspark-4, which is not a profile id: the root pom defines spark-3.4, spark-3.5, spark-4.0 and spark-4.1. Maven prints The requested profile "spark-4" could not be activated because it does not exist., continues with no Spark profile active, and that stanza resolves against the root pom's default properties instead of Spark 4.0's.

enforce-spark-profile does not catch it. dependency:go-offline invokes the plugin goal directly rather than running a lifecycle, and that enforcer execution binds to validate, so it never fires. The command exits 0 after the warning, which is why this went unnoticed since #11655.

The defaults lean 3.5, so the stanza caches a mix:

property	-Pspark-4 (no profile active)	-Pspark-4.0
spark.version	4.0.2	4.0.2
delta.version	3.3.2	4.0.1
sparkshim.artifactId	spark-sql-columnar-shims-spark35	spark-sql-columnar-shims-spark40
sparkbundle.version	3.5	4.0

spark.version agrees by accident, because the root default is already 4.0.2, so the Spark 4.0 jars do land in the image; Delta 4.0.1 does not. To be accurate about the scope of that: nothing in this repo consumes apache/gluten:ubuntu22-m2-cache. docker_image.yml builds and pushes it, no workflow references it as a container:, and neither .devcontainer nor any script or doc mentions it. git log -S 'ubuntu22-m2-cache' --all returns only #11655, so it never had an in-repo consumer that was later dropped. The effect today is therefore confined to the published image: the stanza labelled # Spark 4.0 does not pre-populate what it says it does, for whoever pulls that image by hand. After this fix the stanza does real work, so the image build takes longer and the image grows by whatever Spark 4.0's Delta tree weighs. That is the intent.

Split out of #12902, which removes Spark 3.3 support and had to edit the same RUN for an unrelated reason (deleting the trailing Spark 3.3 stanza left a dangling ; \). This change adds behavior rather than removing it, so it is easier to review and to revert on its own.

How was this patch tested?

The property table is reproducible from the repo root, and it is the whole substance of the change:

for P in spark-4 spark-4.0; do
  for E in spark.version delta.version sparkshim.artifactId sparkbundle.version; do
    ./build/mvn -q -N -Pjava-17,scala-2.13,$P,backends-velox,hadoop-3.3,spark-ut \
      help:evaluate -Dexpression=$E -DforceStdout
  done
done

docker buildx build --check on the file reports the same single pre-existing NoEmptyContinuation warning before and after, so the edit changes nothing structural.

The image is built by docker_image.yml, which triggers on push to main under dev/docker/ubuntu/* and on a weekly cron, so the effect appears when that workflow runs rather than in PR CI.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-5

[LuciferYang]

cc @jackylee-ch

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[jackylee-ch]

Good Catch

[LuciferYang]

Thank you @jackylee-ch