[SPARK-52290][SQL][BUILD] Fix the Maven compilation and testing issues for sql/pipelines module

[LuciferYang]

What changes were proposed in this pull request?

The following modifications are mainly made in this PR:

1. Add the dependency org.scala-lang:scala-library to fix:

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.2:doc-jar (attach-scaladocs) on project spark-pipelines_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-pipelines_2.13

2. Add the dependency org.apache.spark:spark-core to fix:

*** RUN ABORTED ***
A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/spark/internal/config/Tests$
  java.lang.NoClassDefFoundError: org/apache/spark/internal/config/Tests$
  at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:94)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
  at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
  at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
  at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
  at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1324)
  at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
  at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
  ...
  Cause: java.lang.ClassNotFoundException: org.apache.spark.internal.config.Tests$
  at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
  at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:94)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
  at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
  at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
  at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)

Why are the changes needed?

Fix the Maven compilation and testing issues for sql/pipelines module, maven daily test failed after SPARK-52166

• https://github.com/apache/spark/actions/runs/15212144791
• https://github.com/apache/spark/actions/runs/15211074246
• https://github.com/apache/spark/actions/runs/15213443142
• https://github.com/apache/spark/actions/runs/15218341216

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass GitHub Actions
• Local check:

 build/mvn clean install -pl sql/pipelines -am -DskipTests
 build/mvn test -pl sql/pipelines

Run completed in 356 milliseconds.
Total number of tests run: 9
Suites: completed 3, aborted 0
Tests: succeeded 9, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

cc @gengliangwang @jonmio
also cc @dongjoon-hyun @zhengruifeng @HyukjinKwon

[dongjoon-hyun]

+1, LGTM. Thank you, @LuciferYang .

[LuciferYang]

Merge into master to see if the Maven daily tests have resumed. Thanks @dongjoon-hyun

[LuciferYang]

manually triggered an execution: https://github.com/apache/spark/actions/runs/15228255125

all passed:

[jonmio]

Thank you for fixing @LuciferYang