Skip to content

Commit 8d24e64

Browse files
salamonpavelsalamonpavel
authored
spark 3.4.x and 3.5.x support added
2 parents b4d622a + e97f8f5 commit 8d24e64

File tree

9 files changed

+47
-69
lines changed

9 files changed

+47
-69
lines changed

‎.github/workflows/jacoco_check.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ jobs:
3535
run: sbt jacoco
3636
- name: Add coverage to PR for scala 2.11 and spark 2.4
3737
id: jacoco211
38-
uses: madrapps/jacoco-report@v1.3
38+
uses: madrapps/jacoco-report@v1.7.2
3939
with:
40-
paths: ${{ github.workspace }}/spark-commons/target/spark2.4-jvm-2.11/jacoco/report/jacoco.xml
40+
paths: ${{ github.workspace }}/spark-commons/spark2.4-jvm-2.11/jacoco/report/jacoco.xml
4141
token: ${{ secrets.GITHUB_TOKEN }}
4242
min-coverage-overall: 0.0
4343
min-coverage-changed-files: 80.0
@@ -49,9 +49,9 @@ jobs:
4949
echo "Changed Files coverage ${{ steps.jacoco211.outputs.coverage-changed-files }}"
5050
- name: Add coverage to PR for scala 2.12 and spark 3.3
5151
id: jacoco212
52-
uses: madrapps/jacoco-report@v1.3
52+
uses: madrapps/jacoco-report@v1.7.2
5353
with:
54-
paths: ${{ github.workspace }}/spark-commons/target/spark3.3-jvm-2.12/jacoco/report/jacoco.xml
54+
paths: ${{ github.workspace }}/spark-commons/spark3-jvm-2.12/jacoco/report/jacoco.xml
5555
token: ${{ secrets.GITHUB_TOKEN }}
5656
min-coverage-overall: 0.0
5757
min-coverage-changed-files: 80.0

‎README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,25 @@
44
[![Build](https://github.com/AbsaOSS/spark-commons/actions/workflows/build.yml/badge.svg)](https://github.com/AbsaOSS/spark-commons/actions/workflows/build.yml)
55
[![Release](https://github.com/AbsaOSS/spark-commons/actions/workflows/release.yml/badge.svg)](https://github.com/AbsaOSS/spark-commons/actions/workflows/release.yml)
66

7-
`spark-commons` is a library offering commonly needed routines, classes and functionality. It consists of three modules.
7+
`spark-commons` is a library offering commonly needed routines, classes and functionality. It's build for Apache Spark of versions 2.4.x, 3.2.x, 3.3.x, 3.4.x and 3.5.x.
88
* spark-commons-spark2.4
99
* spark-commons-spark3.2
1010
* spark-commons-spark3.3
11+
* spark-commons-spark3.4
12+
* spark-commons-spark3.5
1113
* spark-commons-test
1214

1315
**spark2-commons** and **spark3-commons** both offer the same logic for the respective major versions of Spark addressing
1416
usual needs of Spark applications.
1517

1618
**spark-commons-test** then brings routines to help in testing Spark applications (and it's independent of Spark
17-
version used)
19+
version used)
1820

19-
20-
| | spark-commons-spark2.4 | spark-commons-spark3.2 | spark-commons-spark3.3 | spark-commons-test |
21-
|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
22-
| _Scala 2.11_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11) | | | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11) |
23-
| _Scala 2.12_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12) |
21+
| | spark-commons-spark2.4 | spark-commons-spark3.2 | spark-commons-spark3.3 | spark-commons-spark3.4 | spark-commons-spark3.5 | spark-commons-test |
22+
|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
23+
| _Scala 2.11_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11) | | | | | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11) |
24+
| _Scala 2.12_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.4_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.4_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.5_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.5_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12) |
25+
| _Scala 2.13_ | | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.13/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.13) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.13/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.13) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.4_2.13/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.4_2.13) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.5_2.13/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.5_2.13) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.13/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.13) |
2426

2527
## Spark-Commons
2628

‎build.sbt‎

Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,10 @@ lazy val scala211 = "2.11.12"
1919
lazy val scala212 = "2.12.18"
2020
lazy val scala213 = "2.13.13"
2121
lazy val spark2 = "2.4.8"
22-
lazy val spark32 = "3.2.4"
23-
lazy val spark33 = "3.3.2"
22+
lazy val spark32 = "3.2.4"
23+
lazy val spark33 = "3.3.2"
24+
lazy val spark34 = "3.4.4"
25+
lazy val spark35 = "3.5.5"
2426

2527
import Dependencies._
2628
import SparkVersionAxis._
@@ -47,24 +49,41 @@ lazy val commonJacocoExcludes: Seq[String] = Seq(
4749
)
4850

4951
lazy val parent = (project in file("."))
50-
.aggregate(sparkCommons.projectRefs ++ sparkCommonsTest.projectRefs: _*)
52+
.aggregate(
53+
sparkCommonsSpark2.projectRefs ++
54+
sparkCommonsSpark3.projectRefs ++
55+
sparkCommonsTest.projectRefs: _*
56+
)
5157
.settings(
5258
name := "spark-commons-parent",
5359
publish / skip := true
5460
)
5561

56-
lazy val `sparkCommons` = (projectMatrix in file("spark-commons"))
62+
lazy val sparkCommonsSpark2 = (projectMatrix in file("scala-spark2.4-jvm"))
5763
.settings(commonSettings: _*)
5864
.sparkRow(SparkVersionAxis(spark2), scalaVersions = Seq(scala211, scala212))
65+
.settings(
66+
Compile / unmanagedSourceDirectories := Seq((Compile / sourceDirectory).value / "main" / "scala")
67+
)
68+
69+
lazy val spark3Versions = Seq(spark32, spark33, spark34, spark35)
70+
lazy val sparkCommonsSpark3 = (projectMatrix in file("scala-spark3-jvm"))
71+
.settings(commonSettings: _*)
5972
.sparkRow(SparkVersionAxis(spark32), scalaVersions = Seq(scala212, scala213))
6073
.sparkRow(SparkVersionAxis(spark33), scalaVersions = Seq(scala212, scala213))
61-
.dependsOn(sparkCommonsTest % "test")
74+
.sparkRow(SparkVersionAxis(spark34), scalaVersions = Seq(scala212, scala213))
75+
.sparkRow(SparkVersionAxis(spark35), scalaVersions = Seq(scala212, scala213))
76+
.settings(
77+
Compile / unmanagedSourceDirectories := Seq((Compile / sourceDirectory).value / "main" / "scala")
78+
)
6279

6380
lazy val sparkCommonsTest = (projectMatrix in file("spark-commons-test"))
6481
.settings(
6582
commonSettings ++ Seq(
6683
name := "spark-commons-test",
67-
libraryDependencies ++= sparkDependencies(if (scalaVersion.value == scala211) spark2 else spark32),
84+
libraryDependencies ++= sparkDependencies(
85+
if (scalaVersion.value == scala211) spark2 else spark35
86+
),
6887
Compile / unmanagedSourceDirectories += {
6988
val sourceDir = (Compile / sourceDirectory).value
7089
if (scalaVersion.value.startsWith("2.13")) {

‎project/SparkVersionAxis.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ object SparkVersionAxis {
4242
scalaVersions = Seq(scalaVersion),
4343
axisValues = Seq(sparkAxis, VirtualAxis.jvm),
4444
_.settings(
45-
moduleName := camelCaseToLowerDashCase(name.value + sparkAxis.directorySuffix),
45+
moduleName := camelCaseToLowerDashCase(
46+
name.value.replaceAll("(Spark2|Spark3)$", "") + sparkAxis.directorySuffix
47+
),
4648
libraryDependencies ++= sparkCommonsDependencies(sparkAxis.sparkVersion),
4749
jacocoReportSettings := jacocoSettings(sparkVersion, scalaVersion),
4850
jacocoExcludes := jacocoProjectExcludes(sparkVersion, scalaVersion)

‎publish.sbt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,12 @@ ThisBuild / developers := List(
6565
name = "Ladislav Sulak",
6666
email = "[email protected]",
6767
url = url("https://github.com/lsulak")
68+
),
69+
Developer(
70+
id = "salamonpavel",
71+
name = "Pavel Salamon",
72+
email = "[email protected]",
73+
url = url("https://github.com/salamonpavel")
6874
)
6975
)
7076

‎spark-commons/src/main/scala-spark3.2-jvm/za/co/absa/spark/commons/adapters/CallUdfAdapter.scala‎ renamed to ‎spark-commons/src/main/scala-spark3-jvm/za/co/absa/spark/commons/adapters/CallUdfAdapter.scala‎

File renamed without changes.

‎spark-commons/src/main/scala-spark3.2-jvm/za/co/absa/spark/commons/adapters/TransformAdapter.scala‎ renamed to ‎spark-commons/src/main/scala-spark3-jvm/za/co/absa/spark/commons/adapters/TransformAdapter.scala‎

File renamed without changes.

‎spark-commons/src/main/scala-spark3.3-jvm/za/co/absa/spark/commons/adapters/CallUdfAdapter.scala‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎spark-commons/src/main/scala-spark3.3-jvm/za/co/absa/spark/commons/adapters/TransformAdapter.scala‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

0 commit comments

Comments
 (0)