-
Notifications
You must be signed in to change notification settings - Fork 36
Open
Labels
good first issueGood for newcomersGood for newcomers
Description
We're building spark application on AWS EMR service, and seeing error when running the application:
Exception in thread "main" scala.ScalaReflectionException: object org.apache.spark.sql.KotlinReflection not found.
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:185)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:29)
at org.apache.spark.sql.KotlinReflection$$typecreator1$6.apply(KotlinReflection.scala:777)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
at org.apache.spark.sql.KotlinReflection.localTypeOf(KotlinReflection.scala:1404)
at org.apache.spark.sql.KotlinReflection.localTypeOf$(KotlinReflection.scala:1402)
at org.apache.spark.sql.KotlinReflection$.localTypeOf(KotlinReflection.scala:53)
at org.apache.spark.sql.KotlinReflection$.$anonfun$serializerFor$1(KotlinReflection.scala:777)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:73)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:1389)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:1388)
at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:53)
at org.apache.spark.sql.KotlinReflection$.serializerFor(KotlinReflection.scala:694)
at org.apache.spark.sql.KotlinReflection$.serializerFor(KotlinReflection.scala:681)
at org.apache.spark.sql.KotlinReflection.serializerFor(KotlinReflection.scala)
at org.jetbrains.kotlinx.spark.api.EncodingKt.kotlinClassEncoder(Encoding.kt:165)
at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:144)
I can confirm that that runtime uses Spark 3.3.0, Scala 2.12, and we use the matching version of koltin-spark-api org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1
and org.jetbrains.kotlinx.spark:core-3.3.0_2.12:1.2.1
This seems like an issue of missing dependency. Any thoughts on which dependency we should check?
Thanks!
Metadata
Metadata
Assignees
Labels
good first issueGood for newcomersGood for newcomers