Skip to content

Questions regarding scala.ScalaReflectionException #194

@RuizhuYang

Description

@RuizhuYang

We're building spark application on AWS EMR service, and seeing error when running the application:

Exception in thread "main" scala.ScalaReflectionException: object org.apache.spark.sql.KotlinReflection not found.
	at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:185)
	at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:29)
	at org.apache.spark.sql.KotlinReflection$$typecreator1$6.apply(KotlinReflection.scala:777)
	at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
	at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
	at org.apache.spark.sql.KotlinReflection.localTypeOf(KotlinReflection.scala:1404)
	at org.apache.spark.sql.KotlinReflection.localTypeOf$(KotlinReflection.scala:1402)
	at org.apache.spark.sql.KotlinReflection$.localTypeOf(KotlinReflection.scala:53)
	at org.apache.spark.sql.KotlinReflection$.$anonfun$serializerFor$1(KotlinReflection.scala:777)
	at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:73)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:1389)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:1388)
	at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:53)
	at org.apache.spark.sql.KotlinReflection$.serializerFor(KotlinReflection.scala:694)
	at org.apache.spark.sql.KotlinReflection$.serializerFor(KotlinReflection.scala:681)
	at org.apache.spark.sql.KotlinReflection.serializerFor(KotlinReflection.scala)
	at org.jetbrains.kotlinx.spark.api.EncodingKt.kotlinClassEncoder(Encoding.kt:165)
	at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:144)

I can confirm that that runtime uses Spark 3.3.0, Scala 2.12, and we use the matching version of koltin-spark-api org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 and org.jetbrains.kotlinx.spark:core-3.3.0_2.12:1.2.1

This seems like an issue of missing dependency. Any thoughts on which dependency we should check?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions