-
Notifications
You must be signed in to change notification settings - Fork 36
Description
Hi @Jolanrensen ,
I'm sorry toss these issues/questions at you without being able to provide much of assistance myself. I really appreciate you taking your time to help out.
We're in the progress of updating our code to Spark 3.3.0, but we're seeing some errors. We're not experiencing the errors locally in our unit tests, but in Databricks and their runtime 11.2. I can confirm that that runtime uses Spark 3.3.0, Scala 2.12 which is also what we use together with org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1
. My theory is that Databricks doesn't use the open source version of spark, hence we get the following error message:
java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.expressions.Expression org.apache.spark.sql.catalyst.DeserializerBuildHelper$.deserializerForWithNullSafetyAndUpcast(org.apache.spark.sql.catalyst.expressions.Expression, org.apache.spark.sql.types.DataType, boolean, org.apache.spark.sql.catalyst.WalkedTypePath, scala.Function2)'
at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:671)
at org.apache.spark.sql.KotlinReflection.deserializerFor(KotlinReflection.scala)
at org.jetbrains.kotlinx.spark.api.EncodingKt.kotlinClassEncoder(Encoding.kt:169)
at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:144)
at xyz.dune.arrakis.spark.AutoBackfillingJoin.streamJoin(AutoBackfillingJoin.kt:533)
Any thoughts on whether that theory might be correct? As far as I know, these errors normally is a consequence of mismatching versions. Could it be that the databricks runtime has some custom spark code where deserializerForWithNullSafetyAndUpcast doesn't exist or have a different signature?
Thanks again. Much appreciated!