Skip to content

Questions regarding java.lang.NoSuchMethodError #184

@hawkaa

Description

@hawkaa

Hi @Jolanrensen ,

I'm sorry toss these issues/questions at you without being able to provide much of assistance myself. I really appreciate you taking your time to help out.

We're in the progress of updating our code to Spark 3.3.0, but we're seeing some errors. We're not experiencing the errors locally in our unit tests, but in Databricks and their runtime 11.2. I can confirm that that runtime uses Spark 3.3.0, Scala 2.12 which is also what we use together with org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1. My theory is that Databricks doesn't use the open source version of spark, hence we get the following error message:

java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.expressions.Expression org.apache.spark.sql.catalyst.DeserializerBuildHelper$.deserializerForWithNullSafetyAndUpcast(org.apache.spark.sql.catalyst.expressions.Expression, org.apache.spark.sql.types.DataType, boolean, org.apache.spark.sql.catalyst.WalkedTypePath, scala.Function2)'
	at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:671)
	at org.apache.spark.sql.KotlinReflection.deserializerFor(KotlinReflection.scala)
	at org.jetbrains.kotlinx.spark.api.EncodingKt.kotlinClassEncoder(Encoding.kt:169)
	at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:144)
	at xyz.dune.arrakis.spark.AutoBackfillingJoin.streamJoin(AutoBackfillingJoin.kt:533)

Any thoughts on whether that theory might be correct? As far as I know, these errors normally is a consequence of mismatching versions. Could it be that the databricks runtime has some custom spark code where deserializerForWithNullSafetyAndUpcast doesn't exist or have a different signature?

Thanks again. Much appreciated!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions