Skip to content

java.lang.NoClassDefFoundError: scala/Product$class #79

@jsubm5

Description

@jsubm5

I'm trying to extract data form Salesforce via Spark in databricks
When doing a spark.read, getting "java.lang.NoClassDefFoundError: scala/Product$class"

soql = "select id from opportunity where 1=2"
sfDF = (spark.
read.
format("com.springml.spark.salesforce").
option("username", SFDCusername).
option("password", SFDCPwd+SecToken).
option("soql", soql).
option("version", "37.0").
load())

Databricks version : 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)
Here are the dependencies installed to databricks clusters
com.springml:spark-salesforce_2.11:1.1.3
com.springml:salesforce-wave-api:1.0.10

Here is the detailed error
java.lang.NoClassDefFoundError: scala/Product$class

Py4JJavaError Traceback (most recent call last)
in <cell line: 2>()
1 soql = "select id from opportunity where 1=2"
----> 2 sfDF = (spark.
3 read.
4 format("com.springml.spark.salesforce").
5 option("username", SFDCusername).

/databricks/spark/python/pyspark/instrumentation_utils.py in wrapper(*args, **kwargs)
46 start = time.perf_counter()
47 try:
---> 48 res = func(*args, **kwargs)
49 logger.log_success(
50 module_name, class_name, function_name, time.perf_counter() - start, signature

/databricks/spark/python/pyspark/sql/readwriter.py in load(self, path, format, schema, **options)
182 return self._df(self._jreader.load(self._spark._sc._jvm.PythonUtils.toSeq(path)))
183 else:
--> 184 return self._df(self._jreader.load())
185
186 def json(

/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in call(self, *args)
1319
1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
1322 answer, self.gateway_client, self.target_id, self.name)
1323

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
194 def deco(*a: Any, **kw: Any) -> Any:
195 try:
--> 196 return f(*a, **kw)
197 except Py4JJavaError as e:
198 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o655.load.
: java.lang.NoClassDefFoundError: scala/Product$class
at com.springml.spark.salesforce.DatasetRelation.(DatasetRelation.scala:35)
at com.springml.spark.salesforce.DefaultSource.createRelation(DefaultSource.scala:99)
at com.springml.spark.salesforce.DefaultSource.createRelation(DefaultSource.scala:50)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:375)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:331)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:331)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:306)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
at java.lang.Thread.run(Thread.java:750)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions