Skip to content

Commit 1f6bd7a

Browse files
[LIVY-863] Missing JVM class imports for Spark3
## What changes were proposed in this pull request? Description of the problem: https://issues.apache.org/jira/browse/LIVY-863 The proposed fix consists of adding the missing imports that upstream Spark has when it's initiating the Java Gateway. See the imports of Spark here: https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153 As far as I am aware, the java_import() does not fail or fails silently if the imported class does not exist. But I might need to add some code to account for different Spark versions, looking for some guidance on this. ## How was this patch tested? Tested with both valid and "bogus" imports by reviewers.
1 parent 6097af1 commit 1f6bd7a

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

repl/src/main/resources/fake_shell.py

+3
Original file line numberDiff line numberDiff line change
@@ -589,8 +589,11 @@ def main():
589589
java_import(gateway.jvm, "org.apache.spark.SparkConf")
590590
java_import(gateway.jvm, "org.apache.spark.api.java.*")
591591
java_import(gateway.jvm, "org.apache.spark.api.python.*")
592+
java_import(gateway.jvm, "org.apache.spark.ml.python.*")
592593
java_import(gateway.jvm, "org.apache.spark.mllib.api.python.*")
594+
java_import(gateway.jvm, "org.apache.spark.resource.*")
593595
java_import(gateway.jvm, "org.apache.spark.sql.*")
596+
java_import(gateway.jvm, "org.apache.spark.sql.api.python.*")
594597
java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
595598
java_import(gateway.jvm, "scala.Tuple2")
596599

0 commit comments

Comments
 (0)