Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Allow specifying non-local files to spark-submit (python files, and R files) #530

Open
wants to merge 4 commits into
base: branch-2.2-kubernetes
Choose a base branch
from
Open

Allow specifying non-local files to spark-submit (python files, and R files) #530

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
79ffa72
Allow specifying non-local files to spark-submit (python files, and R…
paulreimer Oct 18, 2017
ecfa6f2
Avoid formatting python path (when using --py-files) in kubernetes cl…
paulreimer Oct 19, 2017
a621c5f
More general comment about cluster modes which should skip python pat…
paulreimer Oct 19, 2017
e8ecab2
Merge branch 'branch-2.2-kubernetes' into SparkSubmit-allow-k8s-non-l…
foxish Dec 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ object SparkSubmit extends CommandLineUtils {
// Require all python files to be local, so we can add them to the PYTHONPATH
// In YARN cluster mode, python files are distributed as regular files, which can be non-local.
// In Mesos cluster mode, non-local python files are automatically downloaded by Mesos.
if (args.isPython && !isYarnCluster && !isMesosCluster) {
if (args.isPython && !isYarnCluster && !isMesosCluster && !isKubernetesCluster) {
if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
printErrorAndExit(s"Only local python files are supported: ${args.primaryResource}")
}
Expand All @@ -336,7 +336,7 @@ object SparkSubmit extends CommandLineUtils {
}

// Require all R files to be local
if (args.isR && !isYarnCluster && !isMesosCluster) {
if (args.isR && !isYarnCluster && !isMesosCluster && !isKubernetesCluster) {
if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
printErrorAndExit(s"Only local R files are supported: ${args.primaryResource}")
}
Expand Down Expand Up @@ -682,10 +682,10 @@ object SparkSubmit extends CommandLineUtils {
// explicitly sets `spark.submit.pyFiles` in his/her default properties file.
sysProps.get("spark.submit.pyFiles").foreach { pyFiles =>
val resolvedPyFiles = Utils.resolveURIs(pyFiles)
val formattedPyFiles = if (!isYarnCluster && !isMesosCluster) {
val formattedPyFiles = if (!isYarnCluster && !isMesosCluster && !isKubernetesCluster) {
PythonRunner.formatPaths(resolvedPyFiles).mkString(",")
} else {
// Ignoring formatting python path in yarn and mesos cluster mode, these two modes
// Ignoring formatting python path in yarn, mesos, and kubernetes cluster mode, these modes
// support dealing with remote python files, they could distribute and add python files
// locally.
resolvedPyFiles
Expand Down