Skip to content

Use Spark (python) from CLI

andreacondorelli edited this page Sep 1, 2015 · 1 revision

To launch from CLI: spark-submit mySparkCode.py argv1 argv2 argv3

While in the '.py' file you must also create the spark context:

from pyspark import SparkContext, SparkConf
conf = SparkConf()
sc = SparkContext(conf=conf)

To read an input from CLI:

import sys
argList = argv[1:]

Clone this wiki locally