Download Spark

Spark

Download Jupyter

https://jupyter.org

Update your .bash_profile

​x
1
export SPARK_HOME=/opt/spark
2
export PATH=$SPARK_HOME/bin:$PATH
3
​
4
export PYSPARK_DRIVER_PYTHON=jupyter
5
# You may have to increase the default iopub_data_rate_limit which is 1000000
6
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --NotebookApp.iopub_data_rate_limit=\
7
100000000.0'

Java 8

Hadoop and Spark run with JDK 8 only. Download JDK 8.

JDK 8

Edit spark-env.sh

xxxxxxxxxx
1
# cp from /opt/spark/conf/spark-env.sh.template to create spark-env.sh if non-existant
2
# add the following line
3
export JAVA_HOME=/path_to_jdk/jdk8u232-b09/Contents/Home

Edit spark-defaults.conf

xxxxxxxxxx
1
# Set Path JDK1.8 
2
# cp from /opt/spark/conf/spark-defaults.conf.template to create spark-defaults.conf
3
# add the following line
4
# uncomment the spark.driver.memory and change it according to your use. 
5
spark.driver.memory 5g
6
spark.executorEnv.JAVA_HOME=/path_to_jdk/jdk8u232-b09/Contents/Home

Run Spark

xxxxxxxxxx
2
1
# run
2
/opt/spark/bin/pyspark

Set up PyCharm

In Project Structure click Add Content Root to add

/opt/spark-2.4.4/python/lib/py4j-0.10.7-src.zip and

/opt/spark-2.4.4/python/lib/pyspark.zip