Apache Spark - Install Spark3, PySpark3 on Ubuntu 20.04, Debian, Python 3.8 - Part 1b

0:00 - download the Spark library from Apache website
2:19 - uncompress and install the Spark tgz file to a target directory
6:04 - check version of Python
7:48 - update .bashrc file to include environment path variables for both Spark and PySpark
13:13 - install PySpark, and install pip3 if required
16:34 - install Java JRE and JDK
20:18 - validate Spark is working via the Spark shell (Scala prompt)
22:01 - validate PySpark is working via the PySpark shell (Python prompt)
22:45 - connect SparkSession from a Python script

Install Apache Spark and PySpark on Ubuntu 20.04 Linux Debian, Python 3.7 - Part 1a

[felipesantos]

You rock, man ! I spent an afternoon trying to make pyspark works on my own and after seeing your video I got it immediately!

[carolchen]

Thank you so much!!!
I've been searching for guides for days and this is just what I want!

[sammydude]

Awesome!! Spark 3.0.0 working fine on Python 3.8. This video needs more likes!

[danielh.vediajerez]

Well explained and running without problems.

Thanks for this tutorial.

[TrospeK]

Well explained and helpful.
Thanks for your help!

[franciscoyanezcontreras]

If you got this error:
py4j.protocol.Py4JError: does not exist in the JVM

Add this line to ~/.bashrc

export

Then re-run source

[Liteship]

Hi, I've Spark vle roersion 3.1.1 and Pyspark 3.1.1 spark-shell is working fine without any errors but i am unable to load Pyspark

[nguyenduydatnguyenduydat]

Thank you so much for sharing this tutorial about setting spark with pyspark. I follow you and get done all step except the last step: connect SparkSession from a Python script. When I run the script
it appears Py4JError( py4j.protocol.Py4JError: does not exist in the JVM ) Hope that you can help me to fix this error.

[sambodhisengupta]

spark-shell
/usr/local/bin/spark-shell: line 60: No such file or directory

please help...what should I do?
I followed each and every step of your video