Enable Apache Spark(Pyspark) to run on Jupyter Notebook - Part 2 | Install Spark on Jupyter Notebook

This video titled "Enable Apache Spark(Pyspark) to run on Jupyter Notebook - Part 2 | Install Spark on Jupyter Notebook" explains the final three steps to install or enable Apache Spark or Pyspark to run on Jupyter Notebooks so that you can run Spark/Pyspark related commands or jobs on Jupyter notebook in an interactive way just like you run Python specific commands. This topic is very useful for budding data scientists or data analysts who want to explore, scale-up and develop machine learning abilities or wants to perform some data analysis using distributed computing frameworks like Spark and that to in Jupyter notebooks.

FOLLOW ME ON:

About this Channel:
The AI University is a channel which is on a mission to democratize the Artificial Intelligence, Big Data Hadoop and Cloud Computing education to the entire world. The aim of this channel is to impart the knowledge to the data science, data analysis, data engineering and cloud architecture aspirants as well as providing advanced knowledge to the ones who already possess some of this knowledge.

Please share, comment, like and subscribe if you liked this video. If you have any specific questions then you can comment on the comment section and I'll definitely try to get back to you.

*******Other AI, ML and Deep Learning Related Video Series*****

******************************************************************

#SparkonJupyter #PysparkonJupyter #ApacheSpark

[TheAIUniversity]

Did you like this two
video series to install PySpark on windows machine in order to run spark based commands and jobs using Jupyter Notebook ?

[XiwithHighPing]

I am a MS student in a German university. I needed to set up spark for big data module. Saw a lot of videos which included installing virtual box and what not. This video series saved a lot of hustle. Thank you so much sir!

[mineatalay]

I tried a lot of different tutorials, this one is the only one works for me thank you so much! Awesome!

[simbatherealdog]

This was a complete and to the point guide, with no unnecessary fluff. Thanks a lot!

[AlecKennedyWalker]

Had issues with getting prompt to accept 'pyspark' command, but overcame them by simply doing 'pip install pyspark' and then choosing 'y' for yes.. Why were there issues to begin with? Perhaps due to overwriting the "PATH" variable in the "User variable" part of environment variables in the last video (first "PATH" variable set to Java bin folder, and then later in the video we overwrite address to Spark bin folder). Or could be compatibility issues based on this video being a year old! Either way, I appreciate your help getting this set up.

[adityarastogi]

thanks a lot sir !
your both the videos (part1&2) really helped me. I was not able to create spark environment in my jupyter notebook.
Thanks a lot!! Amazing :)

[samahosman]

I did all the steps as you mentioned and i set all the environmental variables, and when i type pyspark in the anaconda cmd it says : The system cannot find the path specified., any idea how to fix it ?

[fealgu]

Simple, understandable, instructions that actually work. Thanks!

[animebeast]

i follow all steps but got erroe as
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified

[kritikasingh]

Hey i need a help.
#Testing pyspark was successful
But
#Initiate Spark Content
Shows Exception : Java gateway process exited before sending its port

Please help me out for this

[Abecedarianme]

Is this still valid? I tried to follow the steps but in the part of creating NEW PATH in the user variables, it is overwriting the existing values.

I want more Pyspark videos from you! You're awesome!

[close_to_life]

Your video is love. It's accurate, to the point, proper described, and well communication skill.

[manivasagamkarunakaran]

Thank you so much! These videos helped install Pyspark on windows.

[vamsibapatla]

Hi I downloaded spark and winutils.exe from git hub and set the environment variables, but i am not able to get the pyspark version, when I am typing in Anaconda prompt

[dhruvloya]

i am getting this error "'pyspark' is not recognized as an internal or external command,
operable program or batch file." how to fix this ?

[ShivKumar-gjel]

Thank you so much for really simple installation video

[OrangePotatoLeo]

Hello, I am not able to find the github link for the jupyter notebook in the description.

[skviknesh]

Cool, i replicate all steps from both videos... it's working successfully now in jupyter as well as spyder (in anaconda environment).

[kya-baat]

@AIUniversity I tried doing the same thing but while entering pyspark on anaconda prompt, I'm getting the following error. was unexpected at this time.