Get familiarized with Scala and Python shells

Apache spark provides 2 types of shell windows, which allows us to type the spark commands interactively. These awesome tools help us to learn apache spark quickly and conveniently.

“spark-shell” is a command which takes us to a scala-based shell window, where you can type spark commands in “scala” programming language.

The other one is the “pyspark” command, which takes us to a python-based shell window where you can type spark commands in “python” programming language.

Based on your programming language preference, whether it is scala or python, you can use the appropriate shell window.

Java is not supported in the shell window. However, you can write the spark commands in Java in an IDE, package it as a jar, and then submit the jar using spark-submit tool. We will look in detail on the spark-submit command, later in the video. For now, install spark in your machine and let’s get started with the shell window.

Once you are in the spark shell, spark provides an entry point called sparkContext. Programmers can access sparkContext by using “sc”.
sparkContext has been the entry point for spark functionality, since the earlier versions of spark. It provides access to the spark execution environment.

From spark 2.0 version onwards, spark provides sparkSession, which is the single point of entry to the spark functionality. You can do everything that you do in sparkContext, using sparkSession as well.