Master Databricks and Apache Spark Step by Step: Lesson 10 - Creating the SQL Tables on Spark

In this video, you will learn how to create and load the project CSV files into SQL tables on open-source Apache Spark using Zeppelin Notebook. The prior video, Lesson 9, showed you how to create the tables using Databricks. The files are available with the notebook and slides at the link below. This video lays the foundation for the ones that follow so make sure you watch it and create your own database.

Example Slides & Notebook at:

You need to unzip the file and import the notebook into Zeppelin to run the code.

Video on The Data Science Process

Video on Dimensional Modeling

Databricks Spark SQL Data Types

[tinaxu]

Thank you for sharing the great tutorial. One of the cool thing is: at the end of each video, you review the content that previously taught in the video. ✅💯👍💖

[sharad]

someone who has already watched Lesson 9, can directly jump to 5:05

[Raaj_ML]

In Databricks and HDinsight, you don't need to install Spark separately as they come with Spark already. How about local premise (say laptop ) ? How do we install Spark ? Is installing pyspark equal to installing Spark ?

[Raaj_ML]

Hi Bryan, I was wondering why data engineering job market is getting more huge than ML jobs...Probably you answered it in this video.."EDA is the place where many companies end the process.." they don't go past that to create predictive models.. :)

[amaytrivedi]

Hi Bryan, important
For spark and all those code in sql, can i use Jupyter notebook instead of Zeppelin as Zeppelin is not free and I would like to stick with the free thing. Please let me know is setting up hdinsight really important.

Thanks!

[yizhengfeng]

Where can I download all the data used in this lesson (all the .csv files) as well as the .dbc file for this lesson.
Thank you so much for sharing!!!

[ChrisUK]

Thanks Bryan sorry another question when a table is created does it lock the file so it cannot be deleted from the file system?