Master Databricks and Apache Spark Step by Step: Lesson 14 - Using SQL Set Operators

In this video, you learn how to Set Operators in Spark Structured Query Language (SQL), i.e. UNION, INTERSECT, and EXCEPT. Spark SQL is the most performant way to do data engineering on Databrick and Spark. I'll explain the concepts and demonstrate them with code in a Databricks notebook.

Get Master Azure Databricks Step by Step at

Example Notebook for lesson 14 at:

You need to unzip the file and import the notebook into Databricks to run the code.

Video on Creating and Loading the tables used in this video

Video on Dimensional Modeling - with an explanation of Snowflake Schema

[paulntalo]

Am glad i have watched the videos up to this far in 4 days. My plan is to first watch through all the videos and then rewatch the second time as i code along.

[guillermojastrzebski]

Great video. It's more a SQL course than a Databricks, but it's really helpful. There is no much instruction around on these aspects of SQL. I only used the UNION operator and didn't know there were others like INTERSECT. I like the way it is explained (as a whole concept calld "Set Operators". Thanks!

[anthonygonsalvis]

@Bryan Cafferky, You're the first one to explain WHY Union All allows for duplicates and Union doesn't. Thanks.

[tauseefguard]

Hi there, could you please explain how to find the running total for past six months in a table, or if you can reference me to any of your videos where you have explained about this topic.

[paulntalo]

Am glad i have watched the videos up to this far in 4 days. My plan is to first watch through all the videos and then rewatch the second time as i code along.