Master Databricks and Apache Spark Step by Step: Lesson 15 - Using SQL Functions

In this video, you learn how to use Spark Structured Query Language (SQL) scalar and aggregate functions. Spark SQL is the most performant way to do data engineering on Databricks and Spark and you want to leverage SQL functions as much as possible versus writing custom code. I'll explain the concepts and demonstrate them with code in a Databricks notebook.

Get my boo Master Azure Databricks Step by Step at

Example Notebook for lesson 15 at:

You need to unzip the file and import the notebook into Databricks to run the code.

Video on Creating and Loading the tables used in this video

[anthonygonsalvis]

Love the exercise element. It's light but helps reinforce the learning and test the understanding. Please include or expand the exercises in future videos.

[andreasafranekova]

Thank you for making these videos, it helps a lot! :)

[Raaj_ML]

Superb. As some one has already suggested, please give more exercises.

[claudiocordova]

how do aggregate function work when data is partitioned?