Master Databricks and Apache Spark Step by Step: Lesson 25 - PySpark: Using SQL Aggregate Functions

In this video, you learn how to use the built-in Spark SQL aggregate functions from Python. This gives you the benefit of good performance and functionality offered by any SQL function all within Python code. Scalar functions can also be used from Python.

Video demo notebook at:

For information on how to upload files to Databricks and create tables see:

Blog on using SQL Aggregate functions from PySpark

[generalcvs]

Thanks for your Sparks lessons. Really helpful and clear. Would be great to have lesson about Deltalake and tuning use cases.

[f_lyru]

Always a pleasure to watch, thank you for your content

[hmishra]

Yeaa bryan is back ❤️❤️ thanks for amazing content .

[DeezNutzRHugeous]

Can you do a pyspark withColumn case when then sum abs column?

[vuppalanaveenkrishna]

Really Amazing content sir.Can we have that notebook

[Ron-gndv]

One question. When to use direct SQL or pyspark functions(chain functions) ? I mean spark.sql('....write all agg/scalar function...') or df.groupby ...like that. When to use what ?

[generalcvs]

Looks like this particular lesson is the duplicate of previous one.