Master Databricks and Apache Spark Step by Step: Lesson 24 - Creating PySpark Dataframe Scalar UDFs

In this video, you learn how to create PySpark dataframe User Defined Functions (UDF) to perform distributed transformations on each row. You will learn about using Apache Arrow to get optimal performance and how to use these functions from Spark SQL and dataframes.

Video demo notebook at:

For information on how to upload files to Databricks and create tables see:

Blog on creating Apache Spark Scalar UDFs

[manuever]

This series is amazing. I just started working with Azure databricks and this tutorial helps me greatly than any doc I can find within microsoft!

[ravitutika]

Oh my waiting notification when you upload video on this series ...

[hmishra]

Was waiting for this ❤️ Thanks bryan 👍

[arifshaik]

Amazing Tutorial, Thanks for making such awesome tutorial

[Raaj_ML]

Bryan, any example you plan for a really "big data" set which will need spark cluster, not just the driver node in community edition of databricks ?

[rahuldey]

Hello Bryan. First of all thanks a lot for sharing knowledge. Just wanted to do you have a link to the series playlist?