Lect 7: UDF user defined functions in spark. | pyspark | UDF | Big Data |Data Engineer

Well the discussion starts with why do we need to define our own functions. Even though python has many inbuilt functions and many libraries but still we tend to define a lot of functions, this gives the answer to the question of why, which is not all the objectives wont be met with those inbuilt functions. Similarly when we are working with spark, we need customized functions that deals and results the desired output. Well defining functions in python and in spark has similarities but not exactly same those are also discussed in this lecture. Don't forget to look in to the documentation from the developers of spark itself cause I always believe that "always look for/at the source to understand the cause". So, with that saying, let jump into the lecture of understanding the why we need udf (user defined functions) and how to define one, their syntax and also look at a case study where we define our own UDF to solve the problem.

In this video you will be learning about with time spans:

00:00 - Intro
01:15 - Agenda for the lecture.
02:05 - Looking at the documentation on UDF.
03:40 - Take aways from documentation with example.
07:10 - Difference between python's UDF from pyspark's UDF
08:24 - Why we need UDF.
10:35 - Test case or case study on/using UDF.
13:25 - Specifying the objective for the test case.
14:45 - Defining udf step by step.
22:38 - Revising what we have learned.

#spark #pyspark #bigdata #bigdatahadoop #python #databricks #cluster #apachespark #udf