Master Databricks and Apache Spark Step by Step: Lesson 27 - PySpark: Coding pandas UDFs

PySpark pandas user defined functions are custom code you can run in parallel over the cluster nodes getting top performance. Spark 3.0 launched a new way to code traditional Python User Defined Functions (UDF) (introduced in video 26). This video teaches you how to code the new PySpark pandas user defined functions.

Slides at:

Intro to PySpark User Defined Functions Video

[shriramsudrik]

I was looking for Pandas UDF and I am glad that I found your videos. 10/10 to you Bryan!

[mohamedalryah]

thanks a lot, Mr. Bryan for these videos, they are very informative and detailed! thanks for putting in time and effort

[JoaoOliveira-rkgv]

You are awesome Bryan. Thank you so much for all this quality content for free. So much respect

[haneulkim]

Amazing tutorial! so we can not do more processing in between function and its return only when its `series -> series`? So I can't initialize model with broadcasted weights inside function when using pandas_udf that receives series and return series?

[dchandrateja]

Hi Bryan, it was a great explanation.
Is it possible to write functions with spark context, like writing spark code in a fucntion which has a bunch of transformation fucntions to calculate a value.
That would really solve my problem.
I tried writing but I get this error “It appears that you are attempting to reference sparkcontext from a broadcast variable, action or transformation. SC can only be used on the driver, not in code that run on workers)
Thank you in advance

[Gerald-izmv]

hi nice video - do you have another video which covers Vectorized UDF?

[IvanPerez-vkdj]

Hi Bryan. Thanks a lot for your time and effort doing these series. All of your content is pure gold. Not only for the level of detail in the explanations, but also for how well structured they are. You have a great talent explaining things. I really enjoy your channel, congratulations!

A question ... In cell 15 of this notebook, the type hints of the UDF shouldn't be Iterator[int] ? I think we are passing a pd.series right? which in this case is a column of ints, so what the function receives is an iterator if ints ... Not sure if I'm right.

Live longer and prosper dear Bryan!
🖖🏼

[ryanjadidi]

Dont you think the first way of calling the panda udf is faster than iterator because its using vectorization?

[severalpens]

Hi Bryan, I'm loading a bunch of JSON files with nested objects and arrays using Autoloader. This part works well but I was looking to create a scalar UDF that could parse and extract values from the resulting 'struct' cells.

eg getTimeStamp(json_field) where json_field = {Id: 23, name: "foo", timestamp: 123413}.

I know I can query within struct field but I've got complex requirements that I'd like to encase in a UDF.

[cssensei]

The modeling example was hard to follow:--

Can you show me a pyspark groupBy and K-Means scikit model inside pandas_udf?