Master Databricks and Apache Spark Step by Step: Lesson 29 - PySpark: Coding pandas Function API

You use the PySpark pandas Function API to write custom code you can run in parallel over the cluster nodes getting top performance. Spark 3.0 launched this way to write paralyzed code delivering new functionality. This video teaches you how to code functions using the new PySpark pandas Function API.

Join my Patreon Community

Twitter: @BryanCafferky

Notebook at:

Creating Databricks Spark SQL Tables

[mallutornado]

I can not tell you how valuable this is. I should have just come here rather than wasting hours reading poorly explained databricks manual pages and examples and not getting them to work on real use cases as it wasn’t obvious how to use them. Life Saver !

[shibuvm]

Hi Bryan,
Here is a variation of the mapinpandas, that i tested and works. It runs as spark job by the looks of it -

spark_df=spark.createDataFrame([("Kishore", 100), ("Kishore", 200), ("Kishore", 300), ("SPB", 400), ("SPB", 500), ("SPB", 600)], ("SINGER", "SONGS") )

rate = 1000

def label_expensive (row):
if row['FEES'] < :
return 'No'
if row['FEES'] >= :
return 'Yes'
return 'Other'

def filter_func(iterator):
for pdf in iterator:
pdf["FEES"] = pdf.SONGS * rate
pdf["EXPENSIVE"] = pdf.apply (lambda row: label_expensive(row), axis=1)
yield pdf

spark_df.mapInPandas(filter_func, schema="SINGER string, SONGS long, FEES long, EXPENSIVE string").show()

[siddeghamid]

Thanks Bryan, we do really appreciate it. Any examples with Spark streaming/Kafka would be awesome

[illiakailli]

Hi Bryan, thanks for the videos you’ve shared! Lots of useful information.
Wanted to share a wish list regarding what else would be great to cover:
- practical example of working with some large datasets, 100GB or more
- creating cluster with multiple nodes to illustrate why it is useful to partition data and how it impacts query performance
- making more transparent how and when data gets copied to spark cluster nodes, for how long it stays there …. etc

[mallutornado]

Bryan, for an ML workload, would it be better to keep a fixed worker count rather than auto scaling ?

[ichtot]

Honest question isnt split apply combine just map reduce?