Apache Spark - Pandas On Spark | Spark Performance Tuning | Spark Optimization Technique

#apachespark #sparktutorial #pandasonspark
Apache Spark - Pandas On Spark | Spark Performance Tuning | Spark Optimization Technique

In this video, we will learn about the new feature of Pandas on Spark that was released in Spark 3.2.0 version. We will also have a small demo to understand the Spark Performance tuning and Spark Performance improvement while using Pandas on Spark over native pandas library.

Blog on Pandas API on Spark:

==================================
Blog link to learn more on Spark:

Linkedin profile:

FB page:

#pyspark
#apachespark
#azure
#databricks
#dataengineering
#sparkwork
#interview
pyspark interview questions and answers

[yank]

You are not directly comparing the performance. You should only run the describe function not include the conversion time.

[ganeshdhareshwar]

Very helpful video. Thanks bro. I will be explore more on pandas on spark

[ManojKumar-cgft]

Hi Bro, Your videos are really helpful. Can you help me on below points. 1. How many partitions will be created by default while reading file from S3 bucket and how can we cahnge defualt spark read dataframe partition. 2) How to decide the number repartitions/coalesce required based on EMR cluster configuration.
3) how many partitions will be created if we have 50K small files in S3 bucket folder, how efficiently we can read it.
Thank You!!!

[tayal]

Better comparison would be where we create a pandas df with 1M key records and then compare it. This comparison is a big vague here since we are declaring df in spark, then converting it to pandas and then using the describe method for comparison.

[samwelemmanuel]

It would have been more awesome and accurate if you could split conversion from describe in different cells and only compare describe stage between Pandas Local and Pandas on Spark.

[hashisgod]

anybody have any channel recommendations on this subject where the voiceover has an English/American accent? I can't listen to the Indian

[mohitupadhayay]

I am facing No module found pyspark.pandas everytime.

My commands are :

import pyspark.pandas as ps

Error : No module found pyspark.pandas everytime.

DBR 7.3 LTS, Spark 3.0.1

What is going on?

[MUHAMMADNOMAN_ALIG]

I regularly watched your videos and it helped me a lot in solving scenario based questions during Big Data interviews.
Your videos are very informative and practical. The best part is the videos are short.
Thanks a lot. 👍