Spark Interview Question | Handle Data Skewness in Apache Spark | LearntoSpark

preview_player
Показать описание
In this video, we discuss about the skew-ness issue in spark and ways to over come this issue in Spark.

Blog link to learn more on Spark:

Linkedin profile:

FB page:

Github:
  1. Azarudeen Shahul
  2. Spark Interview Question
  3. Pyspark
  4. Spark
Рекомендации по теме
Spark Interview Question 0:06:18

Spark Interview Question | Handle Data Skewness in Apache Spark | LearntoSpark

Spark Interview Question 0:05:58

Spark Interview Question | How many CPU Cores | How many executors | How much executor memory

Apache Spark Interview 0:50:31

Apache Spark Interview Questions And Answers | Apache Spark Interview Questions 2020 | Simplilearn

Spark memory allocation 0:08:53

Spark memory allocation and reading large files| Spark Interview Questions

NULL Values in 0:05:57

NULL Values in Spark ☹️| A Common mistake ❌ | Spark Interview Question

Spark Interview Question 0:08:13

Spark Interview Question | Scenario Based Question | Multi Delimiter | LearntoSpark

Spark Interview Question 0:04:53

Spark Interview Question | Scenario Based | Multi Delimiter | Using Spark with Scala | LearntoSpark

Spark Scenario Based 0:07:09

Spark Scenario Based Question | Handle JSON in Apache Spark | Using PySpark | LearntoSpark

Spark Interview Questions 0:04:01

Spark Interview Questions Preparation Course

Data engineer interview 0:11:44

Data engineer interview question | Process 100 GB of data in Spark Spark | Number of Executors

Spark Interview Question 0:08:38

Spark Interview Question | Scenario Based Question | Explode and Posexplode in Spark | LearntoSpark

Null Data in 0:09:32

Null Data in Spark | How to Handle Null Type Data | Remove Null from Spark | Spark Interview

Processing 25GB of 0:14:20

Processing 25GB of data in Spark | How many Executors and how much Memory per Executor is required.

Spark Interview Question 0:10:29

Spark Interview Question | Scenario Based |DataFrameReader - Handle Corrupt Record | LearntoSpark

Batch vs Stream 0:02:06

Batch vs Stream processing | Spark Interview Questions

Spark Out of 0:07:38

Spark Out of Memory Issue | Spark Memory Tuning | Spark Memory Management | Part 1

Spark performance optimization 0:20:20

Spark performance optimization Part1 | How to do performance optimization in spark

Spark Interview Question 0:05:52

Spark Interview Question | Online Assessment Question | Coding Round | Spark Scala | LearntoSpark

Spark Interview Question 0:17:03

Spark Interview Question | Scenario Based | Merge DataFrame in Spark | LearntoSpark

Managing Spark Partitions 0:11:20

Managing Spark Partitions | Spark Tutorial | Spark Interview Question

Handling Non splittable 0:02:52

Handling Non splittable Files | Spark questions from comments

Apache Spark interview 0:13:58

Apache Spark interview questions & Points to remember-Part 1 | Session-19

Spark Scenario Based 0:08:57

Spark Scenario Based Question | Handle Nested JSON in Spark | Using Spark with Scala | LearntoSpark

2.2 Fault Tolerance 0:04:03

2.2 Fault Tolerance in Spark | Spark Interview question #spark #bigdata #hadoop

Комментарии
Автор

Thanks Azarudeen, How can we do salting technique in Pyspark for data skewness?

vijeandran
Автор

Thanks for your explanation. Can you please provide this with example. I mean code and data.

krishnarupeshnagubandi
Автор

Salting technique we can apply to avoid the skewness when we know data skew before we running the job.But in real time if job is failed due to skew how to handle it? and how to avoid such skew failures in production?

ramesh-bigdatalearner
Автор

Thanks for your efforts in making videos on spark. It would be helpful if you show sample code for these options for better understanding

srinuch
Автор

Hi Bro... This is Ultimate... Very well explained...
I am a begginer... So I couldn't understand...
1.. What is single dominant partition id mean...? Why partitions with single dominant partition id cannot be repartitioned...
2. Also, what is multiple dominant partition id...
3. How to change in the source itself... Partitions are made by spark - based on partitions size, right... How can we specify one or more columns for partitioning... What is the command for that...
Please help me in these doubts...
Thank you, bro...

gurumoorthysivakolunthu
Автор

please explain us the salting technique in detail

monangiavinash
Автор

Please explain solution with a real timecode scenario

ashwinc
Автор

Hi Shahul i have sent the screenshots as per your description

awanishkumar
Автор

Can you pls add English subtitles as well to all your videos

swagatikaskavita-ekkoshish