Apache Spark | Distinct Vs Drop Duplicates | Basic of Spark SQL | LearntoSpark

preview_player
Показать описание
In this video, we will learn about the difference between Distinct and drop duplicates in Apache Spark. We will discuss on what is the advantage on one over other. We will discuss with one example.

Blog link to learn more on Spark:

Linkedin profile:

FB page:
  1. Azarudeen Shahul
  2. Spark Interview Question
  3. Spark
  4. LearntoSpark
  5. PySpark
  6. Spark with Scala
Рекомендации по теме
Apache Spark | 0:07:37

Apache Spark | Distinct Vs Drop Duplicates | Basic of Spark SQL | LearntoSpark

Group by vs 0:09:41

Group by vs distinct in Apache Spark SQL

Deep Dive, distinct() 0:04:53

Deep Dive, distinct() vs dropDuplicates() in Apache Spark with Scala Part 3

91. Databricks | 0:11:41

91. Databricks | Pyspark | Interview Question |Handlining Duplicate Data: DropDuplicates vs Distinct

Deep Dive, distinct() 0:05:02

Deep Dive, distinct() vs dropDuplicates() in Apache Spark with Java Part 3

22. distinct() & 0:07:30

22. distinct() & dropDuplicates() in PySpark | Azure Databricks #spark #pyspark #azuredatabricks

Funnel Analysis with 0:26:16

Funnel Analysis with Apache Spark and Druid

Getting Started switching 0:15:11

Getting Started switching Apache Spark distinct() to External Append Only Map

Databricks Data Engineer 2:58:03

Databricks Data Engineer Professional Exam Practice Questions - ANALYSIS JULY 2024 (67Q)

Koalas: Pandas on 0:58:16

Koalas: Pandas on Apache Spark

Apache Spark Vs 0:26:11

Apache Spark Vs Apache Flink – Looking Through How Different Companies Approach Spark And Flink

Apache Spark Core—Deep 1:30:18

Apache Spark Core—Deep Dive—Proper Optimization Daniel Tomes Databricks

L8: Use of 0:06:19

L8: Use of .select() and distinct() in PySpark

Last Minute: Micro 0:11:03

Last Minute: Micro benchmark Apache Spark distinct changes

PySpark Tutorial 25: 0:06:57

PySpark Tutorial 25: Count Distinct, Concat, Length, Collect List | PySpark with Python

Challenging Web-Scale Graph 0:49:24

Challenging Web-Scale Graph Analytics with Apache Spark

Partitioning Vs Bucketing 0:00:46

Partitioning Vs Bucketing | Apache Spark Optimization Techniques #interview #question

PySpark Tutorial 21: 0:08:47

PySpark Tutorial 21: Alias, Distinct, OrderBy | PySpark with Python

Continued Apache Spark 0:11:52

Continued Apache Spark distinct() improvements PR update

Spark Transformation Types 0:14:05

Spark Transformation Types and Actions

Improve Apache Spark™ 0:37:55

Improve Apache Spark™ DS v2 Query Planning Using Column Stats

Apache Spark Optimization 0:00:59

Apache Spark Optimization techniques | Partitioning Vs Bucketing | Column Cardinality | #interview

Live Coding on 1:57:22

Live Coding on Apache Spark - In Scala: Improving distinct performance & looking at Scala 2.12

How to use 0:11:04

How to use Ranking Functions in Apache Spark | RANK | DENSE_RANK | ROW_NUMBER

Комментарии
Автор

Very well explained, its not just about the syntax but how drop duplicates works with subset of columns.

rahuldey
Автор

Easy, crisp and very useful videos as concepts are explained through code.
Thank you Azarudeen. Hope you continue making videos with different technologies.

ritikas
Автор

Very helpful tutorials. Could you please share the notebook presented in the Tutorial if possible ?

meditation_in_nature
Автор

Could you please tell how to pass as variable in drop duplicate function. Variable might have composite keys.. Can you give me tips

maheshk
Автор

Thanks for the tutorial Azar. It really help me to understand the concepts. I am new to pyspark. When I tried to same scenario using display function. It throws me a error. Could you please explain what’s the difference between show and display

Balajionceagain
Автор

I read somewhere that dropduplicates will remove the random rows is it correct if yes is it good approach

ravikirantuduru