Pyspark Scenarios 9 : How to get Individual column wise null records count #pyspark #databricks

Pyspark Scenarios 9 : How to get Individual column wise null records count #pyspark #databricks
Pyspark Interview question
Pyspark Scenario Based Interview Questions
Pyspark Scenario Based Questions
Scenario Based Questions
#PysparkScenarioBasedInterviewQuestions
#ScenarioBasedInterviewQuestions
#PysparkInterviewQuestions
employee data file location :

Databricks notebook location:

Complete Pyspark Real Time Scenarios Videos.

Pyspark Scenarios 1: How to create partition by month and year in pyspark
pyspark scenarios 2 : how to read variable number of columns data in pyspark dataframe #pyspark
Pyspark Scenarios 3 : how to skip first few rows from data file in pyspark
Pyspark Scenarios 4 : how to remove duplicate rows in pyspark dataframe #pyspark #Databricks
Pyspark Scenarios 5 : how read all files from nested folder in pySpark dataframe
Pyspark Scenarios 6 How to Get no of rows from each file in pyspark dataframe
Pyspark Scenarios 7 : how to get no of rows at each partition in pyspark dataframe
Pyspark Scenarios 8: How to add Sequence generated surrogate key as a column in dataframe.
Pyspark Scenarios 9 : How to get Individual column wise null records count
Pyspark Scenarios 10:Why we should not use crc32 for Surrogate Keys Generation?
Pyspark Scenarios 11 : how to handle double delimiter or multi delimiters in pyspark
Pyspark Scenarios 12 : how to get 53 week number years in pyspark extract 53rd week number in spark
Pyspark Scenarios 13 : how to handle complex json data file in pyspark
Pyspark Scenarios 14 : How to implement Multiprocessing in Azure Databricks
Pyspark Scenarios 15 : how to take table ddl backup in databricks
Pyspark Scenarios 16: Convert pyspark string to date format issue dd-mm-yy old format
Pyspark Scenarios 17 : How to handle duplicate column errors in delta table
Pyspark Scenarios 18 : How to Handle Bad Data in pyspark dataframe using pyspark schema
Pyspark Scenarios 19 : difference between #OrderBy #Sort and #sortWithinPartitions Transformations
Pyspark Scenarios 20 : difference between coalesce and repartition in pyspark #coalesce #repartition
Pyspark Scenarios 21 : Dynamically processing complex json file in pyspark #complexjson #databricks
Pyspark Scenarios 22 : How To create data files based on the number of rows in PySpark #pyspark

pyspark count null values in each row,
find null values in pyspark dataframe,
pyspark groupby count null,
pyspark count rows,
percentage of null values in a column pyspark,
pyspark check if column is null or empty,
pyspark filter null values,
pyspark is null or empty.
count no of null values in dataframe,
count number of nulls in a column sql,
count number of nulls in a column pyspark,

pyspark sql
pyspark
hive
which
databricks
apache spark
sql server
spark sql functions
spark interview questions
sql interview questions
spark sql interview questions
spark sql tutorial
spark architecture
coalesce in sql
case class in scala

databricks,
azure databricks,
databricks tutorial,
databricks tutorial for beginners,
azure databricks tutorial,
what is databricks,
azure databricks tutorial for beginners,
databricks interview questions,
databricks certification,
delta live tables databricks,
databricks sql,
databricks data engineering associate,
pyspark databricks tutorial,
databricks azure,
delta lake databricks,
snowflake vs databricks,
azure databricks interview questions,
databricks lakehouse fundamentals,
databricks vs snowflake,
databricks pyspark tutorial,
wafastudies databricks,
delta table in databricks,
raja data engineering databricks,
databricks unity catalog,
wafastudies azure databricks,
unity catalog azure databricks,
delta lake,
delta lake databricks,
how to get delta in red lake,
delta sleep lake sprinkle sprankle,

[mohitupadhayay]

I am impressed way beyond words now. Thanks, Siva. Teachers like you are a boon to students like us.

[ianmendes]

Brilliant. That "col is null" expression is so simple. I was trying with when, select, and isnan statements and failed. This is much better, thanks mate!

[sussyguevara]

I rarely comment on videos. But thanks so much. I really needed to know how to use this to find Null!! And it was so easy vs other methods!

[alwalravi]

Thanks for this video..its help lot for me

[shoaibulhaque]

This is really helpful Thanks.
I have a follow-up question on this, can we also get the percentage of nulls for each column.

[prabhatgupta]

d={}
for i in df.columns:
d[i]=df.filter(f"{i} is null").count()

[sushmamc]

Hello..how do I create a column with all null values using with column function?

[arijitdutta]

Hi Sir,

Can you please help me to achieve below transformation & logic for newdjoin dataframe using list comprehension or any other method in fewer lines

from pyspark.sql.functions import when

newdjoin = (
djoin.withColumn(
"New_ID", when(djoin.t_id.isNull(),
)
.withColumn(
"New_firstname",
when(djoin.t_firstname.isNull(),
djoin.t_firstname
),
)
.withColumn(
"New_middlename",
when(djoin.t_middlename.isNull(),
djoin.t_middlename
),
)
.withColumn(
"New_lastname",
when(djoin.t_lastname.isNull(), djoin.s_lastname).otherwise(djoin.t_lastname),
)
.withColumn(
"New_dob", when(djoin.t_dob.isNull(),
)
.withColumn(
"New_gender",
when(djoin.t_gender.isNull(), djoin.s_gender).otherwise(djoin.t_gender),
)
.withColumn(
"New_salary",
when(djoin.t_salary.isNull(), djoin.s_salary).otherwise(djoin.t_salary),
)
)


Source Dataframes for your reference


d1 = [(1, 'James', None, 'Smith', '1991-04-01', 'M', 20),
(2, 'Miel', 'Ros', None, '2000-05-19', 'M', 40),
(3, 'Rt', None, 'Wams', '1978-09-05', 'M', 40),
(4, 'Ma', 'An', 'Js', '1967-12-01', 'F', 40),
(5, 'Jn', 'Mry', 'Brn', '1980-02-17', 'F', -1)
]

d2 = [(11, 'ABC', 'XYZ', 'MNO', '1991-04-01', 'M', 30),
(12, 'CED', 'JKL', None, '2022-05-19', 'M', 30),
(13, 'Robert', None, 'Will', '2000-09-05', 'M', 40),
(14, 'Maria', 'Ann', 'Jones', '1967-12-02', 'F', 40),
(15, 'JLM', 'Mary', None, '1970-02-14', 'F', -1),
(1, 'James', None, 'Smith', '1991-04-01', 'M', 20)
]

columns1 = ["t_id", "t_firstname", "t_middlename", "t_lastname", "t_dob", "t_gender", "t_salary"]
columns2 = ["s_id", "s_firstname", "s_middlename", "s_lastname", "s_dob", "s_gender", "s_salary"]
d11 = spark.createDataFrame(data=d1, schema = columns1)
d12 = spark.createDataFrame(data=d2, schema = columns2)

djoin = d11.join(d12, d11.t_id == d12.s_id, "fullouter")
djoin.display()