Tutorial 3- Pyspark With Python-Pyspark DataFrames- Handling Missing Values

Apache Spark is written in Scala programming language. To support Python with Spark, Apache Spark community released a tool, PySpark. Using PySpark, you can work with RDDs in Python programming language also. It is because of a library called Py4j that they are able to achieve this.
---------------------------------------------------------------------------------------------------------------------------
Subscribe my vlogging channel
Please donate if you want to support the channel through GPay UPID,

Please join as a member in my channel to get additional benefits like materials in Data Science, live streaming for Members and many more

Connect with me here:

[mbmathematicsacademic]

walking into the weekend knowing PySpark, what a feeling

[utkarshkapil]

Watching this in April 2024!
Thanks for making this unbeatable content Krish!! Also love your ongoing langchain series!

[tanishasharma]

The na.fill('Missing values') was not working for the integer columns like Age/Experience/Salary
Only if we read the dataset as 'inferSchema=False' and all the columns are taken in string format by default we can fill the missing values with a string like 'Missing goes for the string columns if we fill the missing values with a '0'
Just an input....:)

[prernakhurana]

Thanks for this Playlist, please include deep learning with Spark and Koalas library too in this Playlist. 🙏

[bhargavikoti]

Thank you sir.. expecting more for pyspark playlist🙏

[khushboojain]

@11:02 the Missing Values could appear in Experience, Age and Salary as the data type of these columns was string. If it was Integer, it would have not appeared. Hope it helps someone!

[tableauvizwithvineet]

thanks Krish,

Note - I was getting bad result using - result was only filling first column and not other columns, using df_pyspark.fillna( { 'Age':0, 'Experience':0 } ).show(), it was working correctly. I am not sure why its happening in my system only.

[faisalmomin]

Hey krish can you pls make a video about communication skills for data scientists and how i develop it .

[ajaysinha]

Can we get the mean or median separatly and apply over the particular column

[sociallyviral]

Can you play only python for data analyst roles. Series

[paulasam]

Hello, Krish
Can you Please make a video on Music genre classification using AI electronic projects.?
It would be a great help.

[syedalinaqi]

Hi Krish.
Can you please let us know when will you make video on Bert ?

[paritoshprasad]

df_spark= df_spark.na.fill('missing value', ['age', 'salary']) I have tried to fill the null values but not working.

[sankarazad]

Hii Krish, When I ran imputer part it is giving me an error saying "IllegalArgumentException: requirement failed: Column names must be of type numeric but was actually of type string."
How to resolve this??

[gururajraykar]

Krish .. could you please explain spark architecture?

[ramthiagu]

How to impute categorical feature. "mode" is not working with categorical feature.

[raviluminary]

Hi Krish, Thanks for the videos..
One query How come it is filling integer columns with 'Missing Value' when you set inferSchema = True.
Also I found the reason why? when I saw your DataFrame (11:54) column data types all are strings, so can you help why it did not read inferSchema

[anirbaansaha]

Hey..in this section when u r writing inferschema=true then also it is taking all as string, hence when u r doing full with missing keyword as a value it is getting applied in all rows.
But when I am doing it with inferschema=true then fill is only happening fr string rows not all...please clarify this

[vigneshbalaji_]

can you also make an video on spark Architecture which will be really helpful for us.

[anand]

jupyter commend suggestion now showing. Can you please help ?