if else in pyspark | when otherwise | case when | Lec-14 | spark interview questions

preview_player
Показать описание
In this video I have talked about how to transform dataframe in spark. In this video I have talked about distinct, drop duplicates, sort and many more concepts. Please do ask your doubts in comment section.

emp_data = [
(1,'manish',26,20000,'india','IT'),
(2,'rahul',None,40000,'germany','engineering'),
(3,'pawan',12,60000,'india','sales'),
(4,'roshini',44,None,'uk','engineering'),
(5,'raushan',35,70000,'india','sales'),
(6,None,29,200000,'uk','IT'),
(7,'adam',37,65000,'us','IT'),
(8,'chris',16,40000,'us','sales'),
(None,None,None,None,None,None),
(7,'adam',37,65000,'us','IT')
]

For more queries reach out to me on my below social media handle.

My Gear:-

My PC Components:-
Рекомендации по теме
Комментарии
Автор

Sir aap bahut acha padhate hai, sab kuch samajh mai aa jata hai

titukumar
Автор

nice lecture manish bhai you are doing good for people like appreciate for your hard work

sinhadeepak
Автор

Thank you Manish bhai for this video:)

luvvkatara
Автор

cant we create a function regarding age status and apply on dataframe???

rhhrccr
Автор

Please provide some data engineering project on the basis of skills which u have earlier mentioned on data engineer roadmap

RajuSingh-mskr
Автор

123
emp_df.withcolumn("adult", when (col("age")<18, "No")
.when(col("age")>18, "Yes")
.otherwise("Novalue"))
AttributeError: 'NoneType' object has no attribute 'withcolumn'
I am getting this error. kindly help me

debritaroy
Автор

I know sql so shall i ignore dataframe api commands?

ChetanSharma-oyge
Автор

TypeError: 'Column' object is not callable ye kaise thik kare?

nikhilhimanshu
Автор

Hello dost yah spark kitna dikhana hai. Aaur kab interview dena suru karna hai. Please batao. I have learned so so maney things already. Your theory video was helpful. Thank you.

narsinghvachane
Автор

Sir, in Industry which is mostly used spark sql or dataframe code ??

anweshkumarsahoo
Автор

hello Manish,
Is it possible to create DF from given data. Basically my question is where data contains "None" vale. If yes how we can create a data frame.

jywvntd
Автор

I want to talk for interview preparation. how can i contact to you?

raajnghani
Автор

Sir paid course laye data engineering ka

titukumar
Автор

emp_data_df.withColumn("age", when(col("age").isNull(), lit(19))
.othwerwise(col("age")))\
.withColumn("adult", when(col(age)>18, "YES")
.othwerwise("No")).show()

column object is not callable is the error

poojajoshi