Spark Interview Question | Scenario Based Question | Explode and Posexplode in Spark | LearntoSpark

In this video, We will learn how to Explode and Posexplode / Explode with index and handle null in the column to explode in Spark Dataframe. Hope this video will be be useful for your Spark Interview Preparation.

[SatishKumar-fryv]

Small correction: explode() won't take Struct types; It takes only array and map types

[localmartian]

I checked and posexplode is still not working with withColumn in spark 3. Analysis exception

[yaniv]

Thank you for the series, it is very useful

[muraliintooru]

Could you please create a video on "how to implement SCD type 2 using pyspark"?

[sumantaghosh]

Hi Shahul, in this example we have only one array, now we have multiple struct array . can you pls explain how to do that ? thanks in advance mate

[SpiritOfIndiaaa]

Thanks, can you please make some samples using agg with struct of arrays??

[rohithreddysureddy]

What if there are two or more columns that need to be exploded ??

[anilkumar]

Hi bro in pyspark dataframe how to find each column null values count

[sangamrathore]

Hey Azar,
I need your help here.
I am trying this example but getting this issue below
This is the input file I am using
input file - explode_posexplode_ex.csv

Name|Age|Education
Azar|25|MBA, BE, HSC
Sangam|27|BE, HSC
Dilip|MBA, BBA, HSC
Ajay|BE, HSC


then I created df as below
df = spark.read.option("delimeter", "|").csv("/FileStore/tables/explode_posexplode_ex.csv", header=True)

But I am getting only one col instead of 3. When i execute below command output is shown below

df.show()
df.columns

(1) Spark Jobs

|Name|Age|Education|

| Azar|25|MBA|
| Sangam|27|BE|
| Dilip|MBA|
| Ajay|BE|


Out[42]: ['Name|Age|Education']

[chaitanyakrishna]

can you post all possible interview questions

[maheshtej]

bro how to remove delimeter from spark i need answer brooo