How to Merge Spark DataFrame - Complex type | Spark Scenario Based Question | PySpark | LearntoSpark

In this video, we will learn how to merge two Spark dataframe of different schema with complex datatypes in it Using PySpark. Hope this Spark Scenario based interview question will be useful in your Spark Interview preparation.

fb page:

[sravankumar]

Superb explanation bro, but little bit difficult to understand 😕

[tarunpothala]

Hi Azarudeen, I find your videos beneficial. Thanks for sharing this content. I request you to make videos on how to read and write data from\to cloud storages into Databricks.

[nareshreddy-lf]

Thanks for uploading nice content videos ...

[khushbuagrawal]

@Azarudeen - It will be good that If you can put the sample data or file in the description/comment section. So that we can copy paste and do the practical.

[wasim]

Thanks for this demo. Can you please help me with any suggestions for one my requirements. It would be great if you have any input on this. I want to do merge like feature in pyspark with target data frame having 2 extra column like isupdated and timestamps. If record is updated in source data frame then isupdated column will be set to yes otherwise no.

[rajasekhar]

Hi azzar, All videos r u show very usefull. Please can show on spark with scala also in shell scripts

[keepsmile]

Nice session, is thre way Age and other columns should come as a different columns Insted of list education column

[rudrasingh]

Hi, Its very helpful . Could you please share data and code with github link, so that we can easily practice. Take it as a request, Please

[samisajjad]

list1=[]
list2=[]
for i in l1:
print(i)



i am getting en error message like
----> 5
IndexError: list index out of range

[ashishkumavat]

Hi Azarudeen Shahul, I following your videos from the last 1 months and I found it very useful. I would like to make a request, is it possible to upload a video showing how to read data from google sheet using spark scala. I searched for it everywhere but couldn't find it.

[gayathrilakshmi]

Hi,

Can you please attach the data file and code file for us to practice..very useful video :)

[TheSumanb]

HI thnks bro for the hard work, could you please share the code repository ?

[sravankumar]

CAN YOU PLEASE SHARE THS CODE IN DESCRIPTION BOX IT WILL BE USEFUL FOR US