1. Merge two Dataframes using PySpark | Top 10 PySpark Scenario Based Interview Question|

Hello Guys, If you like this video please share and subscribe to my channel.

Please enroll in data engineering project courses

Full Playlist of Interview Question of SQL:
Full Playlist of Snowflake SQL:
Full Playlist of Golang:
Full Playlist of NumPY Library:
Full Playlist of PTQT5:
Full Playlist of Pandas:

#databricks #interviewquestion #pyspark

[pratik]

Hey Sagar, we can also merge two data frames even if schema is different
eg.
df1.unionByNaame(alllowMissingColumns="True", df2)
it will merge the both data frames and other columns automatically come as null values

[rashmimishra]

Can it be done using below command.
merged_df = df1.join(df2, ["id", "studentname", "department", "city], "outer")

[siddhantkapur]

Hey you could give lit(None) instead of lit('null') as it a string rather than an actual null

[tanmaydash]

i think by using unionByname we should merge it easily and then with help of fillNA we can handel its null value ..

[grim_rreaperr]

Bro are you performing append operation or merge operation?

[shivamchandan]

Plz upload video on debugging in pyspark

[Tanveer_Shaikh_]

Could you please also include given dataframe data in the description so that we can practice at our end ?

[akashprabhakar]

please provide the dataframe data also in the info so that we can practice alongside :)

[santhoshreddy]

I understood your video…can you please explain all videos in English

[shreyashchoudhary]

please add more videos in this playlist

[thesadanand]

this is hard-coding of 'marks' column, what if we have mismatch in both dfs, how to dynamically do it ?

[avibitm]

bhai aap data bhi daalo
so that any body can practice

[tlshtmx]

How to apply schema to the data using a sql table using pyspark. sql table has 1000 column.

[savitak]

If column names are different, it is not a issue? For example in table 1 student_name in table 2 StudentName?

[gautambhandare]

copy the structure in notes so people can practice it looking at videos

[hackwithharsha]

Thank you, What is databricks, is it jupyter notebook on cloud ?