15. Databricks| Spark | Pyspark | Read Json| Flatten Json

#ReadJsonFile, #SparkJsonFlatten, #JsonFlatten, #DatabricksJason, #SparkJson,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

Python Function to Flatten Json File:

#Flatten array of structs and structs
def flatten(df):
# compute Complex Fields (Lists and Structs) in Schema
while len(complex_fields)!=0:
print ("Processing :"+col_name+" Type : "+str(type(complex_fields[col_name])))

# if StructType then convert all sub element to columns.
# i.e. flatten structs
if (type(complex_fields[col_name]) == StructType):

# if ArrayType then add the Array Elements as Rows using the explode function
# i.e. explode Arrays
elif (type(complex_fields[col_name]) == ArrayType):

# recompute remaining Complex Fields in Schema
return df

[ChrisCrombieOfOregon]

Thank you for this! The flatten heirarchy behavior in the ADF Copy data activity was not working and this is such an elegant alternative solution in Databricks! Just what I needed. I appreciate your sharing it.

[sandrafield]

Thanks!! This is exactly what I needed, such excellent work!!!

[arvindcool]

excellent piece of information. I have a vice versa scenario, Is there a way to unflatten the data back to original structure? any pointer or reference or sample code would be really helpful

[Rafian]

Great video man.. that is what I was looking for.

[deepjyotimitra]

Thank you for this video.. this is wonderful. It's going to help many

[ririraman]

Brilliant and awesome tutorial. Thats what we need...

[BabaMalik]

Thank you so much. This function saves me lots of time.

[Jaipreksha]

Excellent and very neat explanation.Thanks for sharing you wonderful knowledge❤❤

[vipinkumarjha]

Thanks for the video. and sharing the very useful function.

[macdelvalle]

Great video and function! Works great, however I am receiving an error when I try and pass in a 2 data frames to be flattened. The first works as expected, but when running the function on the 2nd, it returns an error " 'str' object is not callable" and points to this line.. expanded = for k in [ n.name for n in complex_fields[col_name]]]. If I redefine the function in between each dataframe it works as expected.

I think it is something to do with the expanded variable. Any thoughts? Thanks again!

[HafizfromIndia]

Thank you so much! Very useful function 👌

[bhanuteja]

Great Videos. Its saved my time. Thank you so much. :)

[MyVaibhavraj]

This is excellent and thank you very much to provide this json parsing solution which will help in many json scenarios. 😍

[ATHARVA]

Hey Raja thanks for your superb real time interview series, I cleared interview process for KPMG global services as an azure data engineer

[sathyanarayan]

Hi Raja sir, I used this function to flatten deeply nested json but it is throwing the error "No such struct field com in ACID...." ...when I went through the schema of the json(nested), I saw cookies column(which is of struct type) has one field 'com.sx.attr : string (nullable=true)...Please help.

[shreyaspatil]

Thanks very much for the tutorial :), I have a query regarding reading in json files.

so i have an array of structs where each struct has a different structure/schema.
And based on a certain property value of struct I apply filter to get that nested struct, however when I display using printschema it contains fields that do not belong to that object but are somehow being associated with the object from the schema of other structs, how can i possibly fix this issue ?

[FranciscoDara]

Thank you, thank you thank you guy, you saved me!

[albinchandy]

what if I create a schema and then try to bind the schema to the json data, provided I know the json data structure already.

[sharathmutya]

Hi Raja, This is wonderful video. I have a quick question can the Json be flattened into multiple tables ?

[yvoncadieux]

Realy good video, can you share the JSON
I Will do some test on my side
Merci, Thanks