How to Split an Array of Structs from JSON into DataFrame Rows in Apache Spark

Learn how to transform a JSON array into individual DataFrame rows in Apache Spark with this step-by-step guide.
---

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Split array of structs from JSON into Dataframe rows in SPARK

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Split an Array of Structs from JSON into DataFrame Rows in Apache Spark

When dealing with data streams in Apache Spark, particularly with JSON formats from sources like Kafka, you may find that your data isn't structured the way you need it for analysis. In this guide, we will address a common problem where we need to split an array of structs from JSON into distinct rows in a DataFrame. If you've encountered an output where all values are grouped in arrays instead of showing individual records, you've come to the right place!

The Problem

Imagine you are reading messages from Kafka through Spark Structured Streaming, and the messages are in the form of JSON arrays, like this:

[[See Video to Reveal this Text or Code Snippet]]

If you define a schema and attempt to read this data, you might run into an issue where your DataFrame output looks like this:

[[See Video to Reveal this Text or Code Snippet]]

The problem here is that the customer, sex, and country fields are shown as arrays instead of individual records. The goal is to reshape this output to flatten it into individual records like so:

[[See Video to Reveal this Text or Code Snippet]]

The Solution: Using Explode in Spark

To achieve the desired output, we need to use the explode function in Spark before making our selections. This function helps in expanding an array into separate rows, allowing us to get each element in its own row.

Step-by-Step Instructions

Define the Schema: You would start by defining the schema for your JSON data. Here’s how your schema would look:

[[See Video to Reveal this Text or Code Snippet]]

Read and Explode the Data: Once your schema is defined, use the from_json function to parse the JSON and the explode_outer method to flatten the array:

[[See Video to Reveal this Text or Code Snippet]]

Here’s what each part does:

from_json($"value", schemaAsJson): This reads the JSON formatted data according to the schema you've defined.

explode_outer(...) as "json": This flattens the array of structs into multiple rows.

.select(...): This fetches the individual fields you are interested in.

Final Output

Running the above code will give you the following output:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

By following these steps, you'll be able to split an array of structs from JSON into individual rows in a Spark DataFrame. This technique is not only critical for clarity in data representation but is also essential for subsequent data analysis and processing. Using explode makes it simple to transform your data into a suitable format for analysis, ensuring each record is distinct and easily accessible.

Now you’re ready to tackle similar challenges with streaming data in Spark! Happy coding!