Live-Coding Session 2: PySpark Structured Streaming | Spark, Python, Streaming

#livecoding #spark #python #pyspark #streaming #softwareengineering #dataengineering #testdrivendevelopment

In this live-coding session, we will:

- continue our work on our PySpark Structured Streaming project,
- implement a slightly more complex use-case including an aggregation,
- adjust our test case to model timing dependencies,
- talk about output modes `append` and `complete`.

Along the way, I will explain important concepts of Spark's Structured Streaming functionality and show you how to find relevant information from official sources.


The following songs appear in this stream:

Song: Poylow & Mandrazo - Loneliness (feat. Barmuda) [NCS Release]
Music provided by NoCopyrightSounds

Song: BIMINI - No Way (with Avi Snow) [NCS Release]
Music provided by NoCopyrightSounds
Watch: AT/youtube

Song: Convex - 4U (feat. Jex Jordyn) [NCS Release]
Music provided by NoCopyrightSounds

Song: CiDE & Vide - Late At Night (feat. Jordan Grace) [Arcade Release]
Music provided by NoCopyrightSounds

Song: Naeleck - Burning Wish (feat. Roniit) [NCS Release]
Music provided by NoCopyrightSounds

Song: Low Mileage - Hold You [NCS Release]
Music provided by NoCopyrightSounds

Song: T-Mass & Britt Lari - Like Me [NCS Release]
Music provided by NoCopyrightSounds

Song: Coopex - Over The Sun [NCS Release]
Music provided by NoCopyrightSounds