Intro to Apache Spark Streaming | NewCircle Training

[Topics Indexed below] Spark Streaming is a scalable, high-throughput, fault-tolerant tool set for consuming continual streams of data using the Apache Spark framework. In this presentation from the Philly Area Scala Meetup, instructor Brian Clapper starts with a brief overview of Apache Spark, before diving into Spark Streaming and how it works.

00:00:00 - Introduction
00:03:26 - Brief Overview of Apache Spark
----------- 00:07:51 - Quick Architectural Overview
----------- 00:16:21 - RDDs
----------- 00:27:05 - Transformations & Actions
----------- 01:00:20 - DataFrames
01:18:29 - How does Spark Streaming Work?
01:42:25 - Things you can Stream / Databricks Notebook Environment

Spark Training from NewCircle:

Philadelphia Area Scala Enthusiasts Meetup:

[kdcool]

One of best talks on Spark Streaming.. Great Work

[jasperavisser]

Excellent presentation. Kudos to Brian, he's a good speaker and he clearly knows his stuff!

[timothymisner]

The actual spark streaming content begins at ~1:18:00 The prior is a very good review of spark, rdd, dataframes.

[SanjayRoy-gmej]

Interestingly "Dataframe" is R concept and i guess with R now being run in Spark cluster, Dataframe is included in Spark. Great presentation Brian!!

[abduljabbarazam]

Really useful video on spark and spark streaming.

[vikramfugro]

Great presentation by a mile compared to others!

[synapss]

Thank you very much Sir, Excellent Job !

[rexche]

Can we stream the data from different sources and try to evaluate their differences? Can someone help me with more info on how we can stream data from many sources?

[nilanjansarkar]

What happens if the machine currently running the receiver thread crash? will another instance be started be on a different thread?

[STEWART.HR]

A very good intro to Spark Streaming. Lots of really useful information. Very good presentation by Brian.

[mycityofsky]

Does anyone know where I can get the PPT used in the video?

[raghavanal]

Does Data frame support an xml schema? Please clarify

[no_more_free_nicks]

@7:15 -> I'm sorry but this example is in my view a very bad piece of Scala code. I would rather use named case classes with the Dataset interface. It takes 2 minutes longer but is much more readable. The code shown here just spreads the word that Scala is a cryptic and hard to understand language. It doesn't really to have to look this bad.

[JulienReszka]

I didn't know George Lucas can do programming too

[CaptainDylan]

Great presentation on Spark and Spark Streaming. Very informative.

[SpiritOfIndiaaa]

thanks a lot Brian, can you please share the notebook?

[mycityofsky]

Amazing tutorial! I feel so lucky to find the brilliant lecture.