Structuring Spark: DataFrames, Datasets, and Streaming - Michael Armbrust (Databricks)

Live from Spark Summit NYC on Feb 2016

// About the Presenter //
Michael Armbrust is the lead developer of the Spark SQL project at Databricks. He received his PhD from UC Berkeley in 2013, and was advised by Michael Franklin, David Patterson, and Armando Fox. His thesis focused on building systems that allow developers to rapidly build scalable interactive applications, and specifically defined the notion of scale independence. His interests broadly include distributed systems, large-scale structured storage and query optimization.

Follow Michael on -

[justlive]

the sample code is not compiled in 8:57

I try to fix it ~

val hist = {
case (name, people: Iterator[Person]) =>
val buckets = new Array[Int](10)
people.map(_.age).foreach { a =>
buckets(a / 10) += 1
}
(name, buckets)
}