New Developments in the Open Source Ecosystem: Apache Spark 3 0, Delta Lake, and Koalas

In this talk, we will highlight major efforts happening in the Spark ecosystem. In particular, we will dive into the details of adaptive and static query optimizations in Spark 3.0 to make Spark easier to use and faster to run. We will also demonstrate how new features in Koalas, an open source library that provides Pandas-like API on top of Spark, helps data scientists gain insights from their data quicker.

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.

Connect with us:

[alaham]

Very nice. Things will be more exciting using spark with all these new features. One question though, on an existing installation running spark2, How easy is the upgrade?

[chinmayabarik]

Deltalake & koalas going to be the game changer in the field data analytics

[NitinPasumarthy]

Where is `plot` function at 17:05 coming from? Does "Apache" Spark natively support displaying dataframes in "Jupyter" notebooks?

[AnirbanNagDev]

Can we get the partition pruning demo or video here in YouTube?

[MPXVM]

when you want to scroll down the Jupyter file in video 30:00 and you end up scrolling YouTube page

[yuanji]

Do I need to install Koalas on every node of cluster or just on the master?

[sahihe]

22:57 can someone give some details about that “forecast=true”

[vincenttan]

6:28.. I wonder if the audience clapped because of 2X or being able to save a few lines of codes..

[UkrozaVR]

Invite me next time I'll initiate loud applause on right moments, just feel like it's missing on such a presentation :)

[dreznik]

nice! when will koalas be available for R?

[albertoandreotti]

People applauding optimizations that SQL had already introduced 25 years ago. Really kids don't study databases anymore?