Apache Spark Announcement | Single node data science meets big data | Keynote Data + AI Summit 2021

preview_player
Показать описание
Reynold, Co-founder and Chief Architect at Databricks, will talk about the latest in Apache Spark and Koalas and how data scientists can move beyond the laptop. Brooke, a data scientist and machine learning expert, will then demonstrate and discuss some of the new features in Apache Spark as the community improves usability for Python and data science.

Stay tuned for some big news!

Register for free to see the rest of the keynotes and exciting announcements live, plus over 200+ sessions. Learn from the creators and top contributors of technologies like PyTorch, TensorFlow, MLflow, Delta Lake, Apache Spark, Hugging Face, DBT and more.

Connect with us:
  1. Databricks
  2. Databricks
  3. Apache Spark
  4. Koalas
  5. Data Scientist
  6. big data
Рекомендации по теме
Apache Spark Announcement 0:15:42

Apache Spark Announcement | Single node data science meets big data | Keynote Data + AI Summit 2021

3 CRAZY Announcements 0:00:59

3 CRAZY Announcements from Databricks AI Summit 🤯 Apache Spark Updates 🔥

What is Apache 0:13:41

What is Apache Spark? Learn Apache Spark in 15 Minutes

Apache Spark Core—Deep 1:30:18

Apache Spark Core—Deep Dive—Proper Optimization Daniel Tomes Databricks

What’s Next for 0:37:48

What’s Next for the Upcoming Apache Spark 4.0?

Enabling Vectorized Engine 0:21:59

Enabling Vectorized Engine in Apache Spark

Deep Dive into 0:48:39

Deep Dive into the New Features of Apache Spark 3.1

What is Apache 0:02:11

What is Apache Spark?

Databricks Cloud Announcement 0:44:48

Databricks Cloud Announcement and Demo at Spark Summit 2014

Koalas: Pandas on 0:58:16

Koalas: Pandas on Apache Spark

Announcing Koalas Open 0:21:33

Announcing Koalas Open Source Project | Reynold Xin (Databricks), Brooke Wenig (Databricks)

Fine Tuning and 0:25:19

Fine Tuning and Enhancing Performance of Apache Spark Jobs

Unifying State-of-the-Art AI 0:16:19

Unifying State-of-the-Art AI and Big Data in Apache Spark - Reynold Xin (Databricks)

Introducing Apache Spark 0:29:00

Introducing Apache Spark 3.0 | Matei Zaharia and Brooke Wenig | Keynote Spark + AI Summit 2020

Spark Wordcount Demo 0:23:09

Spark Wordcount Demo | Apache Spark and Scala Tutorial

Accelerated Data Science: 0:21:26

Accelerated Data Science: Announcing GPU-acceleration for pandas, NetworkX, and Apache Spark MLlib

Apache Spark Performance: 0:27:33

Apache Spark Performance: Past, Future, and Present with Kay Ousterhout

Project Hydrogen - 0:18:38

Project Hydrogen - State of the Art Deep Learning on Apache Spark (Reynold Xin)

Lessons from the 0:36:17

Lessons from the Field:Applying Best Practices to Your Apache Spark Applications with Silvio Fiorito

Accelerating Apache Spark 0:35:41

Accelerating Apache Spark by Several Orders of Magnitude with GPUs

Apache Spark and 0:28:55

Apache Spark and Tensorflow as a Service - Jim Dowling

PySpark Kickstart - 0:37:15

PySpark Kickstart - Your first Apache Spark data pipeline

From Query Plan 1:02:35

From Query Plan to Performance: Supercharging your Apache Spark Queries using the Spark UI SQL Tab

Operationalize Apache Spark 0:22:55

Operationalize Apache Spark Analytics

Комментарии
Автор

Based on the title of this talk, I expected an improved single-node install experience. Still not pip installable nor installable via any OS package managers like apt/brew/chocolatey? And nothing to ease the burden of enabling AWS/Azure/GCP storage? If these are announced elsewhere, please lmk. The workspace install experience is still very painful IMHO, and I still struggle to find turnkey full featured docker images designed for local workstation use cases

aaronsteers
Автор

I dislike the fact that Databricks is not considering Spark for Scala more than it's considered for python.

The main issue I see here is that the flexibility or the ease of writing both programs are equal on databricks, but writing an end to end application in python is straightforward which is not the case in scala. Here, you need to manually download the script, convert to jar file, upload and then it can run. And repeat the cycle for every small change you make.

this is the kind of resistance that I think is driving developers to switch towards python.

on other hand, SQL i agree is really good, but they r often written with main application in python/scala where the above resistance and choice favours python.

.Counting
Автор

Glad to see pandas being implemented into pyspark. I liked koalas, but yeah this will be better

twinkazz