Apache Spark Week Day 2 | #101

preview_player
Показать описание
On day two of the Apache Spark week we look major Apache Spark concepts: RDDs, transformations and actions, caching and broadcast variables.

Also check out Jacek Laskowski book on Spark:

Check out my free 100+ pages data engineering cookbook on GitHub:

Please SUPPORT WHAT YOU LIKE:

(Send a message and I read it on the stream)

- YouTube SuperChats while live streaming

- As an Amazon Associate I earn from qualifying purchases from Amazon. Just use this link:

- I get asked a lot about my podcast gear. This is a list of all the equipment I currently use to create this Podcast:

#ApacheSpark #DataEngineering #PplumbersofDataScience #bigdata
  1. Andreas Kretz
  2. data science
  3. big data
  4. data engineering
  5. data science for beginners
  6. data engineer
Рекомендации по теме
Apache Spark Week 0:58:25

Apache Spark Week Day 2 | #101

Apache Spark Week 1:05:35

Apache Spark Week Day 1 | #100 (2nd try)

Apache Spark Week 1:28:51

Apache Spark Week Day 3 | #102

'Exploring Wikipedia With 3:40:29

'Exploring Wikipedia With Apache Spark' - Part 2, Advanced Training by Sameer Farooqui (Da...

Extending Spark SQL 0:41:00

Extending Spark SQL 2 4 with New Data Sources Live Coding Session -Jacek Laskowski

End of Week 0:56:15

End of Week Apache Spark Code Review - part 2

Starting the Apache 0:47:57

Starting the Apache Spark Week | #099

Advancing Spark - 0:19:39

Advancing Spark - Data + AI Summit 2022 Day 2 Recap

Apache Spark Week 1:02:08

Apache Spark Week Day 4 | #103

OpenShift Commons Big 0:19:27

OpenShift Commons Big Data SIG #2: Running Apache Spark Natively on Kubernetes

[PART 2] Apache 0:37:04

[PART 2] Apache Spark - Making Sense of Big Data Faster and Easier

Deep Dive into 0:34:15

Deep Dive into Query Execution in Spark SQL 2 3 with Jacek Laskowski

Query or Not 0:26:02

Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries

Spark SQL 2 0:37:20

Spark SQL 2 0 Experiences Using TPC DS (Berni Schiefer)

Two days Apache 2:19:22

Two days Apache Spark Workshop Day1

Apache Spark Code 1:01:23

Apache Spark Code Review - Follow up edition - with PR merges

Enabling Vectorized Engine 0:21:59

Enabling Vectorized Engine in Apache Spark

Apache Spark Tutorial: 0:56:50

Apache Spark Tutorial: Apache Spark for Beginners | Apache Spark |@OnlineLearningCenterIndia

Conquering Hadoop and 0:27:20

Conquering Hadoop and Apache Spark with Operational Intelligence (Akshay Rai)

Developer Last Expression 0:00:28

Developer Last Expression 😂 #shorts #developer #ytshorts #uiux #python #flutterdevelopment

Exploratory Analytics 101 0:04:56

Exploratory Analytics 101 with Apache Spark for Synapse and Notebooks

DataFriday #2 - 0:18:04

DataFriday #2 - Basic ETL ops with Apache Spark

Apache Spark Lessons 0:13:16

Apache Spark Lessons Learned Part 2

Keeping the fun 0:46:47

Keeping the fun in Apache Spark Datasets and FP by Holden Karau

Комментарии
Автор

Hi Andreas! It worths to mention that using spark caching (or memory only persisting) for significant amount of data it's almost inevitable to stumble into OOM problem and solving it will require changing of spark memory management settings (see spark.memory.fraction and

And regarding the broadcast stuff and accumulators... First one is just a way to distribute some data needed for computation among executors. It's not writable on workers. And accumulator is like counter in MapReduce framework. It's needed to pass some information from worker to driver. So, it's writable from executors, but can be read from driver only.

dmitryamosov
Автор

Hi Andreas, Thank you such a great initiative, are you doing this streaming daily ?. A question here, what will be difference between SparkContext.addFile() and broadcast variable? first adds a file to a distributed cache later also add smaller file into executors memory.

sumityadav
Автор

It will be ur great philanthropy on me.

KK-lrjq