filmov
tv
Building Data Pipelines with Open Source by Rustam Mehmandar
Показать описание
@CJUG February 16, 2022
ABSTRACT:
A few years ago, moving data between applications and datastores included expensive monolithic stacks from large software vendors with little flexibility. Now, with frameworks such as Apache Beam and Apache Airflow, we can schedule and run data processing jobs for both streaming and batch with the same underlying code. This presentation demonstrates the concepts of how this can glue your applications together and shows how we can run a data pipeline locally with Apache Beam, and using Dataflow, and BigQuery by changing a few lines of Java in our Apache Beam code. We will be looking at how this can be deployed in different cloud solutions.
🗣SPEAKER BIO🗣
Rustam Mehmandarov
Passionate computer scientist. Java Champion and Google Developers Expert for Cloud. Public speaker. Ex-leader of JavaZone and Norwegian JUG – javaBin.
ABSTRACT:
A few years ago, moving data between applications and datastores included expensive monolithic stacks from large software vendors with little flexibility. Now, with frameworks such as Apache Beam and Apache Airflow, we can schedule and run data processing jobs for both streaming and batch with the same underlying code. This presentation demonstrates the concepts of how this can glue your applications together and shows how we can run a data pipeline locally with Apache Beam, and using Dataflow, and BigQuery by changing a few lines of Java in our Apache Beam code. We will be looking at how this can be deployed in different cloud solutions.
🗣SPEAKER BIO🗣
Rustam Mehmandarov
Passionate computer scientist. Java Champion and Google Developers Expert for Cloud. Public speaker. Ex-leader of JavaZone and Norwegian JUG – javaBin.
0:05:25
0:10:41
0:12:21
1:06:06
1:04:27
3:03:43
0:10:34
1:17:53
0:31:38
1:07:55
1:28:37
0:49:07
0:00:09
0:12:38
0:09:48
0:00:58
0:11:32
0:07:55
0:36:48
0:02:26
0:50:02
0:57:54
0:32:07
0:32:28