DSS Asia 2021 | Creating Standard Patterns to Load Data into Yugabyte with Apache Airflow

Apache Airflow is an open source workflow orchestrator that is used by data teams of all sizes. Given its flexible nature, Airflow is used to interact with everything from legacy on-prem systems to distributed, Kubernetes-native applications. As organizations undergo digital transformations, users responsible for data workloads are often tasked with coordinating these legacy and cloud native systems concurrently. The growing number and complexity of these workloads causes teams to require standardization layer around for maintainability, reusability, and extendability.

This talk will showcase how data platform engineers can provide a standard template for users to perform business-specific actions with Apache Airflow to replicate Postgres data to a distributed Yugabyte instance. It will end with a brief overview of the Airflow Provider framework.

Viraj Parekh
Head of Field Engineering,
Astronomer

and

Pete DeJoy
Product Manager,
Astronomer