Operating Data Pipeline with Airflow

Presenter: Ananth PackkilDurai @ Slack

Slack is a communication and collaboration platform for teams. Their millions of users spend 10+ hrs connected to the service on a typical working day. Airflow is the core system for orchestrating our data pipeline. About six months back, they started on-call rotation for our data pipeline. This talk covers the incremental steps they took to solve on call nightmares & Airflow scalability issues to make their data pipeline more reliable and simple to operate.