Airflow: Automating ETLs for a Data Warehouse, Natarajan Chakrapani, SF Python July 2018

Natarajan Chakrapani, a software engineer at Optimizely, describes using Airflow to automate ETL pipelines for a data warehouse.

Like the talk? Please give our speaker a thumbs up!

This video is through the generosity of our sponsor PlanGrid
PlanGrid is creating software for the 14 trillion dollar construction industry. Our mission is to be the record set for every job site in the world. Construction is stuck in the past - we are bringing it into the future through field first mobile applications backed by advanced machine learning techniques. PlanGrid has adopted Python as its language of choice for rapidly building advanced file processing pipelines and cloud services optimized for mobile offline use. With Python as the backbone of our product, we’re a proud sponsor of the SF Python Meetup.

[reachmalgan]

Thank you Natarajan .. I am very new to airflow, your presentation is very helpful to me.

[Albert-fejx]

Thank you Natarajan. Airflow in incubator, donated from airbnb.

[l_combo]

so he said Airflow isn't doing the file transfer, what are people normally using for that SFTP or similar?

[denger]

Thank you natarajan . good presentation. i need the airflow commands to setup in my environment on centos 7, can pls provide if any one knows.

[vamsibhagavatula]

Can I use it for datastage and schedule with Airflow?

[venkyownstyle]

can anyone help me with kafka integration with airflow please