Apache Airflow for Data Science #7 - How to Work with REST APIs

Показать описание

Learn to send and receive data between Airflow tasks with XComs, and when you shouldn't use it.

00:00 Introduction
00:38 Configure Airflow for communication with REST APIs
04:06 Write the Airflow DAG
10:28 Outro

FOLLOW BETTER DATA SCIENCE

FREE “LEARN DATA SCIENCE MASTERPLAN” EBOOK

GEAR I USE

Рекомендации по теме

Комментарии

Thanks a lot, Dario! Your lessons are really useful.

paleface_brother

great explanation, I have been struggling to find a resouce on how to trigger a dag run through Stable Rest Api which will trigger a dag placed in different airlow environment.

AmitSharma-yidr

If the method save_posts is failing it's because you are getting no response. Set up the dependencies like this

task_is_api_active >> task_get_posts >> save_reports

You can test the entire DAG with the command: airflow dags test dag_id 2023-03-6

Date should be in the past.

WEST

Hello! First of all thanks for the video! It is very good. I am doing a GET to a public API, when I run it outside Airflow it runs perfectly. But when I do it inside Airflow, just like the video, it returns me a 403 error. What could it be? I am using Airflow with Docker.

LuisFelipe-mbdq

Hi, How to use timeout option in httpsensor task. I have tried in extra_options ={'timeout':120 } also I have set poke_interval=30 means for every 30sec it should poke the endpoint and get the response and wait for 120sec of timeout and exit. But this scenario is not working in my case. Can anybody help.? Much Appreciated and thanks in Advance.

jaganmohan

Hi Dario,

Thanks for this. I would like to know what advantages does this simplehttpsoperator offer compared to pulling data from API using Python operator. thanks.

forumpost

Amazing! Thank you for keeping it sweet 'n' simple

DataNgineering

Thanks a lot for the video!

A question, do you must use xcom to fetch api output to save to disk? Is there maybe a more efficient way?

jonimatix

I have a question, how to do 200 asynchronous api calls to different end points?

praveenreddy

Apache Airflow for Data Science #7 - How to Work with REST APIs

Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

Axel Goblet: Scheduling machine learning pipelines using Apache Airflow | PyData Eindhoven 2019

Talk: Daniel Imberman - Bridging Data Science and Data Infrastructure with Apache Airflow

Luther Hill: Use Apache-Airflow to Build Data Workflows | PyData Indy 2019

Data Science Platform at PlayStation and Apache Airflow

Building the Data Science Platform with Airflow at Near

Daniel Imberman & Gonzalo Diaz: Data Products, From 0 to data science pipeline | PyData Córdoba

Apache Airflow for Data Science #7 - How to Work with REST APIs

Building (Better) Data Pipelines with Apache Airflow

Data Engineering Course | Apache Airflow Tutorial | Data Engineering Training | Intellipaat

Apache Airflow for Data Science #10 - How to Work with Variables

Apache Airflow at Wise

Apache Airflow Meetup: Elegant Data Pipelining with Apache Airflow

Airflow tutorial for beginners | Airflow tutorial python | Airflow tutorial 1

Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project

Getting Started with Apache Airflow

Apache Airflow for Data Science #4 - Migrate Airflow MetaData DB to Postgres and Enable Parallelism

Apache Airflow for Data Science #1 - How to Install Airflow Locally in 10 Minutes

Apache Airflow for Data Engineering | Apache Airflow | Data Engineering | Intellipaat #Shorts

Simplifying the Creation of Data Science Pipelines with Airflow

Data Science at Scale: From Notebook to Production with Airflow and Ray

Book Launch - Data Pipelines with Apache Airflow

Apache Airflow in under 60 seconds

Jean-Martin Archer & Michael Petro - Running Apache Airflow at Scale | PyData Global 2022