Introduction to workflow orchestration with Prefect - Kevin Kho, community engineer, Prefect

Talks from 2022's Data Reliability Conference (DRE Con)

Modern data applications are as complex as ever, and along with this complexity also comes an ever-growing number of ways these applications can fail. Data sources can go down intermittently. Input data may come in malformed. To guard against this, a non-trivial amount of effort is spent creating code paths to handle these failures gracefully. These include task retrying, execution timeouts, and notifications in the event of failure.

Collectively, the effort spent guarding against failure is called negative engineering. Workflow orchestration frameworks are specifically designed to reduce the effort spent on negative engineering by enabling the scheduling and monitoring of workflows. Failure code paths can be added so that the application knows how to respond in the event something unexpected happens.

This talk will cover basic workflow orchestration functionality and we'll familiarize ourselves with the features that let us handle these problems gracefully.