AWS re:Invent 2023 - Netflix Maestro: Orchestrating scaled data & ML workflows in the cloud (NFX308)

As big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have become increasingly important for Netflix. To better serve various business needs, Netflix has developed a new workflow orchestrator called Maestro, which greatly improves the usability and scalability of the orchestration system and boosts the productivity of the engineers working with data. Maestro provides support to run super-large-scale workflows, consisting of hundreds of thousands jobs with nested structures, and also maintain the lineage info between event signals, workflows, and tables.

Subscribe:

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#AWSreInvent #AWSreInvent2023

[MrQsam]

the Maestro source code is now open-sourced