Advanced Step Functions Patterns with Ben Smith

In this episode, we continue chating with Ben Smith, a developer advocate at AWS.
We talk about the advanced patterns Ben has developed through hands-on experience and experimentation with step functions. These patterns, used widely by organizations on a daily basis, include handling large lambda functions, REST CRUD API using step functions, Saga and Circuit Breaker patterns for error management, and leveraging step functions for choosing the right compute resources.
The discussion covers dynamic parallelism with the map state and distributed map state for handling up to 10,000 parallel executions, showcasing real-world applications and optimizations.
Ben also shares tips for efficient state management and the value of learning through building and iterating on solutions.
The conversation highlights several serverless architectures, demonstrating how step functions facilitate complex workflows, handle errors gracefully, and optimize costs and computing resources.

Table of contents:
00:00 Welcome to the World of Step Functions with Ben Smith
01:27 Diving Deep into Step Function Patterns
02:13 Exploring the REST CRUD API Pattern
10:53 The Saga and Circuit Breaker Patterns: Managing Errors in Step Functions
16:30 Choosing the Right Compute: Lambda vs. Containers
21:18 Exploring Advanced Step Functions
22:21 Parallel Processing with Step Functions
24:34 Introducing the Data Preserver Pattern
25:26 Dynamic Parallelism and Map State
29:52 Optimizing with the Distributed Map State
31:13 Real-world Application: GIF Generator
33:46 Leveraging Distributed Map State for Data Analytics
36:28 The Evolution of Building and Learning
39:00 Wrapping Up and Looking Forward

Find Ben in the internet

Links mentioned in this video:

#foobar #serverless

#foobar #serverless

☆☆ FOLLOW ME ONLINE ☆☆

📷 Instagram: foobar_codes

☆☆ ABOUT FOOBAR ☆☆
In this channel, you can find mostly coding tutorials related to cloud and serverless.
In addition, I like also talking about architecture, software design, motivation, and leadership.
There is a new video every Thursday, so stay tuned :)

[alanwalsh]

This two part series is very useful - especially the patterns. Love the channel!

[Nova-ulvv]

Just came around here after looking into a giant black hole of a workflow, where analyzing all state transitions is super painfull. Will look into data preserver pattern thx :)

[taicodev]

Well, actually it is not hard to have one codebase to run in both Lambda and Container.
#!/bin/bash
# Assuming "index.js" exports a function named "handler, " Adjust the required path as necessary to match your project structure
node -e "const { handler } = require('./index'); handler()."

From here on, u can tweak that Shell script as well and simulate lambda call by passing parameters to handle that you would be extracting from env variables once u start your task :)