AWS Tutorials – Building ETL Pipeline using AWS Glue and Step Functions

In AWS, ETL pipelines can be built using AWS Glue Job and Glue Crawler. AWS Glue Jobs are responsible for data transformation while Crawlers are responsible for data catalog. Amazon Step Functions is one approach to create such pipelines. In this tutorial, learn how to use Step Functions build ETL pipeline in AWS.

[arunr]

your channel is gold for data engineers. thanks for sharing the knowledge

[vaishalikankanala]

Clear and concise. Great work, thank you very much!

[coldstone]

This is amazing. Glad I found this on youtube. A million thanks.

[harsh]

Thank for your session, it helped me !

[pravakarchaudhury]

It is really awesome. A million thanks to you.

[veerachegu]

Really helpful and no institute will come to give training on this thankyou so much

[simij]

thank you a ton lot for doing this!!! <3

[terrcan]

Thanks for your this kind of tutorials,
Could you please share some of the scenarios for AWS Glue job along with Session as well as for AWS lambda.
And Also would like to understand the incremental load scenarios in AWS GLUE using HUDI DATASET and other scenarios on same topic

[kamrulshuhel]

Thank you so much for your nice tutorial.

I will be grateful can you respond, I have some understanding issues -
while I use condition in step functions workflow - not ($.state == "READY")

I am getting this error,

An error occurred while executing the state 'Choice' (entered at the event id #13). Invalid path '$.state': The choice state's condition path references an invalid value.

[nlopedebarrios]

Considering the continuous evolution of AWS Glue, what do you think is more suitable for a newbie: orchestrating the ETL pipeline with Glue Workflows or Step Functions?

[nlopedebarrios]

If the purpose of the ETL pipeline is to move data around, and the sources, stages and destination are already cataloged, why would you need to run the crawlers after each glue job is finished?

[rishubhanda]

Amazing video!! Could you please go over how to build something like this with a CDK? The visual editor is helpful, but I find it easier to provision resources with code.

[ravitejatavva]

@AWSTutorialsOnline, Appreciate your good work. AWS glue has evolved so much now, how can we in-corporate data quality checks to the pipelines and send email notifications to the users with dq fail results such as rules_succeeded, rules_skipped, rules_failed and publish the data to a quicksight dashboard. Do we still need step-functions ? Any thoughts / suggestions please.

[PipatMethavanitpong]

Thank you. This is a nice ELT demo. I wonder how do you handle past extracted and cleaned data.
Glue jobs are appending write only, so the raw bucket will contain both old and new extracts and the cleaning job will perform on both the old and new.
I think there should be some logic to separate old files and new files.

[veerachegu]

Really awesome video no where available this content small request can you do the one lab like while daily or hourly fils uploaded in to S3 and trigger the function from S3 to step function pipeline to end of the job

[lqgctwv]

Hello, good video. Maybe someone knows when use Glue workflows and when use StepFunctions?

[picklu]

Thanks for the video. If i use step function to orchestrate glue workflows, will that slow the whole process down?

[simij]

What would you advise if we have 150 tables to move from mySQL into S3 ( No business transformation- just dump load raw), to have them all in one step function to run parallelly or create individual pipelines to reduce the risk of if one fails all fails with all being clubbed together.

[BradThurber]

It looks like Step Functions Workflow Studio includes AWS Glue Start Crawler and AWS Glue Get Crawler states. Could these be used directly instead of the lambdas?

[anmoljm]

my data source is CSV files dropped into an s3 bucket which is crawled, and I trigger the crawler using a lambda to detect when an object has been dropped into the s3 bucket, how do I trigger the start of a pipeline consisting of Glue jobs upon the completion of the first which crawls my source s3 crawler?
I could use Workflows which is part of Glue but I have a Glue DataBrew job that needs to be part of the pipeline.