Nikolay Matvienko I Build and automate serverless ETL job orchestration with AWS Lambda

Dynamic Talks I JavaScript meetup I December, 26, St. Pet.

With the growth of steps in Extract Transform Load (ETL) jobs the complexity of managing data flow orchestration is growing. While serverless computing allow us not to worry about infrastructure and scaling, calculations are performed on the distributed nodes of the pipeline, the number of which can reach thousands in scale. This complicates the control of completion of the data processing steps and the ETL job in general. And in this case automation of ETL plays a critical part in defining and reporting processing status and job completion.