Sharing a distributed computing system design from a real software problem

I recently had to help design a system to help improve the performance of a feature in our application at work. This is a typically approach you'd need to take in using async workers and a message broker you might often see in the industry for larger systems needing faster computation.

------------

[appel-]

honestly i find this content amazing, i just found your channel and i already watched like 5 videos. The only thing that got me thinking is the email throttling, couldn't you use another sqs or queue to process those emails? (1 email every 5 seconds so you don't hit that throttle), that way you don't have to limit the worker. I'm kind of a beginner in this so don't go to hard on me! 😅

[eshw]

That is so true, solving complex backend system issues is so much more rewarding than most FE stuff since it has such a big impact on users.

[ayushw]

Amazing video man, would love to see more system design related content, especially connected to real-world scenarios!

[lakhangupta]

The really amazing video for the system design beginners. Would love to see more videos and loved the tool you are using. Thanks

[Sergeyzp]

Your main problem is Chrome core in puppeteer, it is too resource intensive, as I understad you need to save specific state of page in pdf, one of possible solutions is creating aws spot instance, or make docker container for aws ECS with ec2 or fargate, and run it when new tasks come, that will parallelize it, and you only need to free resources via api after task is done.
Another solution but harder one, is rust wry engine, it much more efficient, but not sure is it possible to run it headless, and I think pdf still not implemented here, only system print dialog with pdf option, so pdf still need to be coded in rust.

[ihort]

You can parallelize merging pdfs too, this approach will be resource intensive but faster.

[MarkJKellett]

This pattern is scalable. However it's worth noting that an issue with this is observability and testing. Especially offline. So you need to invest in a decent offline setup so that engineers can work with this end to end.

Serverless offline is ok. But as these systems grow in complexity, often the difficulty becomes debugging and trying to e2e test it in a meaningful way.

[MarkJKellett]

Another issue Is that Web API gateway only allows for connections of a maximum of 30 seconds (or at least it did). This experience is poor for a user. So you may need to consider user experience and opt for notifications for a user that a long running task is complete

[steveoc]

Nice vid.

Have you done much go yet ? The design you are describing here is pretty much how you would solve this in go - using goroutines in place of workers, channels instead of SQS, and a waitgrp to synch everything.

Same design, but it fits in 1 small binary that can be run in 1 lambda. Less $aws bill maybe.

The bottleneck still seems to be puppeteer though. Be nice to find an alt method that uses a different approach, but gets the PDF you need. Hmmm.

[capital]

Very good job my man, how can I learn this types of advance topics

[window.location]

You could have stored completed tasks in another queue instead of Dynamo db right?

[muwongenicholus]

Which language were you developing this in?

[alxizr]

This system design is bad, not awful like the monolithic one at the start of the video.

You guys can improve 1000x if you add another queue after the first lambda worker and group messages of the same task request. If your final pdf consists of a static 250 standalone pdfs then you can publish each standalone pdf to the second queue which waits for 250 messages of the same task and triggered only when they are done and pending to be handled to create the final large pdf. Write that one pdf to the s3 bucket and use socket to notify the client side to fetch it.

You should work always to 100 % automation rather than consisting on recursive polling and having the user to manualy perform extra actions

[jpdsah]

You look like bollywood singer Sonu Nigam