This aws lambda issue was impossible to debug

Got any suggestions of how we could have debugged this better, leave a comment.

My Courses

My Products

Useful Links

[festusyuma]

I tend to create dev environments on AWS so we can test on the exact same environment rather than running locally. that way we're able to eliminate a lot of the issues. I've spent the last year building different pipelines for different types of projects that enables my team to deploy their projects with one click and they can actually test it on their code on the proper environment

The issue with the package is it was probably declaring and using global variables, global variables are not redeclared on warm starts so it also has the same value as the previous run

[WebDevCody]

I left my space heater running on accident because it was cold in my office, sorry for the background noise for the first 5 minutes 🤣

[marwanaouida]

I like to always use `npm ci` instead of `npm i` on the CD script to make sure the same version I used locally is used on the server. Many package maintainers ship breaking changes in minor releases.

[eshw]

Hey Cody I want to say these AWS videos are so good. Cloud computing is a skill even entry level job seekers like me are sometimes required to know. I learn alot from you talking about your app. I usually just go the easy route and deploy my apps to Vercel but watching you talk about this is very helpful. Keep doing it!

[TizzyD]

Memories...I was the architect for a District Attorney's Office in a major US city a little less than a decade ago. I lead the creation of their discovery management system that collected information provided by the PD's investigators (including docs and multimedia file types), ran it through a review and redaction process for that discovery, then provided a portal for the indicated representation of a defendant to be able to download that content. Of course, full records management was implemented along side digital fingerprinting, case-by-case security, workflow management and tracking, access logging, etc. It had about 500 active lawyers and paralegals managing the redaction and publishing workflow. So, your technical discussion reminded me well of the project and the transformation that took place.

[hnto]

man second time almost missing the video cause of the new thumbnails😂
they fire tho and content on top as always🙏🏼

[rembautimes]

This is a very insightful video. So important to emulate production in test - thanks for helping to reinforce this point.

[marinajordan]

your debugging videos are always great 🔥 keep it up, it's awesome that you're making this information more accessible for everyone

[renanlisboa]

Your videos have great value. Great job! 👏

[sfulibarri]

At work we got on the habit of generating a uuid on each cold start, holding it module scope, and attaching it to the context object that gets written out with every logger call. Makes it really easy to tell if errors are instance specific.

[manchuratt]

Very important point. I am anal about ensuring there is never any dynamic versioning for dependencies. Ran into an issue due to dynamic versioning once and it was such a headache to deal with I made a decision to forge utilizing dynamic versioning as there are often cases where a patch version leads to breaking changes. Since then haven't faced an issue relating to dependency versioning. It is very critical that once you finalize your cargo that nothing is allowed to change unless you do it intentionally and with consideration for possible side effects.

[ryanway]

I have a few questions about some things mentioned:

1. What is the motivation for having someone update all the packages weekly? I would kind of expect things to break often if library versions are changed often. Is this used to stay ahead of security concerns?

2. You mentioned that it was hard to pinpoint which library bump was causes the issue. But shouldnt the logs have indicated that it was some sparticus call that was failing?

3. How was it actually determined that sparticus was the issue? You mention that it was a caching within the lambda runtime with that particular library, but what test finally determined that? Or maybe there was a bug reported against the new version of sparticus


4. Lastly, I think if your prod runs in aws and you are not able to develop against an aws account, you would run into tons of issues like this, where you cant reproduce bugs occurring in live environments, or having issues where something developed locally doesnt deploy properly to your testing environment, etc.

[rafavieceli]

I always try to avoid puppeteer because of the weight, it’s easier to transform html into pdf, but it has a price. About the upgrade version, I am the guy who do that and it’s hard do to, I think every project should have a logbook about each dependency, explaining why need need it, know issues. At the end of the day every dependency is a technical debt, some cost a little and others a lot.

[giuppidev]

I hope to bring on my channel the same clarity, you're always INSANELY clear! 🚀

[roguesherlock]

You explained the architecture and the bug so elegantly!
1. it's weird that it started breaking for a minor version. Does it follow a semver policy? I assume you also reported the bug upstream?
2. Do you have a checklist you go through when you start getting errors? Say you get a email that you've got 1k errors today, what next steps do you take ? I get that essentially we want to a. put the server into a known state, and b. figure out what went wrong. But there are so many steps in between. check logs, check error traces, check db status, check package updates, check git commits, ask the team, etc

[roku_on_it]

This was a really good and simple explanation.

[OetziOfficial]

I had issues creating pdfs from (altered) docx files on lambda, so after 4 days of reading and looking for all sorts of third-party libraries and toolings I use a container (docker) that has all the needed binaries and tools that I could use locally. then I had to fix a few bugs due to aws-lambda and it worked. Took me several days to hunt that bug down, but it was worth it. Now I don't have to hunt it down anymore.

Oh and yes, I tend to get rid of the `^` symbol on almost any package. Then I check if there is updates on any of them and I update only those where I see no/little differences in prod and update them. Overall great overview of the aws-lambda issue and good that you guys figured out how to solve it / how to maintain the same environment for prod / dev / test / staging. it's always good to know that the code running on dev is similar (or even better: same) as prod.

[verte]

Chromium on lambda layer is a tough topic. We were thinking about container lambda, because it will be possible to run chromium on AL2023, and if im not mistaken lambda is already using that image under the hood.

[TheWhoIsTom]

Woe awesome content. Im a beginner. But at sometime I will remember this if I ran into issues.

[Dalamain]

hey cody great video - at 11:21 what was the magical keyboard combo you used to highlight the carets like that... I'm aware you can select multiple lines vertically but only in a straight line instead of character matching per line? Thanks