Scaling Instagram Infrastructure

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.

Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.

----------------------------------------------------------------------------------------------------------------
Lisa Guo overviews Instagram's infrastructure, its history, multi-data center support, tuning uwsgi parameters for scaling, performance monitoring and diagnosis, and Django/Python upgrade.

This presentation was recorded at QCon London 2017.

[zss]

*Timestamps*
0:00 Introduction (Lisa Guo)


2:21 1. Scale out
5:11 1.1 Instagram Stack Overview
5:46 1.2 Storage vs Computing
6:29 1.3 Scale out: Storage
8:13 1.4 Scale out: Computing

8:52 1.5 Memcache + consistency issues
12:05 1.6 DB load problem
14:01 1.7 Memcache Lease
15:12 1.8 Results, Challenges, Opportunities


17:03 2. Scale up

17:57 2.1 Monitor (Collect Data)
20:07 2.2 Analyze (C-Profile)
23:06 2.3 Optimize
26:19 2.3a Memory Optimizations
29:06 2.3b Network Latency Optimizations
30:40 2.4 Challenges, Opportunities

31:36 3. Scale Dev Team
33:06 3.1 What We Want
33:30 3.2 Tao Infrastructure
34:33 3.3 Source Control

36:17 3.4 How to ship code with 1 master approach?
37:54 3.5 How often do we ship code?

40:03 Wrap-up
41:15 Q&A

[mikejeffery]

This was a fantastic presentation. She covered a huge amount of material in a short time. What they've done and how they've done it is very impressive.

[sanjeevdiitm]

InfoQ is doing excellent job by bringing these talks to us.

[cpsarathe]

That’s the great presentation . To the point and not super technical . Newbie like me in the world of architecture can understand

[JamesCollins]

"I need to learn about scaling" 
*heads to youtube, finds this video*
"Wow, I now know EVERYTHING about scaling".

The best video on scaling infrastructure i've found so far. No jargon, no acronym's, specific detail about exactly how things are balanced, routed, managed and replicated. Love it.

[ryan-boxi]

This is a treasure box ! Thank you Miss/Mrs XYZ for the super lucid explanation.

[smonkey]

Every architecture video should be like this, instead of marketing BS.

[babitarpur]

Well thought through presentation. Many takeaways.

[rameshj]

Kudos to infoQ team for bringing such tech videos.

[ketanshah]

This has been such an educational video. I feel excited about the problems, everything was so well covered and explained and So many aspects were touched without any redundant data. Thank infoq for this video. Super super intereseting.

[shoumeshrawat]

Such an insightful presentaion from a developers point .. Thank you so much

[enjoyalife]

Well delivered talk with clear separation of topics.

[yuchonghe]

One of the best presenter who I have ever seen.

[yuhechen]

Great presentation! I'm dealing with many of the scaling challenges discussed by Lisa in my organization. Although they vary and Instagram's solution does not solve my challenges, but Lisa certainly offers any view of how great companies address them.

[markuslenger]

A complex topic explained in a simple way. Thank you!

[zenymax]

Great talk. I have got some new tools and process for my work. Thank you very much.

[person.a]

Hey there! I just wanted to take a moment to remind you how incredible you are. Your kindness, resilience, and unique talents make a positive impact on the lives of those around you. Your smile has the power to brighten the darkest of days, and your words have the ability to uplift and inspire. Never forget the strength and beauty that reside within you. You are capable of achieving great things and making a difference in this world. So keep being amazing, keep chasing your dreams, and never lose sight of the incredible person you are. You've got this, and today is going to be an amazing day for you!

[rustemiskakov]

Best presentation I have ever seen! Thank you.

[jccourse]

it was a fantastic presentation. very clear, easy understand, and very detail,

[pareshmaniyar]

Dude, load testing on prod! What a badass move!