Designing a Realtime Gaming Leaderboard - Horizontally Scalable and Highly Available

Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.

### Other links

CS Engineering and Software Development books that I have read

Research papers that I have read

Until next time, stay awesome :)

Yours truly,
Arpit
#AsliEngineering

[LeoLeo-nxgi]

Suggestion to everyone, also read the technical queries/comments here, and check Arpit's answer, thanks guys and thanks a ton Arpit!!

[MarathiNationOne]

Your content looks real examples and experienced. Not like others just to prepare the topic and making videos.

[anujpanchal]

Hi Arpit,

One doubt I had after viewing the final architecture.
What is the frequency of tailing that rockset does to DynamoDB to fetch the recently written records?
I am asking this because the title of the video says we are designing a real-time leaderboard and we are fetching the data from the transactional DB to an analytics DB like Rockset. This tailing action might have some replication lag wherein the Rockset polls the recently written data on DynamoDB and when DynamoDB sends the data to Rockset. This can introduce some replication lag which can affect some users when we talk about millions of concurrent users. Because if a user tries to access the leaderboard immediately after winning a quiz he/she would expect that his name should be in top segment(s) and if he/she doesn't find his/her name in the top segment(s) then there are high chances that the gaming company might lose out on a genuine user.

Just wanted to know your view on this 😃

[Aditya_Vyas]

Thank You Arpit for always posting without forgetting it. I know today we have less subscribers & views but I know you have travelled it long & one day people will understand the value of your content. They will love your passion of posting videos/content about System Design & they'll understand it's more than just drawing boxes.

Art shines in the darkest days. I'm always with you & I owe you a lot whenever I learn from your piece of content.

Thank You for being there ❤

[rark-of]

great series !!! Complete fascination ... looks more clean, compact, modern architecture solving lot of problems at all level of the architecture !!
thanks for sharing this knowledge with the world

[mukeshtandale]

I always wondered how games scale their infrastructure. Thanks for this great video!♥️
Would love if you continued a bit about how MMORPGs work.

[freecourseplatformenglish]

Hey Arpit, Thanks for detail video Now I can answer a lot os system design questions.

[sachinjain]

It's good to know about Rockset. I have a few questions:

Does Rockset store all the data, including data from 5 years ago, or only the most recent data, such as from the last 5 days? If so, what is the eviction policy?

Is it advisable to store precomputed records for 1 month, 2 months, 6 months, 1 year, and 5 years in an SQL database?

For example, when a request is made to view the leaderboard for the past 45 days, the request would query both MySQL and Rockset. The data for the first 30 days would be fetched from MySQL, and the data for the remaining 15 days would be calculated from Rockset. The combined data would then provide a fast calculation for the 45-day leaderboard.

Throw some light if this fits In the design

[abhaysoni]

So, Arpit, we are implementing the CQRS pattern and segregating read and write operations to different databases with eventual consistency. We are accepting the trade-off that we may sometimes have to deal with stale data.

[shardulsilswal]

As always great video Arpit!
In order to be consistent, do we implement something like write through to ensure the leaderboard queries show the latest data? This would increase the write latency and might affect the availability of the system?

[pratikkulkarni]

Hello, thanks for your videos. I recently found them and have been enjoying them. A slightly off-topic question - I see you have mentioned the technical books you have read mentioned in the video description; have you read them cover to cover or do you follow some other practice? How do you recommend reading and possible remembering technical books?

[namantyagi]

can we use elasticsearch instead of Rockset and use something like CDC to get updates from the transactional system? Elasticsearch too is optimized for aggregation queries just like RocksetDB

[someshu]

Hi Arpit, thanks for a knowledgeable video. I will surely learn more about dynamodb + rockset.
I work in a gaming company and we use Redis for leaderboards.

[ankitraj]

I have on question, you mentioned that performing writes and reads at scale on a single database is not going to work, but lets say in the design we are ingesting data from Dynamo DB to Rockset which means that Rockset is also having a high write workload and simultaneously it’s processing read queries too, so wouldn’t Rockset face issues in this case?

[dippatel]

Rockset is shutting down after acquisition by openai. Customer had to move out. 😅

[gauravmadan]

Great video! this does solve the problem for a quick and reliable solution.

What would be the approach if some company don't want to go for a managed solution? Like can this be implemented via normal data pipelines, using spark. Assuming the SLAs for the metrics/leaderboards are a bit relaxed, like 5 min.

[AbhishekPrakash]

Hi Aprit, can you please explain why did we use dynamo db in the middle and not directly ingested the data to the rockset db? Is rockset lacking some other things that dynamo db does better?

[sachinmukherjee]

Thanks for this awesome video. Can you please make a video explaining the query execution plan of a database?

[divjyotsethi]

Nice video! Why use dynamo on write path and not just have rockset? Are there additional requirements which dynamo supports?

[hariikrishnan]

Could you explain more about why read replica was a bad choice ?