Design a High-Throughput Logging System | System Design

Logging systems are commonly found in large systems with multiple moving parts. For these high-throughput real-time systems, there are a number of challenges and considerations at scale. This video gives a high-level introduction to some of these challenges and how to overcome them.

Table of Contents:
0:00 - Introduction
0:27 - Requirements
1:33 - Naive Solution
2:18 - Sharding
3:07 - Bucketing
4:15 - Sharding and Bucketing Combined
5:05 - Migrating to Cold Storage
7:00 - Next Steps

Socials:

[supragya]

i dont understand, if under same bucket lets say for (2021-2022) we have multiple nodes, how are reads any faster ? for the same bucket logs will be distrubuted across servers and still need to be queried across servers which is slow . Bucketing didnt help in improving read performace, is my understanding .

[wizz]

Kafka -> Loki -> S3
If you're looking for an existing solution :)

[developerjas]

Great Video man! Would how would you go about designing the data ingestion part?

[GoofGoof-csny]

So in 2018 every service was writing logs to node 3, didn't we went back to bad write complexity by doing bucketing?

[ankushraj]

Why not use Kafka for high through put?

[sahanagn]

Great video!!! Please slow down the speed of video as someone new to topic its bit fast to grasp the concept.

[didimuschandra]

Greatt video!! thanks! but, can you create video to develop Effective and efficient Ticketing System?

[weidada]

Suppose every two years, it ingest 2PB and migrate 1PB, how could three sets be enough to cycle after 12 years?

[prakharsrivastava]

I love the cute computer in the background

[taboaza]

By 2026 you will have 2 clusters with 2PB (2022-2023, 2024-2025) of data and one with 1PB of data (2021). What do you do then? 😅