14: Distributed Logging & Metrics Framework | Systems Design Interview Questions With Ex-Google SWE

200 videos and we're still talking about logs - thanks guys, you're the best

[martinwindsor]

Jordan might not be a pregnant, but he never fails to deliver.

[knightbird]

Talking points

Can intro about push (low latency, needs app changes (better data), version skew) vs pull (highly scalable, no app changes(generic data), service discovery)
2:07 High volume, time series, structured vs unstructured, text logs, data sink
4:26 Data aggregation (tumbling vs hopping)
6:30 Time series db (hyper table, chunks, partition (label, timestamp))
10:40 Text logs (elasticsearch)
12:40 Structured data (better encoding, protobuf, avro, schema registry)
16:05 unstructured data (post processing to structured data, flink -> file -> ETL using spark)
17:40 Analytics data sink (column but not tsdb, more like OLAP), use parquet files for loading (S3 vs HDFS).
23:45 Stream enrichment
25:15 Diagram

[siddharth-gandhi]

Bro's single handedly making me question studying ml over systems. Bravo on these videos!

[JapanoiseBreakfast]

How to build distributed logging and metrics in 3 easy steps:

1) Join Google.
2) Run blaze build //analog:all //monarch:all.
3) Profit.

Congratulations, you have now built distributed logging and metrics. Thank you for coming to my ted talk.

[chadcn]

Congrats on 200 videos mate! Keep up the great work 🚀🚀

[doobie]

Thanks a lot for your videos. Currently looking for a new job, brushing up/learning a lot about system design, watched lots of your videos recently. Appreciate your work. Keep it up!

[DavidWoodMusic]

My interview is in 9 hours
I hear your voice in my sleep
I have filled a notebook with diagrams and concepts
And I am taking a poopy at this very second
We just prevail

[VijayInani]

Why are you so underrated!!! You should have been famous until now (more than your current famous index!).

[SunilKumar-qhln]

Very informative video as always.
Was just thinking how the metric is pulled by prometheus (which will eventually store in the DB).
How the different clients responsibility is assigned to the aggregator pods so that metric is pulled exactly once from each client pods.

[OmprakashYadav-nquj]

Great system design content. One think i noticed voice is not sync with real video.

[prasenjitsutradhar]

Great content!....pls make a video on code deployment!

[deepitapai]

Great video as always! Why do you store the files on S3 as well as a data warehouse? Why not just store on the data warehouse directly from Parquet files? Is it that we need a Spark consumer to transform the S3 files before putting the data into the data warehouse?

[guitarMartial]

Jordan can TSDB's run in multi leader configuration as there are perpetually no write conflicts per se? And is that a typical pattern where a company might run multiple Prometheus leaders which just replicate data amongst themselves to get an eventually consistent view? Or is single leader replication still preferred? Thinking multi leader helps with write ingestion. Thanks!

[beecal]

thanks for the video
in 22:00 when we say Parquet files are partitioned by time, do we mean partitioned by the file creation time?

[mukundkrishnatrey]

Hi Jordan,
Regarding the post processing of unstructured data, can we do the batch processing in Flink itself, as it does support that, or it's not suitable for large scale of data?
What could be the size of data which can dealt by flink itself, after which we might need to use HDFS/ Spark?

PS :- Thanks for the amazing content, you're the best resource I've found till date for system design content :)

[shibhamalik]

Hey jordan nice video do u have any video on which databases support cdc and how ?

hey Jordan great video, does this require any sort of API design? given that we need to read through the data metrics does it makes sense to also describe the API structure, let me know your thoughts, thanks.

[shibhamalik]

Hey jordan what is the data source in the last diagram here ? Is it the VM pushing logs / serialised Java objects etc to kafka ?? U mean the application when it logs a statement that statement makes a push to kafka ?

Then what should be the partition key of this kafka cluster ? Should it be server id or a combination of server id + app name or how should be we structure this partition key ?

[timavilov]

U forgot to mention the tradeoff between polling and pushing for event producers

[Dozer]

Is it true that s3 files would still have to get loaded over the network for something like AWS Athena? That seems to be a data warehousing strategy that relies on native s3 and not loading all of it across network