How Rockset achieves zero data latency and workload isolation at scale

Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.

### Other links

CS Engineering and Software Development books that I have read

Research papers that I have read

Until next time, stay awesome :)

Yours truly,
Arpit
#AsliEngineering

[abhinavcv]

Video Idea: An interview with one of the lead engineers who created UPI, how does the tech behind it work? And many more questions you could ask.

[DevNiklesh]

Very well explained. Enjoyed learning this Rockset series. Thanks for the KT

[TheEarthlyEnigmas]

Hi Arpit, thanks a lot for making these videos. One question. Are compute nodes theleaf nodes?? Since in previous video, you told the rocketset architecture contains ingest nodes, leaf nodes and aggregator nodes.
Also if compute nodes are leaf nodes, why do they sync data between other nodes, because the aggregator nodes will anyway query all the the leaf nodes to get their respective data ?

[manujpant]

veryyyy interestinggg. thank you for the video.really curious about the deep functioning of things and your are the few amongst here that explain the deep tech.

[devendrawangikar]

Hello Arpit, Thanks for breaking down complex concepts about Rockset and RocksDb. Kudos to this series. Can you please continue this series and create videos and explain concepts like how rockset does database replication in distributed system, how they optimized the ingestion process for large datasets, how do they stream/replicate the memtable for compute-compute solution?

Than you so much again!

[mavurugangadhar]

Nice explanation,
any idea on how network failure while syncing data between compute nodes handled and how the data is transferred

[raj_kundalia]

Q1. So, SSD's also read from the memtable of the node that is ingesting data and share it with memtable of replica's or there is a separate sync between different memtables?

[prashanthb]

Nice architecture. But this means they should be having a really large memtable which adds to memory costs. Or does this memtable sit on local SSDs ?

[ajaysharma]

Keep this series going, loved the postgres one too ❤

[deekayjindal]

There are alot of aspects where data can go inconsistent in above explanation. What if SST files are not loaded in hot storage and there is write then each compute node will have a piece from SST table. And if query came for that SST table then data has to be read from S3 which can be inconsistent

[arpit]

Multiple applications reading from the same database, why would that happen? Is it not an anit-pattern to let multiple applications access the same database.

"A DB should be owned by exactly one service or two closely related service (EX: feed generation vs feed fetch)"

Please share your thoughts on this.

[vaibhavtyagi]

The content of Arpit's video is generic which will not be outdated wrt technology evolution. This content will remain relevant for a longer time .