DragonflyDB Architecture Overview, Internals, and Trade-offs - hitting 6.43 million ops/sec

### Other links

CS Engineering and Software Development books that I have read

Research papers that I have read

Until next time, stay awesome :)

Yours truly,
Arpit
#AsliEngineering

[ghoulbond]

I recently switched from Redis to DragonflyDB in one of my projects, and it's uncanny how your content always seems to anticipate my needs. Another fantastic video.
P.S. I'm a huge fan of your work! 👏

[ANKITMISHRA-flkq]

Content is truly a gem !! Get to learn a lot

[AbhijeetSachdev]

Thanks Arpit for such videos and deep dive.
In case of common queue for all data threads, each thread needs a mechanism to determine whether it should execute a particular task or not.
What happens when [data-thread-1] picks a task which[ data-thread-2] is supposed to execute ?
High contention will be there while writing in common-queue by IO threads and while reading by data-threads.

Is not having a separate message-queue for each data-thread a better approach ?

[freecourseplatformenglish]

Thanks man. I started watching your video to have better understanding system design and underlying principles.

[hello_interview]

This sounds very similar to the way that Redis cluster operates (shared nothing, partitioned keyspace via slots, etc). What would you say is the major conceptual distinction between this and e.g. running multiple Redis instances on a single host and addressing them as a cluster?

[hareeshrajendran]

I hope these kinda contents keep coming ❤

[kshitijtakarkhede]

Hey brother I am doing DSA, but 😶 it feels like I am forgetting things as I am advancing towards new algorithms ( so much platform out there gfg, leetcode, interviewbit etc) get overwhelmed sooo many questions and how much to solve....I am studying dsa solving standard question not able to solve new question going to new algorithms and the loop repeats ( I try to visualise, try to dry run still i forget) can you please help a little...

[DilipKumar-ijcf]

You said shared nothing.. but "queue" of shard thread shared between owner shard thread and all io thread? Also, wouldn't there be some kind of lock when enqueue and dequed? Even if there is a queue per io/shard thread, it is still shared right?

[comicalInsan]

seems like kafka partition strategy with redis?

[pratyushprateek]

So, message bus is like channels in Golang? Channels are synchronized in Golang and multiple goroutines can wait on channels. Is the message bus similar to that?

[lokeshwaran]

Do people run redis on such high number of cores?
Wouldn't it be same if we split them to much smaller machines with memory optimised (high memory), then dragonfly would perform very similar to redis right?

[vinit]

Hey Arpit, from where do you hear about all these new technologies? I mean do you come across them through your colleagues, or twitter?

[meetmehta]

Amazing explanation. one question : will message bus be shared among each thread?

[sujayispappu]

The same message bus arch also is an old and already existing implementation in low latency trading systems

[manmohanmundhraa]

how reliable is message-bus as mentioned in video ? in case of more load and message reprocess ....

[xGarden]

Hey Arpit please discuss regarding the Cluster mode of dragonflyDB and that compares to redis cluster setup, I read a doc from redis where they compared real life scenarios the cache use

[AashraiRavooru]

Redis too scales pretty well by following the I/O thread pool to worker thread via a queue pattern (like most modern server applications) and since most Ops are lightweight and memory bound its ridiculously fast and queuing doesn't go to infinity.
Dragonfly DB can dramatically reduce the cluster size not eliminate it, we would still want replication over multiple nodes to avoid a single point of failure. This sounds best for ridiculous scales where the cluster sizes were going to an unmanageable count.

[aarushjuneja]

What tool do you use to create these notes?

[basuta-dshrara]

How do you maintain reading and comprehension, I use to be able to just listen in lectures and ace exams.

Now, I can't even focus on reading, IDK what's wrong, any advice for fellow engineers ?

[viktorvelikov]

Great video. Can you make a KeyDB comparison?