Apache Kafka® Brokers: Introduction to the Data Plane

In this module, Jun Rao (Kafka Committer, PMC Member, VP of Kafka, and Co-Founder, Confluent), provides an overview of:

- Apache Kafka® broker architecture and how it works
- The request loop that produces and fetch requests cycle through as they are processed by the broker
- How network threads and i/o threads are utilized to process requests
- How request and response queues operate
- How page cache is utilized as brokers write events to topic partition logs
- The role purgatory plays as the broker waits for events to be replicated to other brokers

LEARN MORE

ABOUT CONFLUENT

#streamprocessing #apachekafka #kafka #confluent

[toenytv]

Learnt lots. Great job. Thank you Confluent.

[SaiSahithiMangalapalli]

Awesome video!
But in the end - it's the I/O thread that gets blocked as we fetch the data from the disk

[koteshwarraomaripudi]

Do we still use zero copy principle in case we dont have enough data to send? As there was a mention of purgatory map incase data is not ready or enough.
Can someone please answer 😊

[smartdude]

i had to watch 4 times to understand the logic

[isravertiz]

Great Job Confluent Team, I have a not-so-smart question: When the buffer of bytes is not full and the consumer request has been moved to the purgatory, how does the fetch request know (Other than time) that the request can be fulfilled now? Is there some sort of bidirectional flag updating the thread to move the request back again to the Network Thread and then to the request queue?
IVS

[SaiSahithiMangalapalli]

It's the I/O thread that gets blocked when fetching the data from Disk

[debabhishek]

@confluent just one question ? when you do write in the kafka, ( producer) . this goes in the socket receive buffer ok .. got it.. but in which broker instance.. as for the specific partition there is specific broker instance is assigned.. suppose I am writing to topic A in partition 1 and partition 2, , partition 1 and partition 2 are not handled by the same broker instance.. now .. how its works.. does where it goes there are 3 instances the Custer. for which instance socket buffer will receive it. ?

[TymexComputing]

LZ4 is nice feature :) - XZ alas could be nice feature