Lessons learned form Kafka in production (Tim Berglund, Confluent)

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.

[sonunitjsr]

One may directly jump to the discussion on production issues which starts at 30:24

[nammals]

70% funny talks, 30% real content, start from 30:24

[BayrangSafar]

Hearing this, I have also got the motivation that I too can be a speaker like him. 🙏

[tekal]

I watched the video to get some lessons learned in production, but more than 50% of it is Kafka basics terminology and marketing.

[pajeetsingh]

Do you fork the send of each microservices so that it goes to Kafka as well as to destination.

[pajeetsingh]

Does one install kafka in every isolated microservice so as to enable it for "streaming"?

[ionolaru]

"Starts with an S I'm blanking on it" ?
I heard "starts with an S I am plunking on it" LOL, I thought that was just a very nice joke from Tim to suggest Splunk.

[KrishnaChaitanya]

To add, "Commodity Hardware" here apparently means machines with a minimum recommended RAM size of 64GB.
Would I call that commodity hardware? Probably no. But just wanted to point that out.

[shellac]

So how many brokers do you guys usually stand up at your companies? Are you doing this in Azure?

[sachintiwari]

Q1: can a broker manage multiple partitions of same topic? In other words, can no. of broker be less than max no. of paritions in a topic?

Q2: If answer to above is 'no' then can we say that in order to add new partition we need to add new broker first?

Q3: What is best way to reduce no. of partitions? Should one just delete that extra partition, also will the broker handles deletion of its replica as well? OR is there a way to deactivate a partition I.e. set that extra partition to read only mode?

[RoboRope]

Tell me slow...what is a consumer group? Are they applications?

[aditya]

TBH ... They should have more experienced person giving these talks ... This guy does not seem to have any hands on real experience

[dokwme]

Just to make it clear
1 Topic
4 Partitions
Replication Factor of 3
and yes of-course 4 Brokers

[ankitlakum]

live simple life is actually a solution ..👍

[rahulguharay]

were you talking about Splunk :), the log processor :)

[balfiman]

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.

[santiagomoneta]

anyone from confluent can add... what changed from this talk to 2020?.. if any..

[jinandtalsania]

Does cloudera official support confluent product?

[nigelrmtaylor]

SIEM - I think he was the word he was looking for. But I might be wrong...

[AnyFactor]

Here I am with a 15 gb json file looking for a versatile db system.