Apache Kafka® Transactions: Message Delivery and Exactly-Once Semantics

In this tutorial, learn how transactional support in Apache Kafka® allows for atomic-level updates from Jun Rao (Kafka Committer, PMC Member, VP of Kafka, and Co-Founder, Confluent). You will learn how transactions enable exactly-once semantics (EOS), which form the basis for this functionality in clients, like the streaming database ksqlDB and the Kafka Streams application library. You'll learn how Kafka handles system failure with transactional functionality and a couple of caveats to using transactions that you should be aware of.

LEARN MORE

ABOUT CONFLUENT

#streamprocessing #confluentcloud #apachekafka #kafka #confluent

[santoshgangadhar]

At 9:03, its not clear how LSO is at 64 even though the txn at 62 seems to be open.

[samwei]

Thanks for the explanation.
I would like to point out a few points where it sounded vague and unclear to me.

1. The arrows at 3:09 are intertwined and are hard to comprehend easily.
2. 8:12, it sounds like the P0 P1 P7 messages are enqueued into the coordinator queue all at once. But I wonder if they are progressively enqueued as the transaction process steps forward.
3. 10:21, "the broker also returns a little bit metadata (for message #57) so the consumer can ignore the aborted record". But the Abort message (#61) has not been read yet. It's kind of unclear to me how Kafka includes the info "message #57 has been aborted" in the metadata.

Maybe I'm just being ignorant as I'm not experienced with Kafka.
Any illuminations would be appreciated.

[rajsaraogi]

Abort marker, fence off, honestly so much jargans to make understand things, kafka teams should come with simple explainations

[sandymenon]

I would say a good start at transactions....but the explanation could have been a bit crisper.

[davidebelvedere]

how can it guarantee that the committed marker is pushed on coordinator and on all destination partitions? it doesn't seem much atomic as an operation

[mehrdadk.]

If one using low-level one way to avoid to disable auto-commit feature in topic, and once write to debit balance topic is done we manually increase the offset. This way we ensure of app crashed, we didn't commit log, so we can restart. Nevertheless this feature of having transaction in stream is really cool.

[kovalevvvvv]

Lol, alot of words about "modern mq", but kafka doesn't support xa-protocol - you can't use kafka transactions with database transactions.

[paulpasquel]

New "revolutionary application" using standards mechanisms, that we have saw in Database and MQ for more than 40 years ... but not 2pc Surely Kafka will implement that in future ...

[r-rtz]

Dont understand a word from this guy. Can you guys get someone who speaks English ?