A Kafkaesque Series of Events • Lily Mara • YOW! 2022

This presentation was recorded at YOW! 2022. #GOTOcon #YOW

Lily Mara - Engineering Manager at OneSignal @lily-mara

RESOURCES

ABSTRACT
You may have heard that Rust eliminates concurrency bugs, and this is largely true.
This talk will be a deep-dive into the investigation of a concurrency bug that could not be caught by the Rust type system. We'll walk through the background on the asynchronous messaging systems at OneSignal that process billions of events per day, and discuss how a massive series of unfortunate coincidences caused almost a total stoppage of work. [...]

TIMECODES
00:00 Intro
00:35 Once upon a time
03:53 Kafka
08:55 Issues
10:40 Subpartition processing
15:02 Concession
16:12 Review
18:04 Goals
22:13 Observability
27:27 OneSignal
30:14 Why is that a problem?
35:37 Q&A
42:02 Outro

Download slides and read the full abstract here:

RECOMMENDED BOOKS

#ApacheKafka #Kafka #Apache #EventStreaming #EventDrivenArchitecture #EventDriven #KafkaTutorial #KafkaInAction #Concurrency #Streaming #Batching #DataIntegration #Programming #OneSignal #DataPipelines #StreamingAnalytics #ETL #Docker #LilyMara #YOWcon

Looking for a unique learning experience?

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.

[edgeeffect]

What we they thinking when they named Kafka? "I know", they said, "what we need is a name that conjures up images of a huge overwhelming oppressive bureaucracy that crushes the humanity out of everybody in it's path"????

[xasm]

why subques? maybe hash based partitioning?

[FourTetTrack]

I skipped a bit of the video, but I wonder how is this different from the Kafka Parallel Consumer (which I think is only available in Java).