Microservice Pitfalls: Solving the Dual-Write Problem | Designing Event-Driven Microservices

When building a distributed system, developers are often faced with something known as the dual-write problem. It occurs whenever the system needs to perform individual writes to separate systems that can't be transactionally linked. This situation creates the potential for data loss if the developer isn't careful. However, techniques such as the Transactional Outbox Pattern and Event Sourcing can be used to guard against the potential for data loss while also providing added resilience to the system.

RELATED RESOURCES

CHAPTERS
00:00 - Intro
00:24 - How the system was supposed to work.
01:18 - Why didn't the system work properly?
02:31 - How to use the transactional outbox pattern to fix it.
04:19 - Why event sourcing might be a better fit.
05:08 - How the transactional outbox and event sourcing can improve reliability.
05:52 - Closing

--

ABOUT CONFLUENT

#microservices #apachekafka #kafka #confluent

[MrGoGetItt]

Exceptional content delivery! Not only were you articulate, but the visuals were an excellent aid. Great work

[petermoskovits]

At 1:25 you cover in great detail how to address the problem when the Kafka write fails and the DB write succeeds. How about the other way around? What if the Kafka write succeeds, and the DB write fails?

[AxrelOn]

Great video, thanks! In case of outbox pattern usage the outbox records should be removed once kafka producer received ack, but could batching prevent that? Should batching be disabled in this case?

[ghoshsuman]

What occurs if the Change Data Capture (CDC) process attempts to write to Kafka, but Kafka is unavailable? how to write the retry logic ?

[soloboy]

hell. I'm working as a Kafka admin. recently I have faced one issue that, for few topics (each topic has 6 partitions) one partition end offset become zero. how to resolve this. it is in production pls help

[Fikusiklol]

Hello, Wade!
Wanted to ask about CDC with ES option.
From my perspective publishing events (domain/internal) is wrong, as they might not contain enough data and consumers would be coupled to internal contracts (there are ways to avoid it, but just for simplicity sake).

Did you mean somehow reading domain event, transform it to integration (ESCT/fat) and then publish using CDC?
Cause ive been doing ES and still used Outbox table to fulfill that intergration event with metadata.

[darwinmanalo]

So instead of manually sending events to Kafka, we save the events to the database first. Then, there is a CDC tool that detects updates and automatically sends them to Kafka?

Another tool adds another layer of complexity. Event Sourcing is quite complex, so people should carefully consider if it's the right tool for the project before implementing it. I wish these inconsistencies/issues were already solved in Kafka itself, not by us.

P.S. The presentation is well-explained though. Wade is a good teacher.

[ilijanl]

You can actually leverage legacy db transaction to publish to kafka with some tradeoffs. The flow can be following:
1. Start transaction
2. Insert into legacy db
3. Publish to Kafka
4. Commit

If step 2 or 3 throws, nothing will be committed and the whole handler will fail, which can be retried later. If for some reason 2&3 succeed and 4 fails, you have published the event to kafka without storing to db however now you have atleast once for publishing.
Tradeoff is ofcourse that your runtime has a dependency on Kafka, and if kafka is down, you never can succeed the transaction. However they say kafka is HA and high performance so the problem might be smaller than it seems.