Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021

Far from a controversial choice, Kafka is now a technology developers and architects are adopting with enthusiasm. And it’s often not just a good choice, but a technology enabling meaningful improvements in complex, evolvable systems that need to respond to the world in real time. But surely it's possible to do wrong! In this talk, we'll look at common mistakes in event-driven systems built on top of Kafka:

-Deploying Kafka when an event-driven architecture is not the best choice.
-Ignoring schema management. Events are the APIs of event-driven systems!
-Writing bespoke consumers when stream processing is a better fit.
-Using stream processing when you really need a database.
-Trivializing the task of elastic scaling in all parts of the system.

It's highly likely for medium- and large-scale systems that an event-first perspective is the most helpful one to take, but it's early days, and it's still possible to get this wrong. Come to this talk for a survey of mistakes not to make.

Topics covered:
-What is Event-Driven Architecture
-Data Mesh Principles
-Scaling
-State management

Recorded at Devoxx Poland 2021

Join us also here:

#Devoxx #DevoxxPoland #IT #Development #SoftwareDevelopment

[alexanderpodkopaev]

"If it is small program, you don't call it a monolith. Architecturally it IS one. I want to give you permission just embrace that" - BRILLIANT! Noted for further discussions with middle developers, thanks a lot Tim!

[atkinpaul]

Thanks! I hope you do engage more with Data Mesh. Zhamak seems to de-prioritize operational data flows but there is a massive opportunity to converge operational and analytic pipelines when you use an event driven architecture combined with polyglot persistence.

[darrenhorwitz]

this guy is brilliant, I love his sense of humor and what a great talk !!!

[KirBirger]

Thanks for this. There's a shortage of honest content out there that doesn't simply evangelize concepts and approaches.

[mosespeter]

This is one of the best talk I've ever listened to

[tanertasim]

This presentation is just gold!! Very clean and precise explanations! Thanks!

[joshzwicker]

Asked event-driven-architecture into your heart 😄👍nice metaphor

[marekchodak]

I usually watch on 1.5x speed but he is so good to listen i kinda dont want to

[ennyoy]

The last 15 minutes of 2001 a Space Odyssey 😂 I’d say that’s where all the meaning of the movie is. And yeah, don’t be afraid of a monolith if you’re building something small and isolated. If you can get something up and running to do the job with little up front effort or admin overhead, it’s probably fine to spend the extra effort to migrate to an event driven architecture in the future once you’re actually generating value. If you over engineer up front it will be far longer before you start to generate business value and it will probably cost more in the long run. Build fast, fail fast and iterate. Just avoid decisions which will lock you down and make iteration difficult. Be pragmatic and agile.

[mehrdadk.]

The reason for developers think when using event-driven is a database replacement can be, there is no example that uses the combination in Confluent materials. Maybe this can be getting done

[vanivari]

Don't want to complain too much, but from an expert like him i kinda expected a bit more than: "only use EDA if you need it", "Use DBs if they are a good idea", "the event schema is kinda important because stuff changes" and "stuff is hard, don't implement your own scaling or state management". I was already suspicious after the claim that there are 5 ways to fail with EDA because i've seen at least 10. And the 5 in the talk are for sure valid points, but not the stuff (except scaling) projects typically struggle with. The real challenge is stuff like eventual consistency and the lack of transactions - issues which could have been discussed pretty well with that reference architecture. But hey, now i know that he wanted to buy tailored shirts once... but didn't. ;)

[smaug]

Another pitfall of event driven architecture is that one service derives the whole context from the message it consumes. But sometimes, there is a business need for a service to maintain its own context in addition to what it reads from the topics. I know this is not the way to go for purely event driven services, but sometimes you are forced into a situation where you have to do this.

[cxmais]

do things right, don’t do things wrong

[sgqkwyp]

Given the event log is the system of record, and lets say we want the customer's address updated in the user service. Does that mean we have to fetch the current version of the user from the event log? Seems challenging and requires specific technologies. Alternatively you'll use the DB in the user service. By doing that, the DB has effectively become the system of record, right?

[RaviChandraEnaganti]

Isn't storing/maintaining user data within the Shipping service an anti pattern? Each micro service is supposed to handle one domain?

[debugin]

How do you handle data security in this scenario where a one business unit should not be able to see / consume the data of another business unit in a company? Some users might be able to see qty/price in their plant, other users maybe qty only.. how does this work in these events?

[shashanksharma]

This is how to make complex things Simple enough.

[h_maina]

This should've been a 15 mins talk max

[BosonCollider]

Just a note that Something can be logically event-driven while being a monolith from a devops perspective.

For example, many programs in Go, Elixir, or using Streams in Python anyio/trio are event/message queue driven. They just happen all to live in the same runtime and don't require Kafka to act as a message bus. Imho using language native multitasking in this way is a good way to build a system that can be refactored into distributed systems.

And especially for the case of Elixir, the BEAM is just so reliable that you will probably not need Kubernetes or anything fancy to keep it going. Just put it in a freebsd or Rocky/RHEL VM with a managed database and you will have very good uptime.

[gerritlikestohike]

A nice talk but I have to admit the takeaways are very little here. Real pitfalls which you need to takle when writing a real QRS or DDD event drive architecture have not really been adressed. So if any1 is looking for answers to those you don`t have to watch this.