Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Guido Schmutz works for the Oracle Platinum Partner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and Application Integration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and Fast Data solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for Fusion Middleware and SOA and a regular speaker at international conferences.

[joeng]

I would say this is as legendary as the iconic Microservices presentation by Netflix. 6 years has passed, but those were presented here are still as relevant as ever.

[charliehubbard]

From listening to this topic you begin to see how we can't really decide on our choice of architecture. We've done request-response, we've done events, and yet we keep changing vacillating back and forth between the two. Coupled but simple, Decoupled but complex. Kafka's main feature that is different is replaying event streams and it's scale. That is what is driving all of this re-evaluation of architecture design. However, microservices have produced a similar consternation over system design about how these services should interact with each other and how much each service should be responsible for its own destiny (ie a service unto itself, or a collaborative member of a larger system). And, now we're suggesting duplication of storage within the microservices. Each service will now store a copy of all of the data its dependent upon is the advice du jour. At best this is mere duplication of work at worst this has the potential to just recreate many monoliths especially for services that depend upon many things which always happens as product cycles demand. Certainly this can't be good advice for storing user information which has PII concerns and security implications. It doesn't make sense to spread data across many storage tiers that requires extract careful logic and governance unless you're just trying to make your CISO's job harder. Reusing APIs isn't just about data; it's also logic encapsulation! Which if you follow this advice of storing other microservice's means you now have to follow the logic that goes with that data too. Just having the data doesn't absolve you from the responsibilities of using and managing that data. The utopian dream of events everywhere and little to zero request response will be limited by these inconvenient facts. Then there is the issue of microservices need reuse to keep them small. If I can push off logic that isn't central to my problem domain then I can simply reuse someone else's work, and that keeps me light and small. Yes it creates a dependency, but I'm trading off my independence for convenience and a smaller footprint.

There is lots to like about Kafka and event streams, and persistence of streams really is what's new here, but I'd be careful about how much this "changes everything" mentality you take away from this because we've seen this play before.

[splashinventor]

This is best presentation and tech talk I heard in recent days. Surely the presenter is highly experienced and precise, concise in explaining the whole picture.

[boubeniamohamed]

I found that this presentation is definitly one of the best synthesis.
Great work Mr Guido Schmutz.

[Pjblabla]

Excellent presentation that talks about different types of application architecture patterns leading to micro services and then different patterns and concerns for micro services architecture

But everyone should make their own assessment on whether microservices is the right architecture for what they are trying to solve

[imten]

Excellent session. Running through multiple architectures was simply great

[vjntstar]

excellent presentation full of very useful diagrams !

[Oswee]

Really great talk!!! This is something im playing with. I mean last CQRS stuff.

[dineshkhetarpal]

Must watch! Covers all the questions you may have moving a traditional monolithic application to microservices and scale it.

[ujvalsanghavi]

Excellent Talk, nice way to demonstrate how you can evolve legacy systems to use event driven model to make more scalable & available across wide variety of consumer applications.

[nihontravel]

Good discussion and clear explanation of architectural aspect

[igrai]

nice architectural overview + good diagrams, thanks!

[medmed]

Thanks for the presentation. It is really helpful.

[hovgrig]

Thank you for an ​extremely interesting talk!

[andrewshiner]

This was a very nice talk. Thank you.

[USONOFAV]

As a full stack java developer working on Angular+Redux application. I somehow get the whole event-driven architecture thing for microservices. But here you are storing the event instead of the the state.

[BryanChance]

"Bus" and "Driver" topics.. I thought words "Bus" an "Driver" are part of Kafka like a message bus or Kafka stream driver, etc. Got me so confused. LOL Great talk.. ..

[rachelleyu]

Oracle monolithic computing - RDBMS Kafka distributed computing and data stores Long live the king !!

[OutboxThinker]

It was really interesting to watch, well explained the problems and the approaches as well.

[stack.]

Very informative and insightful talk, thank you!