Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or Frenemies?

This session discusses how to build an event-driven streaming platform leveraging Apache Kafka’s open source messaging, integration and streaming capabilities.
Learn the differences between an event-driven streaming platform and middleware like Message Queues (MQ), Extract-Transform-Load Tools (ETL) and Enterprise Service Bus (ESB) – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

Key takeaways for the audience
• See typical middleware requirements and challenges in an enterprise architecture regarding messaging, integration and stream processing
• Learn about the components of open source project Apache Kafka as event streaming platform, including messaging, integration, stream processing, and community add-ons like pre-built connectors and the Schema Registry
• Learn the differences between a native event-driven streaming platform leveraging Apache Kafka and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

More information:

[abobakrnasr]

Excellant video Kai, very informative and you explained it in a very nice way. The presentation just put the right context for me to understand the new era technologies. Thank you so much and waiting for more videos

[santiagopouget]

Dear Kai, my congratulation about this material, i was a really good base for non technical people like. Good and great job (accurate, realistic and technical based). Thank you

[shravanparepally]

If I'm clearing my interview at big 4, consider my big thank you already :-) this presentation is an excellent summary

[sitaluk]

This is an excellent presentation, putting this whole thing like a story. Amazing articulation 👏 thank you very much 😊

[sukumard]

Thanks for this Kai, very very useful to explain to a 175 year old company why they need to change their ways of working to enter the new Digital way of doing things.

[elodiechaumet-doucet]

Super! Got a middleware past and need to understand the concept of Events integration and how it could be frenemy with existing middleware.

[philosopher]

So long but worth watching it, thank you ka

[JoaquinPonte]

This video is amazing, you presented everything in a clear way, congratulations

[JohnTubeK]

Nice video. This is what I come across as an EA at my job, there is already a sunk cost in legacy technology and it’s a struggle to get business to update their technology unless there is a true business or technical need.

[ammarhassan]

Indeed very informative, especially the conclusion the right tool for right job, because this is where Architects and companies can't decide well and try to do every job in same tooling and ends up in having very high maintenance code/projects, and sometimes integration layer become bottle neck issues and stop the business from achieving agility.

[tomw]

Nice explanation with a good example of the Host-unloading. But this is "only" the technical side. I assume that the most complex part will be the business definiton of "what is an event", "what data is needed for other systems to process an event", "how do you ensure a certain order of event processing through systems that listen to the event stream - first system A, then System C, then B"? It's not done putting some database changes in the event stream. No one on the other side can do anything without more process context of that changes.

[marsimark]

A/B testing for middleware services is a new idea to me

[welbsantos]

Excellent video !!! It was very helpful!

[KarstenHeymann]

Just a very small note from a fellow german: "Event" is pronounced "Iwänt" with an emphasis on the "ä".

[jksharma]

Thank you for a wonderful knowledge Sir

[srik]

Excellent video.. One question. So you recommend kafka only for read ? what about write ? can we write into KAfka and which intrun updates the mainframe ?

[ComisarioLobo]

Nice video Kai, how does Kafka compare to Apache Pulsar and Nats? Also when Kafka is gonna be fully cloud native?

[TheGeoDaddy]

My basic question is - what is in Kafka - that you can’t “code your own” if you already have MQ implemented on the IBM mainframe and on Websphere across a distributed platform? Yes, Kafka is bullet tested and provided a higher level user interface/functionality. But there’s something to be said about keeping knowledge of something so - application dependent - in-house...

[ashuetrx]

What is the rate that the traditional MQs can handle & Apache Kafka ?
I understand it will also depend on the consumer rate but lets say take an average consumer .
Rate of Kafka answered @ 18:08

[mitenmehta]

whole agenda was described but not really clear even till end how existing middleware can be replaced easily with kafka without any loss of features. also its pull only so if existing clients are using push based then how to replace easily ?