Azure Event Hubs for Apache Kafka | Azure Friday

Shubha Vijayasarathy joins Scott Hanselman to discuss Azure Event Hubs, which makes data ingestion simple, secure, and scalable. As a distributed streaming platform, Event Hubs enables you to stream your data from any source—storing and processing millions of events per second— so you can build dynamic data pipelines and respond to business challenges in real time.

With Azure Event Hubs for Apache Kafka, we're bringing together two powerful distributed streaming platforms, so you can access the breadth of Kafka ecosystem applications without having to manage servers or networks. Event Hubs for Kafka provides a Kafka endpoint so that any Kafka client running Kafka 1.0 or newer protocols can publish/subscribe events to/from Event Hubs with a simple configuration change.
[06:45] Demo Start

Azure Event Hubs docs

Azure Event Hubs samples (GitHub)

Azure Event Hubs overview

Azure Event Hubs pricing

Announcing the general availability of Azure Event Hubs for Apache Kafka® (blog post)

Create a free account (Azure)

[rohigt]

Awesome presentation. Precise and cool interoperability of AMQP with Kafka👏

[VOGTLANDOUTDOORS]

A very succinct comparison of these 2 easily-confused distributed streaming/messaging systems. Your example with AzureEventHub using a "Namespace" to provide a "Kafka Abstracion Layer" to Java-coded Producers & Consumers was PRECISELY the explanation I was seeking for explaining AzureEventHubs to a customer! You just made my day!

[lorapora]

The big question is it as good as kafka as far as performance? i am not sure folks who use kafka today will switch and give away all the producer and consumer configuration fine tuning options and replace it with a magic box in the cloud.

[berndeckenfels]

What’s the persistence properties of Eventhub, can I also store like 60 days and replay form there? How would a consumer manage its offsets?

[sdbhattacharya]

Why Microsoft is not positioning eventhub for service to service communication? Why its always depicted as event streaming?

[vchandm]

@4:32 that sarcastic smile from Scott, says it all. Just scales smile (with cost :))

[thebachelorpad]

Such an amazing discussion! love it! Thank you for sharing!

[mykhailokozlov]

That's incredible! Although funny to see java in demo when we have C# :)

[ArvindNaikSOS]

Do we absolutely need Event Hub to consume this Kafka data? Or can we connect to Kafka using the Azure Data Factory or Databricks?

[harishreddy]

Can we read data from blob storage using Kafka?

[abhiramikaushikaaduru]

Good one and nice explanation on kafka data streaming platforms

[snehanjanaray]

Hi, I am trying same implementation in my Maven Spring Boot + Kafka application for Producer after sending data it is hitting Azure Matrix but mentioned error I am getting,


Error:


2019-07-04 12:27:31.729 INFO 92607 --- [nio-8050-exec-2] : sending message='t'
2019-07-04 12:28:31.730 ERROR 92607 --- [nio-8050-exec-2] : Exception thrown when sending a message with key='null' and payload='Hi' to topic testhub:


Failed to update metadata after 60000 ms.




And for consumer while clean build I am getting below mentioned error:
2019-07-04 12:48:26.837 WARN 94752 --- [ main] : Exception encountered during context initialization - cancelling refresh attempt: Failed to start bean nested exception is Timeout expired while fetching topic metadata

[kumarmanish]

2:41 did she mean when the CONSUMERS r ready ?