Getting Started: Apache Kafka + .NET

Everything you need to get started playing around with Apache Kafka in .NET. This is a walk through of getting Zookeeper and Kafka running in Docker. Then build a simple console app in .NET that contains a Kafka Producer and Consumer to a topic.

💥 Join this channel to get access to source code & demos!

🔥 Don't have the JOIN button? Support me on Patreon!

0:00 Getting Started: Apache Kafka + .NET
0:21 Setup Zookeeper & Kafka in Docker
4:10 Create a Producer
11:11 Create a Consumer

#kafka #dotnet #eventdrivenarchitecture

[DiegoAlvesPereira]

Very helpful video, imma be using it on my next project and it is very good to see a good example of all the theory I've been consuming.

[christopherkemp]

Very helpful. I tried using the Confluent client Consumer but got stuck for quite a while because it was blocking indefinitely when calling consume. Turns out the Kafka was throwing all sorts of errors in the docker container with an error that the offset topic couldn't be replicated, which makes sense because in this simple example we only have the 1 broker. I added this one option to the docker run command for Kafka and it now works: -e

[tims]

The best instruction video I have searched. Thank you for your sharing a great demo!

[become-software-architect]

Also i tried consumer with Confluent.Kafka nuget package. this also not working. Do we need to explicitly create consumer group in Kafka?

[huferry]

Thanks, after a long search this is the first working hands-on demo I can find.

[reginaldoaguiar]

Fantastic, i'm starting with kafka and your video was helpfull! thak you

[MrSkmishra]

Hi in consumer component in constructor while creating clusterClient , new ConsoleLogger() is not working for me, any idea what i am missing

[madhuradole]

Great video. Could you explain how the configuration settings would be if I'm trying to connect to a remote Kafka client via SASL protocol?

[mamalmaleki]

Fantastic video, thank you. You did a great job man

[timothyasigbey]

How will you handle a distributed transaction something like a routing slip so if one service fails other services prior to the current service will get compensated.

[sunilanthony]

This look really cool for a messaging queue. I have to play around and see how it works.

[Steedie]

Thanks for the help! Everything worked the first time around, but after I've restarted my PC, the program can't connect/find the localhost again. How would I get this running again?

[gouki]

Great video and explanation ! Thanks !

[pedrofreitaslima]

Very good this video, the basic about kafka with .net and C#, learn how create one producer and consumer kafka, i think miss learned about how create publisher and subscribed of the messages using this producer and consumers kafka

[acegame]

Hello first of all thanks for the video. I would like to ask you a question as I am currently working on a project with Apache Pulsar which is similar to Apache Kafka. The setup that I chose to work is by hosting the pulsar cluster on a docker container that gets run by a docker-compose. The problem is that the topics need at least 30 seconds to be deployed as I have also pulsar functions that need to be added to the cluster. How can I setup a consumer listening to a topic if the topic has not been generated yet? This is not the question that I am asking cuz I know this has no answer. My question is what would be the best solution in your opinion for this problem?

[thegenxgamerguy]

Would love to use Kafka in an IoT scenario, but there are a few open questions left.

Maybe you can help me with the most urgent one?

I need to use message queues to persist data in the cloud which is coming from the IoT devices. However, records should only be persisted once.

We use Redis currently to ensure that, but switching over to Kafka is part of our next years plan. Any ideas how we can make sure that only one instance of our "persist records"-microservice gets a record and that the records are not repeated?

[vineethganji]

hey I want to implement multithreaded kafka producer can I get any reference code of this

[raghuindra]

Hi, Ive tried implementing the same way but this errored out..

Getting following exception: SSL Handshake failed.. client authentication might be required <see ssl.key.location and ssl.certificate.location and consult the broker logs for more information>

[yonatandaniyel]

Isn't Kafka Sharp dead now? Everyone I talk to seems to advocate using Confluent Kafka for Producer/Consumer, although most people seem to not like it.

[JasonMogera]

How is the Condukter app set up? Do you have to specify the network that was added in docker run command? I am following the example, but Conduckter is not reading any messages.