Asynchronous Processing with Go using Kafka and MongoDB

In this example, I developed 2 microservices using Go to demonstrate asynchronous processing. I also integrated Kafka as the messaging broker and MongoDB as the data store.

Microservice 1 - is a REST microservice which receives data from a /POST http call to it. After receiving the request, it retrieves the data from the http request and saves it to Kafka. After saving, it responds to the caller with the same data sent via /POST

Microservice 2 - is a microservice which subscribes to a topic in Kafka where Microservice 1 saves the data. Once a message is consumed by the microservice, it then saves the data to MongoDB.

[vickymarsaa]

Awesome .. worked like a charm ... Thanks a lot ... !!!!

[kavyanshpandey]

I'm facing error inconsistent vendorring ...
Tried go mod tidy
And go mod vendor.
Still not getting solution

[janetech]

How many concurrent requests can this handle ?, and is it bottleneck in mongodb when too many documents are written at the same time ?

[prabhu]

What is Kafka consumer message consumption rate? How fast it reads from kafka and dumps data to db?

[ShivamSharma]

What was the significance of using Kafka in this?
I am asking because, I can do the same thing with just 1 microservice.

[zulhilmizainudin]

Does it mean the message will be lost if the consumer is not running/dead?

[Oswee]

I believe you should store events into Kafka in binary format? Tnx a lot for sharing this!
BTW - setup on Windows is nightmare!!! And why you are using for loop in your consumer to read a message?

[nickernara]

good but can you make it lil active, we feel like you are not interested or doing it by force. presentation is squeeing our energies.