Create a Kafka Cluster Using AWS MSK And Stream Data - Full Coding Demo

In this lesson we create a Kafka cluster using AWS Managed Streaming For Kafka (MSK) and stream data to that cluster from an EC2. We use the AWS 'Getting Started' guide for MSK as our reference point as we create a Kafka streaming solution utilising the Managed Streaming Kakfa service. The end result produces a solution on which we can send a typed message on an AWS EC2 onto to the stream, which in turn is polled by a consumer using a topic so the message is rendered back to the user in real time.

😎 About me
I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies. My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server.

[theamithsingh]

great tutorial mate, would love to see more 'hands-on' tutorials like this on youtube.

[lilax]

I keep getting timeout errors:" WARN Client session timed out, have not heard from server in 10027ms for sessionid 0x0 when I try to create the topic. Is there a certain permission I need to grant on either the MSK service or the EC2 instance I'm currently running? Both are in the same VPC. TIA!

[Quivun]

Thanks! These demos help learning immensely!

[nikhilmittal]

Hi guys, getting below error when trying to write message on the kafka topic. Any help would be appreciated.

ERROR Error when sending message to topic AWSKafkaTutorialTopic with key: null, value: 14 bytes with error:
Topic AWSKafkaTutorialTopic not present in metadata after 60000 ms.

[Resolved] Its working for me now, had to change few security settings,
"access control method"= unauthenticated access
Encryption= TLS encryption

[matheusmandotti]

Just incredible, congratulations !! Great video

[Chavoosi]

Great tutorial mate. Thanks for your efforts

[PoojaGupta-bikw]

how can i connect it with springboot producer app?

[SacrificialGoat]

Instead of opening kafka up to everything or using the public ip you can just specify the security group of the ec2 instance ?

[johnsaviello]

I am getting the following error when I go to create the topic " Exception in thread "main" Timed out waiting for connection while in state: CONNECTING". Any way to resolve this?

[nozzom]

Hi Johnny, I am not getting the kafka bootstrap brookers - aws kafka get-bootstrap-brokers --region ap-southeast-2 --cluster-arn For some reason I do not get any response. When I run the describe kafka command I do get the full response with Zookeeper id and cluster details.

[Explore_with_Azizul-maqsud]

Awesome & Very nice content... but, unfortunately, the lowest sound ever in the youtube videos I experienced here..?? Please record with loud speaking. Thank YOU

[nainaarabha]

I have one doubt. Can we host multiple kafka producers in one ec2 instance?

[arrjay]

Looks like the path in the wget command to download Kafka is now invalid. Can you share the latest command to download Kafka? Thanks for supporting the community Johnny!

[prithviboinpally]

Great tut. Audio is really quiet though

[nishokganesan]

Can we connect this EC-2 instance from out local and send messages to topic?

[samjoshuaberchmans]

Do you have any tutorials for Kafka/Spark streaming data ? Would like to view them.
EMR will be having Spark, and we should be running spark from there. Consumer/Producer will be available in EC2.
Would like to start a HBASE EMR to insert the values final into the HBASE tables.
If you have any sample Videos for the same post it.

[yovelb]

hey man great video!
im trying to run the "aws kafa describe-cluster command but im getting the following error:
An error occurred (NotFoundException) when calling the DescribeCluster operation: The requested resource doesn’t exist.
any chance for help?

[AkashGupta-sypi]

Hi, I am getting following issue
Messages are rejected since there are fewer in-sync replicas than required"
Can u please inform me what might be the case

[sridharshan]

we can do it in private to public
like bastion host

[ravindharanshanmugam]

Thanks for the nice demo. one question from my end, how to stream data from filebeat to AWS MSK then to logstash?
FileBeat => AWS MSK => Logstash
(both file beat & logstash is in same vpc as of AWS MSK)
I tried to connect as usual but hard luck. do we need to do any special config change?