Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda

Kafka is a top-notch industry platform for streaming data processing at scale. No surprise that first-class citizens of Kafka world are 24/7-running producer/consumer applications (e.g. classical servers, k8s-pods, etc.).
But what about the rapidly rising world of AWS Serverless ecosystem?

In this video ,let’s explore how to publish messages in MSK Kafka Topic from Serverless computing platform, AWS Lambda.

Prerequisites:
----------------------
Learn AWS VPC , Subnet , Route Table, Internet Gateway with Hands-on Demo
VPC with Public and Private Subnets in-depth intuition with Hands On
AWS VPC NAT Gateway In-depth intuition
AWS VPC ENDPOINT in depth intuition & hands on
Using Amazon VPC Endpoints to Access DynamoDB in-depth
Getting Started with AWS Managed Streaming for Kafka with in-depth service setup

Steps for the Lab:
-------------------------------

You can download the Lambda Layer zip from here :
------------------------------------------------------------------------------------------------

Check this playlist for more Data Engineering related videos:

Apache Kafka form scratch

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

[ATHARVA]

hi @knowledge amplifier do you have any video related to testing in data engineering? if not could you suggest some resources

[rushikeshkarkhanis]

Great tutorial, you explained with Security group configuration. Most of the videos on youtube have public access, which can be vulnerable.
Thanks for sharing knowledge.

[NatureAndMyself]

I am not sure why it is coming "errorMessage": "Unable to import module 'lambda_function': No module named 'kafka'" ?
I have tried with S3 and direct uploading .zip file to lambda layer

[abhishekdubey-pn]

you r running loop for Provisioned in lambda can i do with serverless cluster for same i have tried by adding client property's and all but facing some issues (Address family not supported by protocol) and some more pls help me out on this

And special thanks for this great video

[digitalnomadprabhu]

i creted msk cluster and how to use it in nodejs

[vipinsharma-jtqt]

Nice tutorial, it would be really helpfull if you can create some tutorial on mutilple zookeeper and kafka broker using docker compose with scheme registy with avro in Python. Done research online but didn't find any solution, If you can do that reallly helpfull. keep creating wonderfull content @KnowledgeAmplifier1.

[KhalidKhan-krit]

Nice video..I tried but getting error... "errorMessage": "NoBrokersAvailable",
"errorType": "NoBrokersAvailable", when i run the producer code from cmd it works fine but not with lambda function