IMPORTANT: Watch this before you use APIs and Message Queues!

You want to build streaming data pipelines using APIs and Message Queues? Check out these design tricks. I'll also show you an important drawback using queues that are going to "break" your http request. Once your data is in the Queue the API can no longer control what is going on with the transmission. The API can only return 200 or 201 to the client.
Sounds complicated? It's actually quite easy. I'll explain in the video.

►Learn Data Engineering with my Data Engineering Academy:

►LEARN MORE ABOUT DATA ENGINEERING:
-Check out my free 100+ page data engineering cookbook on GitHub:

►PLEASE SUPPORT WHAT YOU LIKE:
Just use this link whenever you buy something from Amazon:

(As an Amazon Associate I earn from qualifying purchases from Amazon)
This is free of charge for you but super helpful for supporting this channel.

#DATAENGINEERING #DATAENGINEER #APACHEKAFKA

[andreaskayy]

What's the biggest fail you did working with APIs or Message Queues?
Mine was definetly when I sent in too big messages (over the max limit) and was wondering for hours why nothing was in the queue :D
Yes, Kafka doesn't return an error in this case. It just drops it

[ruixue]

3:11 problem with using the normal approach
3:24 example case
5:09 2nd option - *use message queue*

[thaboshikwambane]

So you took a lecture with 4 people and made it a public video? Super smart lol
Killing two stones with 1 bird 👏🏽

[xA]

As part of a current project, source data is requested from an API, the relevant data is extracted during the scheduled batch process before transforming the extracted data which is then sent to a message queue for subsequent ingression into a database. To make this data available to consumers an API will be built to replace the current "database dump" style of API. Learning about message queues has been a very interesting aspect of this project.