Batch Data Processing with AWS Kinesis Firehose and S3 | Overview

In this video, I go over AWS Kinesis Firehose and how it is useful to batch data and deliver it to other destinations.

📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚

🎙 MY RECORDING EQUIPMENT 🎙

💻 MY DESKTOP EQUIPMENT 💻

📚 References:

☁Topics covered include:
- Kinesis Firehose
- Kinesis Firehose Example
- Buffer Size
- Buffer Interval
- Best Practices
- Gotchas

🌎 Find me here:

#AWS
#Serverless
#Kinesis
#Lambda

[raphael-okere]

Thank you so much for all the content you provide. You really help people become better devs

[DarkApplesTasteBest]

Now, I finally understand why it is called Firehose, if you want to kill a fire and you walk up and down with a bucket(single event), it's not efficient and will cost you extra energy. With a firehose you direct large quantities of water (events) to the fire(database).

[kino-xwxg]

thank you so much for your videos, your channel has become my go-to channel for anything aws related!

[TomerBenDavid]

Best practice section is a great added value! Thanks!!

[SimoneIovane]

Clear explanation of a business case. Thank you

[henryeleonu]

Great! This is straight to the point, and I like that you explain these AWS services with use cases. thanks for your content.

[TivoKenevil]

You're awesome!! Keep up the great content

[krishnasanagavarapu]

Hello, love your content . Can you please create hands-on for the 2nd case where data is transformed using lambda before sending it to s3

[pk_]

Thank You for this very good overview 🤝👍


Can you please tell me, how good the job opportunities for AWS kinesis?

[asdfasdfasdfasdf]

Good video. Please put the AWS Service name at the same time as its logo. Must of us don't remember the name each each logo from AWS Services put the name at the same time

[ralfrolfen]

Question:
From the explanation it sounds like Kinesis is doing nothing else but grouping message together. Is this really everything there is to Kinesis?

It also seems like these kind of use cases could be accomplished with other services as well. E.g. store every transaction in a dynamo DB and read from there with a lambda function.
I don't see the "wow, finally we have this service"-Effect.

[abhishekkempanna]

thanks! One question: How can we send data with source as Azure Function(From Microsoft Azure) to AWS Kinesis firehose?

[TheLostBijou]

Is it streaming processing or Batch Data processing?

[professional]

Can we connect Transactions directly to Firehose or do we need a lambda in between ?

[warrenb]

can we setup to invoke Lambda from the Transactions directly without going through SNS? and can we setup to load the Transactions data directly into Firehose? Thanks

[galeop]

so, as a Lambda function is needed to forward the message from SNS to Kinesis Firehose, what is the point of the latter ? Just to aggregate messages based on time/size criterion?

[ray]

It's possible to use kinesis as real-time computing framework like flink

[raghukumar]

Hi, can you let me know whether it is possible to move data from S3 bucket -> Kinesis -> Athena. for example Whenever data is added in bucket, Kinesis service should pick the data and move to Athena.

[reagan]

What if this processing will take longer than 15min, is it possible to use AWS Batch in the middle instead of a Lambda?

[john-danson]

So 1gb [me] @ $0.50 on bare metal versus $3.00 per gb for kinesis. 6 times less expensive.
Does anyone need a secure in-transit feed into s3 over direct connect at 1/4 times the cost of Kinesis?