Real-Time Streaming Project with Smartphone Sensor Data , Kafka , Rest API & Python

preview_player
Показать описание
If you are looking for free real-time or streaming data sources , then you are in right place!
This video explains how to stream data from sensors on your phones using an app, Sensor Logger & ingest them in Kafka Cluster for Processing.

Install Sensor Logger from the below link:

Prerequisite:
-------------------------
Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda

Code:
--------------

Learn Apache Kafka form scratch

Check this playlist for more Data Engineering related videos:

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL
  1. Knowledge Amplifier
  2. Handling streaming data from a mobile app
  3. A Real-Time Streaming Project with Smartphone Data
  4. Real-Time Sensor Data Processing with the Kafka
  5. Real-time IoT Data Streaming from Old Smartphone
  6. free real-time or streaming data sources
Рекомендации по теме
Real-Time Streaming Project 0:24:23

Real-Time Streaming Project with Smartphone Sensor Data , Kafka , Rest API & Python

The Best Way 0:07:39

The Best Way to Multicam Live Stream Wirelessly

Realtime Socket Streaming 1:42:25

Realtime Socket Streaming with Apache Spark | End to End Data Engineering Project

Streaming Smartphone Data 0:00:14

Streaming Smartphone Data with FastAPI, Kafka, ksqlDB, and Docker

Streaming Smartphone Data 0:00:54

Streaming Smartphone Data with FastAPI, Kafka, QuestDB, and Docker

Real-Time Volumetric Capture 0:00:55

Real-Time Volumetric Capture and Live Streaming to Web Browser on Smartphone.

Building a Real-Time 1:25:00

Building a Real-Time Data Streaming Pipeline using Apache Kafka, Flink and Postgres

Data Streaming in 0:03:39

Data Streaming in Real Life: Banking - Real-Time Data Processing in the Financial Services Industry

Streaming LIVE: Aurora 1:48:35

Streaming LIVE: Aurora Special Finance Committee Meeting | 10-25-2024

Real-Time Data Streaming, 0:10:04

Real-Time Data Streaming, Processing with Kafka, Spark & Debezium : Event-Driven Architecture

Build a Reactive 0:53:46

Build a Reactive Data Streaming App with Python and Apache Kafka | Coding In Motion

When you first 0:00:32

When you first time install Kali linux for hacking 😄😄 #hacker #shorts

Real-Time Shake Detection 0:22:20

Real-Time Shake Detection app using Accelerometer Sensor Data, Kafka & Python

Cybersecurity Expert Demonstrates 0:03:27

Cybersecurity Expert Demonstrates How Hackers Easily Gain Access To Sensitive Information

Fast Food App 0:00:24

Fast Food App in React Native 🔥 #shorts #reactnative #expo #reactjs #app #ui

Building a Real-Time 0:07:19

Building a Real-Time Data Pipeline with PySpark, Kafka, and Redshift | By Darshil Parmar

drone #shorts 0:00:25

drone #shorts

Create Powerful Broadcast 0:01:32

Create Powerful Broadcast Graphics with Live Book GFX

Blynk Live Video 0:06:58

Blynk Live Video Streaming using ESP32 cam/ESP eye

[OBS Ninja] Your 0:05:44

[OBS Ninja] Your phone as webcam? FREE, NO APP, IOS & ANDROID.

Arduino project 😎^ 0:00:40

Arduino project 😎^ Arduino #arduino #2022 #2021 #2023 #dc #arduinoproject #diy #foryou

Building Real-Time Data 0:31:30

Building Real-Time Data Pipelines with Kafka , Faust & Snowflake

smartphone video streaming 0:00:11

smartphone video streaming screen | My Telegram ID:- @mxnmdtanvirahmmed 4K watch time on YouTube

How to view 0:02:56

How to view our local host website on mobile phone | web development tutorial

Комментарии
Автор

Amazing tutorial & demonstration. The key issue with other tutorials is that they focus telling the theory part or at max with dummy data. You have overcome that with this tutorial so that we can see real time data generation and consumption. Thank you very much for all your efforts that you put in !!

gauravsrivastava
Автор

Very informative. Can you make a next video on continuenig this video as how to:

1) Process
2) Query
3) Visualize

This data that we captured. I think you can make a series on it, it is my small suggestion you can add it more.

mr.mechanic
Автор

Thanks for the great lecture!!!

following your steps but using MSK, I have observed multiple lambda timeout cases during streaming, which means some data failed to arrive to kafka. Added some logs for troubleshooting the lambda function, it shows "data send success" but not receive status code 200 from kafka. In addition, it shows that 'Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms'. Do we need to add some code to close producer in lambda function before it is timed out? otherwise there may be some dangling connections still keep open and hold on kafka resources. Thanks!

chenguo
Автор

Awesome bro, learned a lot, thank you 💯

சிவபாலன்வெங்கடேசன்
Автор

Good content!!
But when i tried i am getting below error in the logs:
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'kafka'
Traceback (most recent call last):

i have added the layer as provided by you!!
could you please help here.

girishdepu
Автор

what are the important topics required for aws data engineering?

avinash
Автор

can u make a video about streaming magentic sensor from android

geomarinesolutions
Автор

Please provide big data engineering end to end project with involving all components of big data with explaination, data and code

BhakthiYoutube
Автор

I tried for 2 times and follow exactly same step by step but its not showing the logs into the kafka consumer. What can be the reason???

abrarpinjari