filmov
tv
Stream Ingestion with Kafka & Kinesis
Показать описание
In this presentation, you will learn:
1. Ingest data from Apache Kafka & Amazon Kinesis (index and configure Kafka source)
2. Index and configure Kafka & Kinesis source
3. Create and populate Kafka & Kinesis Source
4. Execute search and aggregation queries to expore ingested data
5. Quickwit's internals: Indexing Pipeline, Metastore API, Support for Distributed Queues, Scaling Indexer, and more.
_____
🕰️ TIMECODE :
0:00 Introduction
1:06 Indexing a file with Quickwit
2:15 Indexing pipeline
5:26 Source trait
6:10 DocBatch struct
6:41 Checkpoint API
8:17 Checkpoint Delta
11:47 Metastore API
12:29 Distributed queues
13:21 Distributed checkpoint
14:03 Kinesis source implementation
17:52 Scaling the indexer
20:39 Distributed indexing
21:15 Demo time
31:45 Q&A time
_____
✏️ Mentioned tutorials:
💬 Join our community :
Topics: indexing, ingesting, data, amazon kinesis, apache kafka, kafka topic, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, indexes, index
#softwaredeveloper #programming #developer
1. Ingest data from Apache Kafka & Amazon Kinesis (index and configure Kafka source)
2. Index and configure Kafka & Kinesis source
3. Create and populate Kafka & Kinesis Source
4. Execute search and aggregation queries to expore ingested data
5. Quickwit's internals: Indexing Pipeline, Metastore API, Support for Distributed Queues, Scaling Indexer, and more.
_____
🕰️ TIMECODE :
0:00 Introduction
1:06 Indexing a file with Quickwit
2:15 Indexing pipeline
5:26 Source trait
6:10 DocBatch struct
6:41 Checkpoint API
8:17 Checkpoint Delta
11:47 Metastore API
12:29 Distributed queues
13:21 Distributed checkpoint
14:03 Kinesis source implementation
17:52 Scaling the indexer
20:39 Distributed indexing
21:15 Demo time
31:45 Q&A time
_____
✏️ Mentioned tutorials:
💬 Join our community :
Topics: indexing, ingesting, data, amazon kinesis, apache kafka, kafka topic, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, indexes, index
#softwaredeveloper #programming #developer
0:35:59
Stream Ingestion with Kafka & Kinesis
0:10:35
Challenges Using Apache Kafka | Data Streaming Systems
0:02:35
Kafka in 100 Seconds
0:06:48
Apache Kafka in 6 minutes
0:29:54
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
0:32:19
Streaming Graph Data with Kafka
0:52:24
Building Data Streaming Platform with Kafka
0:04:23
Streaming data into Apache Kafka with kcat and jq
0:25:44
Set up Streaming Data Pipeline to ingest Kafka Data To Snowflake with a Python Produce from Scratch
0:00:40
Kafka Streams to the rescue! ft. Anna McDonald of Confluent #podcast #shorts #kafka #streaming
0:21:04
Machine Learning with Apache Kafka without another Data Lake
1:04:00
📈 Stock Market Real-Time Data Analysis Using Kafka | End-To-End Data Engineering Project
0:04:17
Real-Time Twitter Data Ingestion With Kafka & Spark
0:07:20
Data Pipelines: Using CDC to Ingest Data into Kafka
0:28:39
Streaming Data into Delta Lake with Rust and Kafka
0:46:20
Reliable Streaming Ingestion from Kafka to Iceberg Tables (Amazon Web Services) | Current 24
0:18:07
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsight : Build 2018
0:00:49
Kafka Streams + interactive queries = bffs #podcast #shorts #kafka #streams #queries #tech
0:11:38
Streaming Data Pipelines Demo - Data Ingestion using Kafka Connect
0:00:38
Live Data Streaming with 7 Million Records With #Sockets, #apachespark, #kafka and #elasticsearch
0:07:37
Snowflake Snowpipe | Snowpipe Streaming | Snowflake Connector for Kafka | Examples, Setup, Demo
0:49:32
GCP for Apache Kafka Users: Stream Ingestion and Processing (Cloud Next '19)
0:58:38
Tech Talk | Diving into Delta-rs: kafka-delta-ingest
0:01:40
Cisco's Real-time Ingestion Architecture with Kafka and Druid
Комментарии