An introduction to ksqlDB

You've got streams of data that you want to process and store? You've got events from which you'd like to derive state or build aggregates? And you want to do all of this in a scalable and fault-tolerant manner? It's just as well that Kafka and ksqlDB exist!

This talk will be built around a live demonstration of the concepts and capabilities of ksqlDB. We'll see how you can apply transformations to a stream of events from one Kafka topic to another. We'll use ksqlDB connectors to bring in data from other systems and use this to join and enrich streams—and we'll serve the results up directly to an application, without even needing an external data store.

Attendees will learn:

- How to process streams of events
- The semantics of streams and tables, and of push and pull queries
- How to use the ksqlDB API to get state directly from the materialised store
- What makes ksqlDB elastically scalable and fault-tolerant.

ℹ️This talk was presented at the Kraków Apache Kafka® Meetup, kindly hosted by VirtusLab. My thanks for them for the recording also.

---

--

☁️ Confluent Cloud ☁️

--

Other links that I mentioned during the talk:

For community support and discussion check out:

---

[RaduToev]

Hey, thank you for the presentation. Out of curiosity, does Kafka + ksqlDB kind of replicate what EventStoreDB is doing?
If I understand correctly one could use ksqlDB as a materialized view also, but I'm not sure how the two technologies compare.
Do you see them overlapping, or do you think that they complement each other?
Thanks again

[priyankan]

I have posted it in a confluent forum, but we dint get any response in it. So we posted it here am trying to consume a row from a stream using select query, where as the row has certain properties along with the inserted time as a column. Now, I need to consume the row after 30 mins gap from its inserted time.

I tried using both tables and streams but I noticed that, the select query is applying the conditions at the time of insertion itself and as the condition does not matches at the insertion time, it is not getting consumed. But, my expectation is that, that inserted row has to be consumed after 30 minutes from the isertion time. Could any one suggest me, if it is possible to format a query for this scenario?

The select query for the stream/table that I am trying to use for the same:

insert into my_stream values (1, ‘some data’, ‘some message’, 1625745622700);
//1625745622700 - timestamp in milliseconds at the time of insertion.

select * from my_stream where rowtime > inserted_time_ms + emit changes;

[t]

Great presentation 👍you Menton joining streams with streams. Is it posible to "merge" two or more streams together based on å common key or field? Lets say i want to all events from a given person from several streams and still keep the "order" and timestamps. In short the example greag young uses for his get event store and temporal queries? I want to se all messages from bill inn all chatrooms and handle those events as one stream?

[doganaysahin]

Hi Robin,

It was a great presentation. Thank you. I have a question about table grouping with window. I group the values ​​for who send requested which at least 3 times in 5 minutes, then insert 3 value. Then consumed in kstream from the topic which is tables send. I expect just one value when i insert 3 person which is the same but my consumer got 3 events. Is it a bug? Or can i consume one event after i insert 3 same person in 5 minutes.

[vinitsunita]

We are exploring ksqldb for our usecase and we are planning to create stream on the fly. These streams will be wrapper over simple select query with where caluse. The number of streams could go to around 10k. Is it advisable to create 10k streams in KSQL DB and Can it cause performance issues ?

[mehrdad_masoumi]

hi Robin Moffat, Thank you for your answers in the stackoverflow. I use MySQL and debezuim to create Kafka topic. When I create a stream in Ksqldb, the data is returned as null, pleaze help me

[HelenWilliams-qv]

Hernandez Donna Harris Amy Garcia Eric