Kafka Streams in Scala - a Comprehensive Tutorial

Written form:

In this video, we'll learn Kafka Streams in Scala, from scratch. We're going to go through all of its most important features on a realistic scenario:

- how to set up a Kafka stream in Scala
- how to transform data
- how to join semi-static data with KTables
- how to compose Kafka streams together
- how to start a complete Kafka Streams application

Follow Rock the JVM on:

-------------------------------------------------------------------------
-------------------------------------------------------------------------

[XArticSpartanX]

Absolute best programming channel I've found

[camilongii]

Excellent!, maybe a part 2 with Avro <3 would be perfect.

[sarfarazhussain]

Kafka Streams using Scala looks so clean and concise to me, at the same time easy to understand compared to Java syntax.
Daniel, when can we expect a full course on "Kafka Streams using Scala"?

[janga]

First things first: Thank You!
Feels like self punishment - I am not very experienced in Scala at all, but I am trying to master KStreams by using Scala. Your tutorial helped a lot with that challenge!

Just for give you something back:
I had massive problems with SLF4J. This became very important when I produced broken code (which I did most of the time).
Would be a good side-note for your blog, too add the dependency "org.slf4j" also to your sbt file.
Otherwise some users (like I did) will run into issue like `SLF4J: Defaulting to no-operation (NOP) logger implementation`

It might be obvious for more experienced Java / Scala developers than I am. But that hint would have save me some pain

[mbesida]

This lib is only applicable when there is a need to transform data inside Kafka? I mean if you need to enrich the data from some other system(via http call for example) can kafka-streams be used in this scenario? Also how conceptually it differs from ksqldb from confluent? Thanks.

[cgmds]

Hey Daniel, thanks for the article! When I try producing messages for discount-profiles-by-user I got this error: "WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : do you know how to fix it?

[laiboonh]

Q: won't KTable run out of memory since its statically saved in broker?

[timofeygolovach]

Great video. Thnx! Is there ways to mix in messages of different types in the same topic? How will "Serde"s look like? 1 topic per 1 type does not sound good. Thnx

[zone]

maybe kind of a wrong question to ask here, but if i use the confluent stack to set up a kafka cluster, it seems that i need to pay much much money. As far as i understand people mostly use the confluent kafka docker images, but it seems they only allow one node to be free without costing money? i would like to use kafka, but maybe my company is too small and has not enough money and resources for it. I mean, whats the point in using kafka if you only have one node, and the thing is therefore not really failure tolerant for a production system. Guess i stay with good old shared databases then to just safe and read everything from all microservices lol i hate reading through complicated pricing models as a developer. Can is set up apache kafka docker containers without confluent, or is that too much work?

[victorpre]

hey Daniel, there is one typo in the blogpost on the last command: kafka-console-consumer.sh is actually trying to read from the wrong topic. In the code you write to "payed-orders" and on the kafka-consome-consumer you try to read from "paid-orders". all the best, cheers

[zone]

is kafka streams or alpakka to recommend? is one of them outdated?

[conormcnuggets]

getting this error `There was an error converting the message profile1, Left(io.circe.ParsingFailure: expected json value got 'profil...'`