Using Event-Driven Design with Apache Kafka Streaming Applications ft. Bobby Calderwood

In this episode of Streaming Audio, Bobby Calderwood, founder of Evident Systems and creator of oNote observes that at the dawn of the computer age, due to the fact that memory and computing power were expensive, people began to move away from time-and-narrative-oriented record-keeping systems (in the manner of a ship's log or a financial ledger) to systems based on aggregation. Such data-model systems, still dominant today, only retain the current state generated from their inputs, with the inputs themselves going lost. A converse approach to the reductive data-model system is the event-model system, which is enabled by tools like Apache Kafka®, and which effectively saves every bit of activity that the system generates. The event model actually marks a return, in a sense, to the earlier, narrative-like recording methods.

To further illustrate, Bobby uses a chess example to show the distinction between the data model and the event model. In a chess context, the event modeling system would retain each move in the game from beginning to end, such that any moment in the game could be derived by replaying the sequence of moves. Conversely, chess based on the data model would save only the current state of the game, destructively mutating the data structure to reflect it.

The event model maintains an immutable log of all of a system's activity, which means that teams downstream from the transactions team have access to all of the system's data, not just the end transactions, and they can analyze the data as they wish in order to make their own conclusions. Thus there can be several read models over the same body of events. Bobby has found that non-programming stakeholding teams tend to intuitively comprehend the event model better than other data paradigms, given its natural narrative form.

Transitioning from the data model to the event model, however, can be challenging. Bobby’s oNote—event modeling platform aims to help by providing a digital canvas that allows a system to be visually redesigned according to the event model. oNote generates Avro schema based on its models, and also uses Avro to generate runtime code.

EPISODE LINKS

TIMESTAMPS
0:00 - Intro
1:55 - Data Modeling vs Event Modeling
10:15 - Data-thinking mindset
15:44 - From records of states to data model
20:41 - Conway's Law
22:47 - Use case
25:58 - Test-driven development
28:25 - Event modeling is so visual
29:49 - Career experiences and stories
34:12 - Recording data as events
35:38 - Event modeling yields benefits
37:14 - Creating oNote
43:14 - Avro code generation
47:03 - Solving different mindsets in one web tool
49:05 - It's a wrap

ABOUT CONFLUENT

#eventmodeling #streamprocessing #apachekafka #kafka #confluent

[benisrood]

The first thing I was gonna ask is if Bobby was still big into Clojure, and glad to immediately get positive confirmation :D

[benisrood]

Miro is a very poor fit for most tasks like this because it doesn't integrate with any other system. Yes, you can see it in Miro or generate a PDF, but that's about it. Exactly as Bobby says, you need something that links to JIRA or something else that's used in the actual production workflow. Miro is also very slow and lacks the sense of immediacy of working with sticky notes in the physical world, so you lose the energy and creativity of a in-person meeting.