Trials and Tribulations of Incremental Models (w/ Vincey Au)

Incremental models are core to dbt, and are a key solution to how data teams balance the need for computationally efficient and timely data. This talk will discuss some challenges with building incremental models at scale, and how our approach has evolved over time in a high-growth company with over 30 third-party sources of data.

00:00 Introduction
05:23 More data, more problems
09:33 Figure out the lookback window
17:00 Pros and cons
18:51 Use freshness checks
21:59 Incremental materialization
25:23 When to use these strategies

[gabbaydot]

Great video! very very informative. we just trying to implement an incremental model in our company and we are trying to find the best solution for doing it. Thank you!

[shawn-jung]

It was a fun presentation. Thanks, Vincey.

[thghtfl]

how do you handle duplicated data which was inserted with insert materialization?