Episode 15: Why Data Observability Needs Data Criticality | Some Engineering Podcast

In this episode, we interview Kevin Hu, co-founder and CEO at Metaplane. Metaplane offers data observability for the modern data stack. Kevin calls Metaplane the "Datadog for data," in reference to observability for microservices and cloud-native stacks.

As data volume and tool usage grow, so does the potential for something to break—resulting in errors and data downtime. In the modern data stack, the chain of SQL-based transformations between the original data source and the computed result is long and complex. For this reason, it's often nearly impossible to pinpoint the source of data errors.

Metaplane's focus is data criticality, and Metaplane has built instrumentation to understand exactly where errors occur. When data is mission-critical to the business, data teams become "solution-aware."

We take a walk down memory lane in this episode. We discuss the early days of the cloud warehouse market and the paradigm shift to separate storage and compute that, overnight, turned Snowflake into a market leader.

As a result of this shift, the market for analytics expanded and spawned a new generation of data tooling across categories like data integration and ETL, customer data platforms, data catalogs, reverse ETL, and data observability by companies like RudderStack, Airbyte, Census, Hightouch, and, of course, Metaplane.

0:00:00 Introduction
0:02:08 Too early for data observability?
0:10:53 The "here-on-fire" problem for data observability
0:14:28 What does the data observability market look like today?
0:25:51 Three types of attitudes to data observability
0:42:55 Who are Metaplane customers?
0:50:43 Importance of Hippa compliance
0:57:10 Metaplane stack: Fivetran, dbt, Snowflake, Looker
1:10:22 Database engines and query optimization
1:18:49 How will the data observability market look in the next year?