Iceberg's Best Secret Exploring Metadata Tables Szehon Ho

Iceberg's Best Secret Exploring Metadata Tables - Szehon Ho

A presentation from ApacheCon 2022

Iceberg's secret sauce is its rich metadata, powering core features like time travel, query optimizations, and optimistic concurrency handling. But did you know that everyone can easily access this secret sauce via system tables? In this talk, we go over real life queries on metadata tables that get more insights out of Iceberg. What is the last partition updated and when? Why are there too many small files? Why are certain data files filtered out or not? We explore even more advanced use cases like data auditing and data quality. How many null values are being added per hour? What is the latency of data ingest over time? We will also cover metadata table performance tips and tricks, and ongoing improvements in the community. Whether you are already using Iceberg, or interested in getting started, attend this talk to learn how this under-utilized feature to get even more out of Iceberg.