How To Build A Modern Data Catalog - Mars Lan (Metaphor Data, DataHub, prev LinkedIn)

Mars Lan joined the Utah Data Engineering Meetup to show how to build a modern data catalog. He dives into the past and present of data catalogs, and why the approach taken at Metaphor Data (and LinkedIn's Datahub) stands alone.
-----------------------------------------------
About the talk

Recently, there has been a lot of hype around the data catalog. It seems like almost every tech company has built its own data catalog to solve the search and discovery problem faced by their data scientists and AI practitioners. Many data tools have also started offering some form of builtin data catalog. Even all the major cloud vendors provide data catalog as a service.

On the other hand, data catalogs have existed for a long time. Why is there a sudden renewed interest in this area? How is the modern data catalog different from its predecessors? When do you know you need a data catalog? What’s the right way of building a modern data catalog?

In this session, we’ll attempt to answer these questions and share our journey of building LinkedIn’s modern data catalog, DataHub. We’ll also talk about the lessons learnt as well as other problems modern data catalogs can solve, including data privacy, data governance, and lifecycle management.

Related video:

#datacatalog #metaphordata #datascience #dataengineering #datahub #fundamentalsofdataengineering
-----------------------------------------------
About Mars Lan

Mars is the co-founder & CTO of Metaphor Data, a startup that aims to make the world’s data more actionable. Previously, Mars worked at LinkedIn as the technical lead of the metadata team and a software engineer at Google.

He is also the co-creator of DataHub, an open source search & discovery tool that originated from LinkedIn. Mars received his PhD in Computer Science from the University of California, Los Angeles.