What is Data Observability?

Data observability can help data teams and engineers better understand the health of data in their system and automatically identify, troubleshoot and resolve issues in near real-time.
In this video, Ryan Yackel talks about the difficulty in managing the health and quality of your data sets, what data observability is, and how it can help you fix issues with your data before it impacts your bottom-line.

#databand #observability #DataScience #MachineLearning #Deeplearning #ArtificialIntelligence #BigData

[Flankymanga]

Hello guys. Thank you for this blackboard session. I have a suggestion: Theory is nice and it is the first step to get into the specific IT topic. But as they say: seeing is believing... could you please make a video with practical use case, how data observability practices are being utilized to solve data pipeline issues? In other words: How data observability looks and works in real life on real tools. That would be awesome to see. And if you could merge this topic with Data Governance that would be totally awesome! I am loving these blackboard explanations! Thank you very much!

[dineshshekhawat]

Very insightful, learned a very useful concept on a high level. Keep bringing such videos!

[AISmallBusinessHub]

Thanks for breaking this down clearly. It makes it easy my product designer brain to understand the concepts

[mustufabaig]

What I understand from this video is that data observability is just a grouping of "data quality", "data lineage", "ETL Logging" etc. into one umbrella. These are common concepts within data warehousing/engineering. Some teams solve them with frameworks/vendor tools, some through custom development and some use both approaches. Data observability is just a new name for it.

good presentation though.

[ladonwilliams]

Absolutely great overview. Appreciate the train analogy. I got a good understanding of the premise of Data Observability, plus the bonus of a concise understanding of data engineering and pipelines. Need this gentleman explaining more concepts!

[alexpishvanov]

I really like this idea to shift thinking paradigm from been software engineer to data engineer:)

[billlabranche]

Great illustration to the concept of Data Observability - thanks for making it simple enough to understand.
PS... I'm very impressed with your "backwards writing skills"!!!!

[nikitashrestha]

Hello, thanks for this explanation. I would like to see more videos on NLP domain. To be more specific using pre-trained tranformers for text analytics ( Transfer Learning )

[rajarshisadhya]

Thanks much for the really cool explanation. Wondering whether there is any tool available that provides the capability to really ‘Observe’ any changes/incident in the data pipeline?

[paddyarunachalam]

Thank you. Great session. Could I ask what technology you use to put this together?

[SoniaRaval]

Great overview and love the train analogy

[ARATHI]

Understood in theory..Thanks! Several years ago, one issue we used to run into was some of our Enterprise customers were missing sending data on defined schedule into our SaaS env. Our ETL data ingestion job (sort of data pipeline tool) won't help since it won't be kicked off. So we wrote something outside of the ETL job to ensure we can catch such data misses by the customer...could/should this be part of data observability?

[silicon]

Is Data Observability same as DataOps ?

[viojesus]

we've been moving data for years with ETL tools. Tools that generally don't require a super coder. Some of these tools have some level of monitoring built in, and some have data quality modules. BUT let's go back to hand-writing code to move data. seems like a bad idea.

[davidmurphy]

I think it would have been a better presentation if he had stuck to concrete examples more; like with the betting company.

The train was cute, and I did get it, but he explained something abstract by making it even more abstract. Heading in the other direction and injecting some hard reality would have been more instructive.

At the end of the day, this is not an academic exercise, it's there to solve real world problems.