Data Observability with the DataKitchen #DataOps Platform

This video gives an overview of The DataKitchen DataOps Platform's data observability functionality and benefits.

[Resources]

[About the DataKitchen DataOps Platform]
The DataKitchen DataOps Platform is your command center for DataOps. The Platform automates the key functions of your development & production workflows so cross-functional teams can seamlessly collaborate, quickly innovate, & instantly deliver the kind of error-free, on-demand insight that leads to one successful business decision after another.

[Transcript]
Data errors are crippling data teams. Many teams spend more time finding and fixing errors than creating innovative analytics. These costly and embarrassing errors reduce trust in the data team's work.

By following DataOps principles, data teams can deliver error-free, on-demand insight. Errors are reduced to virtually zero through rigorous testing and monitoring of the entire analytic system. The DataKitchen DataOps Platform allows you to easily integrate automated observability into your data pipelines regardless of system complexity, tool preference, or technical skills. Tests check and monitor data in production as well as validate new analytics before deployment.

Here's how it works. The platform meta-orchestrates diverse toolchains across the end-to-end data journey from data access to governance and visualization. This pipeline is called a recipe. To reduce errors users add tests to recipes to monitor data in production or test new analytics. You can add any number of tests at each step in your pipeline. Every processing or transformation step should include tests that check inputs, evaluate results against business logic, and measure outputs against expected results. Every failure encountered is an opportunity to increase test coverage and the robustness of your pipelines.

Over time, the platform makes it easy to add tests. Users create tests in their preferred language without needing to learn anything new. Here, the user-created a location balance test using SQL to populate variables. Location balance tests ensure that data properties match business logic at each stage of processing. In this case, the test ensures that the line count of the derived fact table equals the line count of the raw table. The test logic statement compares the resulting output to control values or historical values. The user can set the failure action for each test. This test stops recipe execution on failure. Alerts can be configured to notify users via email, Slack, Jira, or other popular collaboration tools.

Once tests are configured, the DataKitchen platform collects order run information from every recipe execution for fine-grained transparency into your end-to-end system. Here the user reviews the test results for a line count test over time. In this case, the counts are always increasing as expected with every addition to the database. The user can also check the order run duration to ensure that everything is running smoothly. Other metrics track tests over time to clearly show when models drift or identify the tests that fail most frequently. This is a form of statistical process control which is another important element of DataOps.

The DataKitchen platform also produces one combined data store with system-wide process metrics for the analytics system as a whole. Data on errors, collaboration, productivity, and deployment time can be used to consistently improve quality and reduce delivery time. For example, these metrics and dashboards show a reduction in errors. While the number of tests rises, tests fulfill a dual role. Tests in a development environment validate changing code while data remains fixed. The same tests are promoted and run automatically in production. Here they validate changing data while the code is fixed.

With a robust testing system in place, errors are caught early and addressed rapidly. This reduces data downtime improves your team's productivity and builds trust in the quality and reliability of the analytics. Now your team can stop worrying about everything that went wrong and start winning respect and appreciation for everything that goes right.