How to write tests for Power BI, Synapse Pipelines, Data Factory, Synapse and Databricks Notebooks

How to ensure quality, and avoid inaccuracies in your data insights.

The need to validate business rules and security boundaries within a data solution is important, as well as the need for ensuring that quality doesn't regress over time as it evolves. Data insights are useless or even dangerous if they can't be trusted, and in this session, James will explain how and why they should treated just like any other software project with respect to testing - by building automated quality gates into the end to end development process.

During the session, he'll walkthrough some practical examples and proven techniques around testing data solutions - including Power BI reports, Synapse Pipelines and interactive Spark notebooks.

00:00 Intro
01:18 Data mistakes during COVID
02:54 The lack of process was the problem
04:31 Does it matter if it's wrong?
05:25 Excuse 1 - We don't have the time!
06:15 Excuse 2 - We don't know what that answer is!
07:14 Excuse 3 - It's hard!
07:28 It's all code and code should be tested!
08:46 Testing Power BI
26:50 Testing Azure Synapse Pipelines
42:15 Testing Azure Synapse Notebooks
55:47 Wrap up

#PowerBI #AzureSynapse #Spark #Pipelines #dataengineering #testing #qualityassurance #jupyternotebook #data #analytics #datascience #insights