Can I use Artificial Intelligence to Create Data Quality Rules?

There has been SO much hype recently about the use of artificial intelligence but can we use AI to create data quality rules? Watch my latest Ask the Data Governance Coach to find out!

Want more support through your Data Governance journey?

[phrumdata]

Glad to see this topic covered. I agree that training an ML model on datasets with data quality issues is a poor idea. If an ML model was trained on high quality datasets it will be more effective of course. Since general AI doesn't exist yet we will need human input, it's not out of this world for a data governance application to analyze a dataset and suggest data quality rules. I'm hoping the data governance applications begin to implement something like this feature, it would reduce the amount of human involvement at least. There could even be an ML model for specific dataset contents, for example a database field "SSN" could be read and identified as a social security field. The model would then suggest data quality rules and ask for human input, this is taking use of the data labeling concept.

[SteveCrosson-Smith]

Spot on, Nicola. Statistical Process Control (SPC) is an example of using AI to look for data that is outside of a statistical norm and its real power is to augment human involvement rather than replace it. It simply isn't practical for humans to adaptively monitor the plethora of data flowing around a company, especially as the window of opportunity to act upon insights derived from it is forever narrowing.