Custom data validation in automated data pipelines

This example shows a data workflow built in CloverDX that's ingesting and processing files from different clients.

The challenge is how to adapt to different data sources, which have different data formats and business rules, without needing to build a new pipeline for each source.

In this video you can see how, instead of having those validation rules for each source explicitly listed in the pipeline, we’re externalizing the rules.

We’re looking at a file for each school, and depending on which school it is, changing the variables, allowing you to change the data validation rules on the fly per source system.