Breaking down a data pipeline monolith to improve security vulnerabilities

Having a robust approach to security vulnerabilities is essential for business safety. When you’re a global financial services organisation, it’s vital to ensure any security issue is spotted quickly and addressed in a fast and efficient way with reduced risk to the business.

Our client sought support improving the data pipelines within its security vulnerabilities team. The current system was a monolith, with one mechanism handling all incoming and outgoing data. It made deploying changes a risky and time-consuming process which often resulted in deployments being rolled back due to errors.

Equal Experts introduced the concept of individual pipelines as products to break down the monolith and separate elements into individual pipelines which could be quickly changed with reduced risk to the rest of the system.

The initial deployment of individual data pipelines marked the start of our client’s journey in breaking down the monolith. The upskilling of the client’s team in frameworks, data strategies, data pipeline design and quality assurance has also seen significant improvement in code quality with the number of deployments rolled back reducing from 80% to almost zero during the engagement.