Continuous Delivery and Data Management

Continuous Delivery for Data is less common than CD for other parts of the system, but if we want to be able to grow and evolve our understanding of the data structures and stores that underpin the systems that we build it is just as important. Data management cannot operate as an afterthought, the data structures and algorithms that we choose have an influence on our ability to manage data change. Data migration is a key tool.

In this episode Dave Farley describes the basics of a Continuous Delivery, DevOps, approach for Data. How do we apply the software engineering disciplines of CD to Data? How do we configuration manage, deploy, migrate and test the data related aspects of our systems?

-------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------

If you want to learn Continuous Delivery and DevOps skills, check out Dave Farley's courses:

📚 BOOKS:

📖 Dave’s NEW BOOK "Modern Software Engineering" is now available on
In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience.

📖 "Continuous Delivery Pipelines" by Dave Farley

📖 The original award-winning “Continuous Delivery" book by Dave Farley and Jez Humble
--------------------------------------------------------------------------------------



----------------------------------------------------------------------------------------------------------------------
Useful Books on this topic:
(Please note, if you buy a book from these links I get a small fee, without increasing the cost to you)

"Accelerate, The science of Lean Software and DevOps", by Nicole Fosgren, Jez Humble & Gene Kim

Refactoring Databases: Evolutionary Database Design, by Scott Ambler & Pramod Sadalage

[seyofori]

really insightful. a practical demonstration of these concepts would be priceless.

[viniricardoferrera]

Amazing!
Recently I watched a video of an brazillian channel (aka. Linux Tips) that talked about the DevOps culture and how we can protect our intelectual property using simple techniques as logs. To manage a big project require more observation and an awareness than everything else. To respect the entropy principle is good for CI/CD too.

[KieranShaw]

Great video as usual Dave. I'll be sharing this at work as I'm currently trying to convince a large organisation to do this and hearing from you will help :) Personally, my tool of choice is liquibase for this normally

[petermanger]

For those looking, I found using Liquibase reasonably successful against Oracle 12 a few years ago. A few bits needed hitting with a hammer from our DevOps automation (Jenkins and Octopus) script expert - but day-to-day this worked quite well for the coding team. New project now: I'm now looking into finding a tool to help with a legacy SQL Server system. Not sure about data version management, but I'm going for at least structure versioning! Wish me luck! (oh - we never really got rollbacks working - we just rolled forward).

[dinoscheidt]

In the age of machine learning, this is just getting more and more important = data.

[frank-michaeljaeschke]

Used Liquibase in a few projects successfully against Oracle databases (and H2 for testing). Compared Liquibase with Flyway, Liquibase has support for undo (Flyway does support this only in commercial version) and you could generate a DIFF SQL script against target DB (needed to use this as the client allowed the DB user, the application is using no DDL permissions). Flyway does support this only in commercial version too.

[zkolonits]

Running undo script on a production system is a partial data loss. Well, it's better than nothing, but it is less than what the word versioning suggests.

[PaulSebastianM]

If only more enterprises knew how to do this.

[rstewart]

Central to the whole philosophy of Continuous Delivery is the assumption that it is inexpensive to quickly create and use copies of the production environment. If that assumption is violated, then Continuous Delivery and testing against a “production like” system is not possible and therefore it is not possible to do Continuous Delivery. Am I correct?

Creating a production like environment includes, in general, cloning and reproduction of production databases, does it not?

How do real companies doing development for real ERP systems, such as SAP or Oracle EBusiness, achieve cheap clones of production so that Continuous Delivery and testing are possible? Again, if you can’t clone production for cheap, nothing else about Continuous Delivery can become reality, right?