Testing Strategy for DevOps: What to Test and When

Your testing strategy is central to your ability to deliver better software faster. Excellent teams have a laser focus on testing, and this focus extends way beyond what we do before we release our changes into production.
Testing to support DevOps and Continuous Delivery is a core idea, and testing in Production is part of that game too. So what should we test and when should we test it? What kinds of testing techniques should we adopt?
In this episode Dave Farley will describe his take on what a holistic test-strategy looks like, and gives us examples from the real world to describe it.

-------------------------------------------------------------------------------------

LINKS:

-------------------------------------------------------------------------------------

🎓 CD TRAINING COURSES 🎓

Get a FREE copy of Dave's "WHAT TO TEST & WHEN?" Guide.
Sign up to the CD Mail List to keep up to date with the latest discussions, other free "How To..." guides, events and online courses.

📚 BOOKS:

📖 Dave’s NEW BOOK "Modern Software Engineering" is now available on
(Paperback version available soon)
In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience.

📖 "Continuous Delivery Pipelines" by Dave Farley

📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble

-------------------------------------------------------------------------------------

CHANNEL SPONSORS:

[fmkoba]

the main factor for me to join ThoughtWorks was knowing I would be able to work with and learn from professionals like Dave, I'm starting next monday as a QA Engineer

[jangohemmes]

You've quickly become a favorite for me. Thank you for your insights

[bearbuttnaked]

Excellent content as usual, so helpfull, thanks Dave for the amazing job of making it so simple when you say it ;)

[jimiwikmanofficial]

An excellent summary of testing strategies!

[mattgraves]

I am an Android focused engineer, but today anymore, I think if your not doing CD, your working towards it.
Or should be...,
I can see that now, even in smaller organizations what a huge benefit it would be to have a continually deliverable and releasable product every planned iteration...I am personally loving it.

[bernardogalvao]

I'd love to hear your take on delivering Machine Learning projects.

[PDuimelaar]

Great work Dave! I'm a QA engineer myself and always looking for nice vids to improve my knowledge and I can tell you I'm totally in line with your story. Thats why i started following you! Keep up the great work!

I was just wondering. I've seen the sheet you're using somewhere else but i can't seem to find it. Do you mind sharing it as I might want to use it as a reference sheet to help customers out improving their QA process. I'm aware not everything on the sheet will be usable with every customer but it's always nice to have some guidelines.

[veliromn]

Thanks Dave! Could you point to the source/article of you mentioned that according to state of devops report says that 60% of critical bugs could have been prevented by unit tests?

[vinhtrinh]

Hello, do you have experience or plan to do any videos in regards to video game development and best practices in the software of the video game industry? A lot of your concepts of course are perfect for backend/server side development of games, but fast iteration and test coverage of client side stuff is very difficult.

[danm]

Thanks, great as always :) 5 minutes for commit tests raised an eyebrow, though - I think depends as always on context but if you're committing say 20 times a day that's 100 minutes of potential flow interruption presuming no re-runs... seems high and 5 minutes definitely feels high to me whenever I've had to deal with it... Faster is better all other aspects equal, it's good to aim low, imo... Maybe a video on build times and various cycle times effects? Anyway, thanks again.

[RoelBaardman]

How is canary testing implemented in practice? Feature toggles based on for example local time, in your Netflix example?

[benfellows]

Great video with many good ideas for testing in production, will definitely be coming back to it!!

Though there's a lot of mentioning of confirming and validating in testing but there's something niggling in the back of my mind that doesn't sit right. Surely the point of testing is to disconfirm and invalidate? Actively look for ways in which it doesn't work or deliver value, not just focusing on how it does. Otherwise you open up yourself to confirmation bias and might be missing huge areas of unknown risk and failure? I think the video hinted at this with some techniques, but only lightly.

[trungkiennguyen]

What is your experience with scheduled releases?
Do you prefer weekly releases or bi-weekly releases in general?
Do you try to limit the PRs that come into each release to control the technical scope-of-change happening each release or do you let the product/project manager decide what can go into every release?

In the place I work in, we would release every week, first 3 days is for merging and testing the PRs ( project manager decide what can go every week ), next day to run stage-deployment and internal users to test, last day for production ( Friday )
So far it is working well for us, sometimes we would move production-release to Monday if the release is critical and need extra care.
Personally, I don't have much experience with release control, so looking to learn more from you regarding this aspect.

I understand the endgoal should be continous delivery ( PR merged automatically goes to stage, press a button and production is updated ), we are working to reach it but still a while away.

Thank you for the great videos! Heard you on the ChangeLog podcast the other day and enjoyed hearing your insights and opinions thoroughly.

[OverPwng]

This is good stuff Dave, this is good stuff!

[TristanBailey]

Could you talk around the testing in prod part and if you keep consistent in your testing and keep it in your logging and db or do you exclude the data points or records they create.

[alejandrochavezbarrera]

I like your videos, thanks for your effort!

[mirzasisic]

Production is the best environment for destructive testing! :D

[elciomello]

Hello Dave, Love your videos :)
Can you tell me how/what you measure the level of confidence that you talk about in the commit cycle?
In the case you say 80%, but what are you using to composite this number? E.g. Code coverage, mutation score and etc
Thanks

[danielwilkowski]

7:55 okay, you realse something for a day to the green marker, and only release it to the yellow market on the day after. How does that correlate to continuous delivery. If I write a new feature, do I also only release it to green and not to yellow? I presume, if I add 10 more features that day, I should release them only to the green market? Does this mean that my feedback loop to the yellow market is 1 day instead of 5 minutes?

[hananurrehman]

Why does it feel like you have completely missed the part where a software testers tests the feature by hand?
Or am I missing something?