A/B Testing, Exploration vs. Exploitation - Ep 59

A/B Testing is a sub-approach of experimental design which compares two variants in order to determine which is most effective. A good example of this is the placebo effect which compares new drugs against the status quo in order to determine how well they function. In this episode we learn why this type of testing is currently still being used in a number of industries and why it has historically been so popular. We learn how this approach is profoundly approach weak when applied to a Supply Chain and the complexities which mean alternative approaches work much better.

In business the rules are constantly changing all of the time. We investigate the trade off between exploitation and exploration and why it can be beneficial to introduce some intentional error to algorithms in order to find out a little more information for the future. We debate whether this will be of interest to companies which normally prioritize profitability and learn whether is is possible to actually quantify knowledge. Finally we learn about the impact introducing noise can have on pricing optimization and understand how a Supply Chain professional can decide between the options which are worth exploring and the options which should remain consistent.

******
Episode Map
0:00:00 - Introduction
0:00:34 - What is A/B Testing?
0:01:50 - What types of testing are we talking about here?
0:02:37 - So the idea is to see what out of two possibilities perform the best?
0:03:31 - Why is this something which is of interest to us here at Lokad?
0:04:37 - How well does this technique actually work in the real world?
0:08:15 - What would be a better approach?
0:11:59 - How can we generate information on all of the possible scenarios within a Supply Chain?
0:14:49 - So we are talking about intentionally introducing a percentage error to find out more about what could possibly happen?
0:19:05 - Companies are normally most interested in maximising their profitability. As such is it difficult to get a company to incorporate these intentional errors?
0:21:19 - Is there any way of quantifying this knowledge and working out what it is worth to a company?
0:22:58 - How does this approach fit in with what we do at Lokad? Surely interesting noise fundamentally goes against this belief of maximising every possible purchase decision?

******
******