R Tutorial: Designing an Experiment - Power Analysis

---

Now that we have a good sense of our baseline numbers we're ready to design our experiment. Here we'll use our knowledge of seasonality along with power analysis to figure out how long we need to run our experiment.

In preparing our experiment we learned about historical conversion rates. On average conversion rates are about 28%, but that can change throughout the year.

What does this mean for building our experiment? Well, it would be bad to run the control condition in August and the test condition September, because the control may look better simply due to seasonality, not because it's actually a better condition.

This is why A/B experiments try to run both conditions simultaneously, to ensure both conditions are exposed to similar seasonal variables.

We also need to consider seasonal effects for knowing how we expect our control condition to perform. If the experiment is run in

January we expect the control to have a conversion rate of roughly 20%, but if it's run in August the control should be closer to 50%.

With this knowledge, we use a power analysis to determine how long we should run our experiment.

Experiment length is one of the big questions in A/B testing. If you stop too soon you may not get enough data to see an effect. Too long and you may waste valuable resources on a failed experiment.

One way to safeguard against this is with a power analysis. A power analysis will tell you how many data points (or your sample size) that you need to be sure an effect is real. Once you have your sample size, you can figure out how long you will need to run the experiment to get your number of required data points. This will depend on variables such as how many websites hits you get per day. Running a power analysis is also good because it makes you think about what statistical test you want to run before starting data collection.

When running a power analysis you should know:
one, the planned statistical test,
two, the value of the control condition,
and three, the desired or expected value of the test condition.

You also need to know:
one, the proportion of the data from the test condition (ideally 0.5, or half),
two, the significance threshold or alpha (generally 0.05),
and three, the power (generally 0.8). Terms such as "alpha" and "power" should be familiar to you already from DataCamp's course on experimental design.

There are several packages you can use to run a power analysis in R. Here we'll use the `powerMediation` package.

The first thing we need to decide is what statistical test we'll be running. Since the value we're collecting is binary (clicked or didn't click) we'll run a logistic regression.

To run a power analysis for a logistic regression we'll use the function `SSizeLogisticBin`. We'll also save the result of our power analysis to a variable `total_sample_size`.

Now we need to fill in each of the pieces of our equation to get our final sample size. We'll work backward to figure out each of our variables.

For sample proportion (beta), alpha, and power we'll use the most common values (0.5, 0.05, and 0.8).

For our conversion rate for our control condition (p1) let's say we're going to run the experiment starting in January, so we expect roughly a 20% conversion rate. Now the last and hardest part is deciding the expected conversion rate for the test condition (p2). Normally this is backed by previous data, but for now, let's guess and dream big. We'll say a conversion rate of 30%, a 10% boost.

We see we need 587 data points in total or roughly 294 per condition.

Let's get some more practice running power analyses.

In Chapter 2 we'll get some results from our experiment and get to analyze them with logistic regression in R.

#DataCamp #RTutorial #ABTestinginR DesigninganExperiment #PowerAnalysis