How long should you run an A/B test?
Run an A/B test for at least one full week, ideally two, and until each version reaches the sample size your conversion rate and target lift need. Stopping as soon as a result looks significant, or after a set number of days regardless of traffic, produces false winners.
The two rules
- Whole weeks. People buy differently on Monday and Saturday, and on payday. One or two full weeks cover that.
- Enough visits. Each version needs the sample size for your rate: about 21,106 visitors per version to see a 20% lift at a 2% conversion rate (full table).
Worked examples
| Visitors a month on the page | Conversion rate | Lift to detect | About how long |
|---|---|---|---|
| 10,000 | 2% | 20% | 18.3 weeks |
| 50,000 | 2% | 20% | 3.7 weeks |
| 50,000 | 3% | 30% | 1.1 weeks |
| 200,000 | 2% | 10% | 3.5 weeks |
95% confidence, 80% power, two versions. Round up to whole weeks.
When it is fine to stop early
Fixed-sample tests should not be stopped early. Bayesian tests and multi-armed bandits are designed to be checked as they run: they move traffic to the leader and stop when the chance that it is best passes a bar such as 95%. Luria works this way, with a floor of 200 visits per version.
Questions
What is the longest an A/B test should run?
Past four to six weeks, cookie deletion, seasonality and outside changes start to blur the result. If a test needs longer, test a bigger change or a higher-traffic page.