Statistical significance in A/B testing, in plain English
In a classic A/B test, 95% significance means a difference this large would show up less than 5% of the time if the two versions were really the same. It is not a 95% chance your version is better. Bayesian tools report that second number directly: the probability the new version beats the control.
Frequentist vs Bayesian, side by side
| Frequentist (p-value) | Bayesian | |
|---|---|---|
| Answers | How surprising is this if nothing changed? | How likely is B better than A? |
| Checking early | Breaks the math | Allowed |
| Typical bar | p < 0.05 | 95% chance to beat control |
Mistakes that create false winners
- Stopping the moment the test turns significant (peeking).
- Testing many goals or many segments and reporting the one that won.
- Too few conversions: 20 orders a side can swing on luck alone.
- Changing ads, prices or the theme mid-test.
What Luria reports
Luria reports the chance each version beats your unchanged page, and keeps a change only when that is 95% or higher after at least 200 visits per version. See how results are measured.
Questions
Is 90% significance good enough?
It is a business choice: at 90% you accept more false winners in exchange for faster answers. For changes that are cheap to undo, some teams accept it; for prices and redesigns, use 95% or more.