Multi-armed bandit vs A/B testing
A classic A/B test splits traffic evenly until a fixed sample is reached, then you pick the winner. A multi-armed bandit shifts traffic toward the better version while the test runs, so fewer visitors see the loser. A/B tests measure the size of a difference cleanly; bandits earn more during the test.
Side by side
| A/B test | Multi-armed bandit | |
|---|---|---|
| Traffic split | Fixed (50/50) | Moves toward the leader |
| Cost of a losing version | Half your traffic for the whole test | Falls as the loser falls behind |
| Best at | Measuring exactly how big a lift is | Finding and using the winner fast |
| Checking early | Not allowed (fixed sample) | Built in |
| Common method | Frequentist z-test | Thompson sampling |
How Thompson sampling works
For each version the bandit keeps a belief about its conversion rate (a Beta distribution built from its visits and conversions). For every new visitor it draws one guess per version and shows the version with the highest guess. Uncertain versions still get explored; clearly worse ones get less traffic over time.
The catch, and the fix
A pure bandit can starve your original page, which makes it hard to show the gain. The fix is a holdout: keep a share of traffic on the original no matter what. Luria never lets your page drop below 10% of traffic and keeps a change only at 95% certainty after 200+ visits per version. (Holdout control explained.)
Questions
When should I use a classic A/B test instead of a bandit?
When you need a precise size of effect for a big decision (a price change, a redesign), or when you will learn from the test for months after. For everyday page copy, a bandit with a holdout earns more.