A/B testing best practices for e-commerce
How to plan, run and read an A/B test, with a worked example and two free calculators.
What an A/B test is
Instead of changing a page and hoping, you show half your visitors the current version (the control) and half a new one (the variant). Both run at the same time, to the same kind of visitors. If one converts better, the only explanation is the one thing that differs.
A skincare store shows 1,200 cart visitors a day. 4.0% of them place an order. The team wants to add a bar to the cart that shows how far the visitor is from free shipping.
Why sample size matters
With 20 visitors per version, 2 orders against 3 looks like a +50% lift. A gap like that shows up about 6 times in 10 even when nothing has changed.
Measured on thousands of visitors, the same lift would be a real finding. Work out how many visitors your test needs before you start, and let it run until it has them.
Write a hypothesis
Write down what you expect to happen and why. This makes you choose one metric and gives you something clear to judge the result against.
If we [change], then [metric] will [move], because [reason].
“If we show a free-shipping progress bar in the cart, the order rate will rise by 15%, because visitors close to the threshold will add one more item instead of leaving.”
Check you have enough traffic
Before you set anything up, calculate how long the test needs. You need three numbers:
- your current conversion rate
- the smallest lift that would be worth acting on
- how many visitors a day will see the test
If the answer is more than two months, change the plan. Over that long, seasons, campaigns and site changes start to affect the result.
4.0% today, a 15% lift to detect, 1,200 visitors a day, 95% confidence and 80% power:
If you don't have enough traffic, test a metric higher in the funnel. More visitors click than buy, so a click or add-to-cart test reaches significance much faster and still shows whether the funnel improves.
- Views
- Clicks
- Add-to-carts
- Orders
Before launching, check that the test fits in under two months, that the lift you need is realistic, and that no other test is running on the same visitors.
Free tool
How many days will your test take?
Set up the test
- Duplicate the current version as your control, so both start from the same base
- Add one variant and change only the element your hypothesis names
- Split traffic 50/50, which reaches significance fastest
- Pick one primary metric, or the closest one your tool can track
- Launch and leave copy, design and audience untouched until the date you calculated
Track the test
Your testing tool tells you whether the variant won on the primary metric.
Your analytics tool tells you what the win is worth in revenue and order value, and whether the lift holds further down the funnel. Check both before you decide.
Read the result
- Did the metric move in the direction your hypothesis predicted?
- Is the lift big enough to matter, compared with the effort to roll it out?
- Did it reach the confidence level you set before the test?
18,000 visitors per version. The control converted 720 (4.00%), the variant 820 (4.56%).
The lift is a little below the 15% the team hoped for, and it is significant, so they roll it out.
| You see | Do this |
|---|---|
| Uplift, significant | Roll outShip the variant. |
| Drop, significant | Keep controlThe change hurts. Note why. |
| Close to the line | ExtendRun to the planned sample if you can. |
| Far from the line | Move onNo real difference. Try a bolder idea. |
Free tool
Is your result significant?
After the test
- Log the hypothesis, dates, sample size, result and what you learned, including for tests that lost
- Apply what worked to similar pages, popups and emails
- Rate your next ideas on impact, confidence and effort, and run the highest score first
Common mistakes
- Stopping the test early because the first days look good, when results usually move a lot before they settle
- Editing the versions or the audience while the test is running
- Running the test during a sale or holiday and treating the result as normal
- Changing several things at once, so you can't tell which one made the difference
- Rolling out a lift that is real but too small to be worth the work
What to test
The same method works on every channel. Good places to start are pages with a lot of traffic and changes that are cheap to make.
Onsite messages
- Headline and offer
- Button copy
- Design and position
- Number of form fields
- Single vs multi-step forms
Email and SMS
- Subject line and preview
- Send time and delay
- Offer vs no offer
- Flow length
- Product blocks and layout
When and to whom
- Trigger: delay, scroll, exit intent
- New vs returning visitors
- Traffic source and device
- Frequency caps and cooldowns
- Hiding after purchase
Frequently asked questions
What is the difference between a one-sided and a two-sided test?
Can I A/B test with low traffic?
How many versions can I test at once?
What is the difference between A/B and multivariate testing?
Should traffic always be split 50/50?
How long should an A/B test run?
Want a testing roadmap?
