A/B Tests: The Five Ways They Go Wrong

A properly run experiment is the most reliable thing in marketing measurement. A badly run one is worse than no test at all, because it produces a confident number that people act on. When experiment outcomes depend on questionnaires or manual declarations, self-reporting bias is another source of error to account for.

The failures are not exotic. Five of them account for nearly everything.

1. Peeking

The most common, and the one most people do not know is a mistake.

You start a test, check it daily, and stop when the difference reaches significance.

This is not a small violation. The p-value assumes a single test at a predetermined sample size. Checking repeatedly and stopping at the first significant result means you are running many implicit tests, and the chance of a false positive rises with every look.

The intuition: a result fluctuates as data accumulates. If you sample randomly enough moments, it will cross the threshold at some point by chance alone. Stopping there records the crossing and discards the fluctuation.

Effects in practice: a large share of "winners" in organisations that peek do not replicate. Teams accumulate a catalogue of wins that do not add up to any change in the business, and nobody connects the two.

The fixes:

Fix the sample size in advance and do not stop early. Simplest, and it requires discipline.

Or use a method designed for continuous monitoring — sequential testing, or a Bayesian approach with a pre-specified decision rule. These allow you to look; they are designed for it. What is not allowed is using a fixed-horizon p-value as though it were one of them.

Watching without acting is fine. Looking at a dashboard is not the error. Stopping because of what you saw is.

2. Underpowering

The test could never have detected the effect it was looking for.

Power is the probability of detecting a real effect of a given size. It depends on the baseline rate, the effect size you care about, the variance and the sample.

Calculate before running: given my traffic and conversion rate, what is the smallest effect I could detect in a reasonable time? If that number is 20% and realistic improvements are 2%, the test is theatre.

The consequence of skipping this is a stream of inconclusive tests interpreted as "no difference," which is a different and much weaker statement. An underpowered test that finds nothing has found nothing about the world; it has found something about the test.

And the less obvious consequence: underpowered tests that do reach significance overstate the effect. When only large observed differences can cross the threshold, the ones that do are the ones inflated by noise. This is why small tests produce implausibly large wins that shrink on retest.

If the required sample is unaffordable, that is the finding. Test something with a larger expected effect, or accept the decision cannot be made this way.

3. Multiple comparisons

You test one change and evaluate it against fifteen metrics. Or you run six variants. Or you slice the result by device, country, new versus returning, and channel.

At a 5% threshold, one in twenty comparisons is significant by chance. Twenty slices produce roughly one spurious winner, reliably.

Where it happens most: post-hoc segmentation. The overall test is flat, so someone looks for a segment where it worked. There is always one. That segment is a hypothesis for a future test, not a result.

The fixes:

Name one primary metric before starting, and treat everything else as secondary and exploratory.

Pre-register segments you will examine. Segments discovered afterwards are exploratory by definition.

Adjust the threshold where you genuinely need multiple primary comparisons.

Say "exploratory" out loud in the write-up. The finding may be real; it needs confirming, and labelling it protects everyone including you.

4. Broken randomisation

The groups differ by something other than the treatment.

Sample ratio mismatch is the diagnostic. If assignment is 50/50 and you observe 52/48, something is wrong with assignment, tracking or filtering. Check this on every test, first, before looking at the result. A test with a ratio mismatch should be discarded rather than interpreted — the imbalance means the mechanism you are relying on did not work, and no amount of analysis repairs that.

Common causes:

Assignment on an unstable identifier. A cookie that resets means the same person is assigned repeatedly and can appear in both groups.

Cross-device leakage. Assigned on mobile, converts on desktop, counted as control.

The variant is slower. If the treatment adds load time, you are measuring the change plus the latency, and latency alone moves conversion measurably.

Filtering applied after assignment. Excluding bots or internal traffic differently across groups reintroduces bias.

Interference between users. Where participants can affect each other — marketplaces, social features, shared inventory — the groups are not independent and standard analysis is invalid.

5. Measuring the wrong thing

The test is clean and the metric does not matter.

Clicks instead of conversions. A button that gets more clicks and fewer purchases is a loss reported as a win.

Conversions instead of value. More orders at a lower margin is worse.

Short-term instead of retained. A discount lifts today's conversion and may reduce lifetime value. If the test window is a week and the effect is on repeat purchase, you measured the part that flatters.

Guardrail metrics missing. Every test should have metrics that must not get worse — refund rate, support contacts, unsubscribes, page performance. A win on conversion with a rise in refunds is not a win, and without a guardrail nobody looks.

Before running one

• [ ] One primary metric, named in advance
• [ ] Guardrail metrics named in advance
• [ ] Minimum detectable effect calculated, and affordable
• [ ] Sample size and duration fixed in advance
• [ ] Duration covers at least one full business cycle, including weekends
• [ ] Decision rule agreed: what result leads to what action
• [ ] Segments to examine pre-registered
• [ ] Assignment identifier stable across sessions and devices where possible

After it finishes

• [ ] Sample ratio checked before anything else
• [ ] Result reported as an interval, not a point
• [ ] Guardrails checked
• [ ] Practical significance assessed separately from statistical significance
• [ ] Post-hoc segments labelled exploratory
• [ ] Result recorded whether positive, negative or null

That last one matters more than it looks. Organisations that record only wins build a distorted picture of what works, and repeat tests they have already run. A test archive including the failures is one of the more valuable assets an analytics function can accumulate.

The summary

Peeking manufactures winners. Fix the sample size, or use a method designed for monitoring.

Underpowered tests find nothing and overstate what they find. Calculate the detectable effect first.

Every extra comparison buys another chance at a false positive. One primary metric, pre-registered segments.

Check the sample ratio before the result, every time.

And measure something that matters, with guardrails on the things that must not get worse. For methodological background, see NIST guidance on experimental design.