Research7 min

A/B testing pitfalls: when data leads designers astray

The A/B testing pitfalls that mislead design teams: peeking early, tiny samples, the wrong metric, ignored segments and novelty effects, and how to avoid them.

Saurabh YadavFounder, Design Studio by Saurabh
ResearchAb testing pitfallsA/b testing mistakesWhen not to ab test
Share

A/B tests feel like the end of every design argument. Two versions go live, the numbers come in, and the data decides. No more opinions, no more taste.

Except data can lie, or at least mislead, when the test is set up badly or read too eagerly. Teams ship "winners" that were never real, kill good ideas after a week, and optimise a button colour while the real problem sits two screens earlier.

I'm a big believer in letting evidence, not opinion, decide design. That's exactly why it's worth knowing the A/B testing pitfalls that make evidence unreliable. Here are the ones I see most often, and what to do about each.

What an A/B test can and can't tell you

An A/B test shows you which of two versions performed better on a chosen metric, for the people who saw it, during the time it ran. That's powerful and narrow.

It can't tell you why one version won. It can't tell you whether a third, better option exists. And it can't judge things that don't show up in your metric, like trust, confusion or long-term satisfaction.

That's why I treat A/B testing as one tool in the kit, alongside interviews and usability tests. My post on qualitative and quantitative data covers how they fit together.

The pitfalls that lead designers astray

Pitfall What it looks like How to avoid it
Peeking and stopping early "It's up after two days, ship it!" Decide sample size and duration before starting
Too little traffic Tests on low-traffic pages that never settle Test bigger changes, or use qualitative methods instead
The wrong metric Clicks go up, sales don't Pick one primary metric tied to real value, plus guardrails
No hypothesis "Let's try a green button and see" Write what you expect, why, and what you'll learn
Ignoring segments A win on desktop hides a loss on mobile Check key segments you planned in advance
Novelty effects A new design spikes, then fades Run long enough to cover full usage cycles
Testing tiny details Endless button-colour tests Test changes that address a real user problem

1. Peeking and stopping early

This is the most common mistake. You launch a test, check the dashboard every morning, and the moment one version looks ahead, you call it.

Early results swing around a lot. If you keep checking and stop the moment something looks significant, you'll often stop on a random fluctuation. That "winner" may perform no better than the original once it's live.

Fix: decide your sample size and how long the test will run before you start, and don't call it early. Many testing tools include a calculator. Use it.

2. Not enough traffic

A/B testing needs volume. On a page with a few hundred visitors a week, detecting a small improvement can take a very long time, if it's possible at all.

Fix: for low-traffic products, test bolder changes that could produce larger differences, or skip the A/B test and use qualitative research. Watching five people try your flow often teaches more than a month of underpowered testing. See usability testing on a budget.

Note

"Not statistically significant" doesn't mean "no difference". It means the test couldn't tell. An inconclusive result isn't a failed idea, it's an unanswered question.

3. Measuring the wrong thing

A new design might get more clicks on "Start trial" but fewer people who actually finish sign-up, or more sign-ups who churn faster. If clicks are your only metric, you'll call it a win.

Fix: pick one primary metric that reflects real value (completed sign-ups, first successful transfer, retained users), and set guardrail metrics like refunds, support tickets or cancellations. If a guardrail gets worse, it's not a win.

4. No hypothesis

"Let's test a bigger button" isn't a hypothesis. "Users miss the main action on mobile because it sits below the fold; moving it into the bottom third will increase completed bookings" is.

A clear hypothesis tells you what to measure and, just as usefully, what you learned even if the test loses.

Tip

Write every hypothesis in one line: "Because we saw [evidence], we believe [change] will cause [outcome] for [users]." If you can't fill in the evidence, do research before you test.

5. Slicing results after the fact

Once a test is done, it's tempting to dig until you find a segment where the new version won. "It lost overall, but it won for Android users in Pune on Tuesdays!" Look hard enough and you'll always find something.

Fix: decide the segments you care about before the test (mobile vs desktop, new vs returning users) and treat anything else as an idea for a new test, not a result.

6. Novelty and learning effects

Returning users often react to change itself. A new layout can spike engagement because it's new, or dip because people need to relearn it. Neither tells you how it'll perform in a month.

Fix: run tests across full weekly cycles, look at new and returning users separately, and be patient with redesigns of familiar screens.

7. Optimising crumbs

Button colours and microcopy tweaks are easy to test, so they get tested. Meanwhile the page takes ages to load on mobile, or the form asks for twelve fields.

Fix: test where the real problem is. Usability research and analytics tell you where people struggle. A/B testing tells you which fix works best.

Watch out

Don't let A/B tests become a way to avoid design decisions. Some things, like accessibility fixes, honest pricing or removing a dark pattern, are simply right. You don't need a test to justify them.

A pre-launch checklist for any A/B test

  • A written hypothesis based on real evidence
  • One primary metric tied to user or business value
  • Guardrail metrics chosen
  • Sample size and duration calculated in advance
  • Key segments listed before launch
  • Both versions checked on real devices, including budget Android phones
  • An agreed plan for what you'll do if it wins, loses or is inconclusive

That last one saves arguments. If everyone agrees the decision rule before seeing results, nobody can quietly move the goalposts afterwards.

Work with us

Need research done for you? Our UX research and usability testing services cover interviews, surveys and usability tests, with clear recommendations.

FAQ

What are the most common A/B testing mistakes?

Stopping tests early when results look good, running tests without enough traffic, measuring a shallow metric like clicks, testing without a hypothesis and digging through segments after the fact. Each one can make random noise look like a real win. Planning before launch prevents most of them.

When should you not use A/B testing?

When traffic is too low to reach a reliable result, when you need to understand why users struggle, or when the change is simply the right thing to do, like fixing accessibility. In those cases, usability testing, interviews or plain good judgement work better.

Can A/B testing replace usability testing?

No. A/B testing tells you which version performed better on a metric. Usability testing tells you why people struggle and what to change. They answer different questions, and the strongest teams use both.

Key takeaways

  • A/B tests show which version won on one metric, not why, and not what else might work.
  • Decide sample size, duration, metrics and segments before launch, and don't peek.
  • Low traffic needs bolder changes or qualitative methods instead.
  • Use guardrail metrics so a shallow win doesn't hide a real loss.
  • Test changes that solve real user problems, not just easy details.

What to do next

Look at the last test your team called a win and check three things: was the sample size decided in advance, was the metric meaningful, and did anyone check guardrails? If you want help pairing experiments with proper user research, tell me about your project. For the metrics side, my post on measuring UX impact is a good next read.

Saurabh Yadav's Memoji

Written by

Saurabh Yadav

Founder of Design Studio by Saurabh, a UI/UX design studio in India. We design apps and websites that feel effortless, from research to shipped pixels. Most recently: Pukaar.ai, as seen on Shark Tank India, and the Demerara Bank app.

Keep reading

All articles →