What is statistical significance in A/B testing?

Statistical significance (typically 95% or p < 0.05) means there is less than a 5% probability that the observed difference between Control and Variant occurred by pure random chance. It does NOT guarantee revenue lift, only that the difference is mathematically distinguishable from noise.

What is the "Peeking Problem" in A/B testing?

The Peeking Problem occurs when an experimenter checks test results daily and declares a winner the first time p < 0.05. Repeated significance testing on accumulating data inflates the true false-positive rate from 5% to over 30%. Tests must run to predetermined sample sizes.

How long should an e-commerce A/B test run?

An A/B test should run for a minimum of 2 full business cycles (14 full days) to capture weekday vs. weekend purchasing behavior variance and pay-cycle swings, even if statistical significance is reached after 3 days.

How do I calculate required sample size before launching a test?

Sample size per variation is calculated using: N = 16 × [p(1 - p)] / (MDE)², where p is baseline conversion rate and MDE is Minimum Detectable Effect in absolute terms. For a 3% baseline CR detecting a 10% relative lift (0.3% absolute), you need approximately 51,700 visitors per variation.

What is the difference between One-Tailed and Two-Tailed tests?

A one-tailed test only evaluates whether the variant is better than control, ignoring the possibility that it harmed conversion rates. A two-tailed test evaluates differences in both directions (positive or negative) and is the required scientific standard for e-commerce experimentation.

A/B Testing Statistical Significance & CRO Guide (2026)

Statistical Significance in E-Commerce CRO: Sample Size, P-Value & Z-Score Playbook

Why most published split test 'winners' are statistical illusions. The complete mathematical guide to two-tailed Z-tests, calculating required sample sizes, preventing stopping rule fallacies, and ensuring test repeatability.

Dr. Sarah Lin

Lead Quantitative Analyst

Last Updated: August 31, 2026

Companion Interactive Tool

A/B Test Statistical Significance & Sample Size Calculator

Calculate two-tailed Z-scores, p-values, confidence intervals, and required visitor sample sizes for CRO tests.

Launch Simulator

1. The Mathematics of A/B Hypothesis Testing

When testing a new checkout layout or product page variation (B) against your baseline control (A), you are testing a statistical hypothesis:

Null Hypothesis (H0)

Conversion Rate B = Conversion Rate A. Any observed difference is purely random variance.

Alternative Hypothesis (H1)

Conversion Rate B ≠ Conversion Rate A. The variation genuinely alters conversion behavior.

For large e-commerce traffic samples, we apply the Two-Tailed Z-Test for Independent Proportions:

Z = (p_B - p_A) / √[ p_pool × (1 - p_pool) × (1/N_A + 1/N_B) ]

Where p_pool = (Conversions_A + Conversions_B) / (Visitors_A + Visitors_B).

For a 95% confidence level (α = 0.05), the absolute Z-score must exceed 1.96. For 99% confidence (α = 0.01), Z must exceed 2.58.

2. Sample Size Estimation & Minimum Detectable Effect (MDE)

One of the most common mistakes in CRO is declaring a test conclusive before reaching the required sample size. The required sample size per variation is given by:

N = [ (Z_α/2 + Z_β)^2 × (p_A(1 - p_A) + p_B(1 - p_B)) ] / (p_B - p_A)^2

Baseline Conv Rate	Target Relative Lift (MDE)	Expected Variant Rate	Sample Size / Variation	Total Test Traffic
2.0%	+5.0% relative	2.10%	155,000	310,000 visitors
2.0%	+10.0% relative	2.20%	39,000	78,000 visitors
2.0%	+20.0% relative	2.40%	10,000	20,000 visitors
4.0%	+10.0% relative	4.40%	19,200	38,400 visitors

3. The Peeking Problem & False Positive Rate Inflation

In conversion testing software (such as VWO, Optimizely, or Google Optimize historical logs), statistical confidence numbers fluctuate wildly during the first few days. A variant might show 98% confidence after 300 visitors simply due to an early cluster of transactions.

Statistical Law: Alpha Inflation on Sequential Tests

When an experimenter checks test results daily and stops the test the moment p < 0.05 is hit, the real false positive rate rises from 5% to over 30%. This means roughly 1 in 3 implemented "winners" is actually neutral or harmful to your business.

4. The 4 Golden Rules of Rigorous E-Commerce CRO

Pre-Determine Sample Size

Calculate required visitor traffic per variant before starting the test and never stop early.

Minimum 14 Full Days

Run experiments across at least two full business cycles to balance day-of-week customer variance.

Track Revenue Per Visitor (RPV)

A variation that boosts conversion rate but reduces Average Order Value (AOV) can lower total net profit.

Segment Mobile vs Desktop

Always evaluate device segments independently to prevent aggregate Simpson's Paradox distortions.