A/B Test Calculator

What is Statistical Significance?

Statistical significance (confidence level) tells you the probability that the difference in conversion rates between Variant A and Variant B is real, and not just due to random chance.

95% Confidence means there is only a 5% chance (p-value < 0.05) that the observed difference is a fluke.

Relative Lift

Relative lift measures the percentage improvement of Variant B over Variant A.

Formula: ((CR_B - CR_A) / CR_A) × 100

Example: If A converts at 2% and B at 2.5%, the lift is ((2.5 - 2) / 2) × 100 = 25% improvement.

Common Pitfalls

• Peeking: Checking results too early and stopping the test prematurely inflates false positive rates.
• Sample Size: Running a test with too few visitors leads to unreliable results.
• Multiple Testing: Testing too many variants at once without adjusting significance thresholds.
• Seasonality: Not running the test for full weekly cycles (e.g., stopping on a Tuesday).

Best Practices

• Form a hypothesis: "Changing X will improve Y because Z."
• Calculate sample size beforehand: Use a sample size calculator to know how long to run the test.
• Run for full cycles: Let the test run for at least 1-2 full business weeks.
• Focus on primary metrics: Don't get distracted by secondary metric fluctuations.