Statistical Significance Explained for Subscription App Founders

The standard 95% confidence level means you're accepting a 5% chance of false positives—declaring a winner when there's actually no real difference. This is called a Type I error. Many founders misunderstand this as "95% chance the result is correct," but that's not quite right. It's specifically about how often you'd see a difference this large if you ran the same experiment infinite times when there's truly no effect.

Choosing 95% versus 90% or 99% involves tradeoffs. Higher confidence (99%) reduces false positives but requires larger sample sizes and longer test durations. Lower confidence (90%) lets you make faster decisions but you'll implement ineffective changes more often. For high-stakes decisions like pricing changes, 95% or higher makes sense. For lower-risk UI tweaks, 90% might be acceptable if speed matters more than certainty.

The p-value is the specific probability of seeing results at least this extreme if there's no real difference between variants. A p-value of 0.03 means there's a 3% chance of observing this result (or something more extreme) in a world where the variants are identical. Since 3% is less than your 5% threshold for 95% confidence, you'd declare the result statistically significant.

The confusion comes from what p-values don't tell you. A p-value of 0.03 doesn't mean there's a 97% chance variant B is better, nor does it tell you how much better B actually is. It only addresses whether there's a detectable difference. You could have a highly significant result (p = 0.001) that represents a tiny, business-irrelevant improvement. Conversely, a marginally non-significant result (p = 0.07) might represent a meaningful improvement that you just didn't collect enough data to confirm.

A result can be statistically significant without being practically useful for your business. If you run an onboarding experiment with 100,000 users and find that variant B converts 10.05% versus variant A's 10.00%, that 0.05 percentage point difference might be statistically significant but it's too small to justify the engineering effort of implementing and maintaining a new onboarding flow.

Define your minimum detectable effect before running experiments. This is the smallest improvement that would actually change your business decisions. For a subscription app doing $50,000 monthly revenue, a 2% increase in conversion might be worth pursuing because it represents $1,000 monthly and compounds over time. A 0.2% increase probably isn't worth the opportunity cost. Set this threshold based on implementation effort, maintenance costs, and strategic priorities—not just on whether you can prove the difference is real.

Statistical power is the flip side of significance—it's the probability that your test will detect a real difference if one exists. While significance controls false positives, power controls false negatives (Type II errors). A test with 80% power has a 20% chance of missing a real improvement because you didn't collect enough data. Most experiments should target 80% power as a reasonable balance.

Power depends on three factors: the size of the effect you're trying to detect, your sample size, and your significance threshold. To detect smaller improvements, you need larger samples. If your baseline conversion is 8% and you want to reliably detect a 10% relative improvement with 95% significance and 80% power, you need roughly 15,000 users per variant. Tools like Spinosis that attribute purchases to experiment variants help ensure you're measuring the right conversion event, but you still need sufficient volume to reach meaningful conclusions.

Real-world testing often doesn't deliver perfectly significant results on your preferred timeline. You might reach day 14 of a planned two-week test and see variant B ahead by 8% with a p-value of 0.08—close to significant but not quite there. Extending the test might get you to significance, or the difference might disappear as a statistical fluctuation. These situations require judgment, not just math.

Consider the asymmetric risks of being wrong. If variant B is a minor UI change that's easy to revert, implementing it at 92% confidence is lower risk than if it's a complete onboarding redesign. Factor in opportunity cost too—continuing the experiment for two more weeks to gain 3% more confidence might not be worth delaying other tests. Look at confidence intervals to understand the range of possible true effects. If the 90% confidence interval for variant B's improvement is +4% to +12%, you know it's very likely better, even if you haven't crossed the 95% significance threshold.