How Much Sample Size a Mobile Paywall Test Actually Needs
·Metrics
A mobile paywall test needs enough users to generate approximately 385 conversions per variant to reliably detect a 20% relative improvement with standard statistical confidence. This translates to very different absolute sample sizes depending on your baseline conversion rate: if 3% of users convert, you need roughly 12,800 users per variant, while a 1% baseline requires about 38,500 users per variant.
The required sample size scales inversely with your conversion rate and the minimum effect size you want to detect. Smaller improvements require exponentially more data—detecting a 10% lift needs four times as many users as detecting a 20% lift. Most subscription apps should design tests to detect 15-25% relative improvements, as smaller changes rarely justify the engineering investment even when statistically significant.
The relationship between conversion rate and sample size
Lower baseline conversion rates require dramatically larger samples to achieve the same statistical power. An app with 5% paywall conversion needs about 7,700 users per variant to detect a 20% lift, while an app with 1% conversion needs 38,500 users for the same test. This happens because conversions are your signal and non-conversions are noise—when conversions are rare, you need much more data to distinguish real improvement from random variation.
Early-stage apps with uncertain conversion rates should use conservative estimates when planning tests. If you think your conversion rate is around 2-3% but don't have solid historical data, plan for the lower end of your estimate. Running underpowered tests wastes traffic and time, often leading to false negatives where real improvements go undetected because you didn't collect enough data.
Minimum detectable effect and why it matters
The minimum detectable effect (MDE) is the smallest improvement your test can reliably identify, typically set between 10-25% for paywall tests. Setting your MDE requires balancing statistical feasibility with business relevance. A 5% improvement might add meaningful revenue at scale, but detecting it requires such large samples that you'd need to run tests for months, during which you could have tested multiple larger changes.
Focusing on larger effect sizes makes experimentation programs more efficient. A paywall redesign that only lifts conversion by 8% probably isn't worth the effort of building, testing, and maintaining a new variant. By designing tests to detect 20%+ improvements, you naturally filter for changes substantial enough to matter. If a change can't produce at least a 15-20% lift, question whether it's worth testing at all.
Two-sided tests versus one-sided tests
Most paywall tests should use two-sided statistical tests that can detect both improvements and decrements, even though you're hoping for improvement. Two-sided tests are more conservative, requiring slightly larger samples to achieve the same power as one-sided tests. The trade-off is worth it because you genuinely don't know if your variant will win—many seemingly promising paywall changes reduce conversion when exposed to real users.
One-sided tests are appropriate only when you'd take the same action regardless of the direction of the effect. For example, if you're testing whether to remove a feature that's costly to maintain, and you'll remove it if it has no effect or negative effect, a one-sided test might fit. For typical paywall optimization where you want to know the true directional impact, stick with two-sided tests despite the sample size penalty.
How traffic volume affects your testing velocity
Apps with low traffic face a difficult choice between testing velocity and statistical rigor. If you receive 5,000 new users per week and need 25,000 per variant to test your paywall, a single two-variant test takes 10 weeks. Running multiple concurrent tests would fragment your traffic further, extending timelines or forcing you to accept lower statistical power. This constraint means low-traffic apps must be more selective about what they test.
High-traffic apps can afford to test smaller improvements and run more experiments simultaneously. With 50,000 weekly users, you can run a properly powered paywall test in under two weeks, or even split traffic across multiple concurrent experiments. This velocity advantage compounds over time—high-traffic apps can iterate faster, learn more, and pull further ahead. If you're in a low-traffic situation, focus experimentation on pre-paywall engagement and retention where sample sizes accumulate faster.
Practical sample size calculation for subscription paywalls
Start with your baseline paywall conversion rate from analytics, then use a power analysis calculator to determine required sample size. Input your baseline rate, desired MDE (typically 20%), significance level (usually 0.05), and power (typically 0.80). The calculator returns the conversions needed per variant, which you divide by your conversion rate to get the required user sample. For a 2.5% baseline targeting a 20% lift, you need about 316 conversions per variant, or 12,640 users per variant.
Account for attribution delays when planning test duration for subscription apps. Tools like Spinosis that attribute RevenueCat purchases require the App User ID to match the experiment user ID, which works cleanly for immediate paywalls but introduces delays when users convert after trials. If 30% of your conversions happen at trial expiration rather than immediately, extend your test timeline to capture those delayed conversions in your sample size calculation, typically adding 7-14 days beyond the point where you reach your user count target.
Questions
Can I reduce sample size requirements by increasing statistical significance threshold?
Increasing your significance threshold from 0.05 to 0.10 does reduce required sample size, but it also doubles your false positive rate—you'll incorrectly declare losers as winners twice as often. This trade-off rarely makes sense for paywall tests where implementing the wrong variant has real revenue costs. Instead of loosening significance thresholds, consider increasing your minimum detectable effect or using Bayesian methods that provide more nuanced probability statements about variant performance.
What if I have different conversion rates across user segments?
When conversion rates vary significantly by segment (new vs. returning users, different traffic sources, iOS vs. Android), calculate sample size based on your blended average rate but be prepared for subgroup analyses to be underpowered. If a particular high-value segment represents only 20% of traffic, you'll have one-fifth the statistical power for that subgroup. For critical segments with different behavior, consider running dedicated tests with traffic filtered to that segment, accepting the longer runtime required.
How does running a multi-variant test affect sample size needs?
Each additional variant increases total sample size requirements but not linearly. A three-variant test (two treatments plus control) needs about 3x the total traffic of a two-variant test to maintain the same power for each comparison. Multi-variant tests are most efficient when you're choosing between distinct alternatives rather than testing minor variations. If you're testing three similar price points, consider sequential tests or a multi-armed bandit approach instead of a traditional A/B/C test that fragments your traffic across all variants simultaneously.
Turn on auto-optimize for one test.
Create a project, install the SDK, connect RevenueCat. Traffic starts shifting on its own.