How to A/B Test Mobile Onboarding Without Guessing

Users who see variant A on Monday must see variant A on every subsequent session. When variant assignment changes between sessions, you corrupt the experiment by exposing users to multiple treatments. This is especially problematic for mobile onboarding, where users often complete flows across multiple app opens over several days.

Implement sticky assignment by storing the variant identifier with the user record at first launch, before any onboarding screens appear. Use the same user identifier throughout the experiment that you'll later use for purchase attribution. This ensures each user experiences one consistent onboarding path from install through subscription decision.

Split new users randomly into variants at their first app launch, not at the point they enter onboarding. Splitting at onboarding entry creates selection bias because different variants may affect whether users reach that point. Your baseline conversion rate becomes unreliable when the denominator shifts between variants.

Maintain a holdout group even after identifying a winner. When you shift all traffic to the winning variant, you lose the ability to measure its ongoing performance against the baseline. A permanent 10-20% holdout lets you quantify sustained lift and detect when a previously winning variant stops performing due to changing user behavior or market conditions.

The gap between onboarding completion and subscription purchase often spans days or weeks, requiring careful user identification across systems. Your onboarding experiment needs to track the same user ID that your payment processor associates with purchases. Mismatched identifiers between systems make it impossible to calculate revenue per variant accurately.

Most subscription apps use RevenueCat or similar services that assign an App User ID to each installation. Pass this same identifier to your experiment framework at variant assignment. When purchases occur, match the App User ID from the payment event to the experiment user ID to attribute revenue to the correct variant without relying on probabilistic matching.

Calculate required sample size based on your current trial-to-paid conversion rate, expected effect size, and acceptable confidence level. A typical subscription app converting 5% of users might need 15,000 users per variant to detect a 20% relative improvement with 95% confidence. Smaller expected changes require proportionally larger samples.

Run tests until you reach statistical significance and observe at least one complete subscription cycle. If you offer a 7-day trial, run for at least 14 days after reaching your sample size to capture users who subscribe at different points in their trial. Stopping early based on positive early signals leads to false positives when day-7 conversion rates differ from day-3 rates.

Start new onboarding variants at 10-20% traffic allocation to detect technical issues or unexpected negative impacts before exposing your entire user base. Monitor crash rates, completion rates, and time-to-subscribe daily during the first week. Automatic traffic shifting based on early metrics often produces false winners, so maintain fixed allocation until statistical significance.

Tools like Spinosis handle sticky variant assignment, gradual traffic shifting to winners while preserving holdouts, and automatic attribution when App User IDs match between the experiment and RevenueCat purchases. This infrastructure lets you focus on designing better onboarding experiences rather than building experiment frameworks, though it requires budget allocation for the API service.