RevenueCat’s paywall comparison is not an A/B test

RevenueCat reports download-to-paid conversion of 10.7% for hard-paywall apps and 2.1% for freemium apps. The report’s metric uses a 35-day horizon.Report and definitions.

Ask what changes with the access model

Apps choosing different paywalls may also differ in audience, price, product category and the promise made before installation. A comparison across those apps does not isolate the effect of putting a paywall into one game. The gap can be informative without being a causal estimate.

Our recommendation is to write the proposed intervention precisely. Is the team changing access, price, trial availability, onboarding or all of them together? A test that changes several components may answer a commercial package question, but it will not identify the separate effect of each component.

Keep the downstream outcomes visible

Use conversion alongside play, refunds and retained value.GameAnalytics’ retention tool illustrates why start and return events need explicit definitions when examining behaviour after a product event. A payer-conversion measure and a later engagement measure answer different questions.

Before borrowing the headline as a target, determine whether the game’s business model even resembles the sampled subscription apps. An ad-supported game could lose substantial access to its audience under a hard paywall. That possibility must be modelled and tested with its own data; the report cannot price the trade-off for it.

Inspect the evidence

Download the source-population and comparability card. The extraction separates documented facts from our analysis and unknowns.

Further reading in the existing archive: The benchmark report is selling you something; The hook is a promise; retention is the invoice. These links provide background; this check does not independently verify their full contents.

Frequently asked questions

How should UA teams read RevenueCat’s paywall comparison when it is not an A/B test?

A cross-app conversion gap can suggest a hypothesis without proving the effect of changing your game. RevenueCat reports download-to-paid conversion of 10.7% for hard-paywall apps and 2.1% for freemium apps; the report’s metric uses a 35-day horizon. Apps choosing different paywalls may also differ in audience, price, product category and the promise made before installation — a comparison across those apps does not isolate the effect of putting a paywall into one game. The gap can be informative without being a causal estimate. Write the proposed intervention precisely: is the team changing access, price, trial availability, onboarding or all of them together? A test that changes several components may answer a commercial package question, but it will not identify the separate effect of each component. Use conversion alongside play, refunds and retained value; a payer-conversion measure and a later engagement measure answer different questions. Before borrowing the headline as a target, determine whether the game’s business model even resembles the sampled subscription apps. An ad-supported game could lose substantial access to its audience under a hard paywall — that possibility must be modelled and tested with its own data; the report cannot price the trade-off for it.