Hyper Hippo: a returning-player comparison is not a remarketing holdout
Hyper Hippo's retention headline becomes more useful when its comparison group is kept attached. Returning players and newly acquired players answer different business questions.
Hyper Hippo's retention headline becomes more useful when its comparison group is kept attached. Returning players and newly acquired players answer different business questions.
What the public case establishes
AppsFlyer's Hyper Hippo case says remarketing for AdVenture Communist and AdVenture Capitalist showed higher Day-7 retention than newly acquired users during the same period. The account also describes audience segmentation and cost data exported into internal BI. It does not compare remarketed users with equivalent lapsed users withheld from ads. AppsFlyer case study.
This debrief analyses a public customer case published by a commercial supplier. It is not an original UA Ledger interview. The chronology below follows the supplier’s account; we did not inspect the advertiser’s dashboards or interview its staff.
The decision worth examining
A person who already knows a game differs from someone encountering it for the first time. Higher retention in the returning group can reflect prior familiarity, past engagement or the selection of eligible users. It may be commercially useful, but it cannot alone establish what remarketing caused. The missing comparator is the plausible outcome for similarly lapsed users without the intervention.
The worksheet therefore has two separate questions: how the reactivated cohort behaves and whether the campaign caused additional return. The first is descriptive and can inform product handling. The second needs an experiment or a defensible identification strategy. Combining them into one retention uplift statement hides both the selection process and the action the team actually wants to justify.
A usable next check
Define lapsed-user eligibility before campaign delivery and document exclusions for recently active players. Where feasible, assign comparable eligible units to campaign and holdout conditions. Measure return and subsequent value using the same clock and revenue rules. Keep the comparison with new users as context, clearly labelled, rather than substituting it for the causal readout.
| Test element | Proposed specification |
|---|---|
| Control | Comparable lapsed users without the remarketing intervention |
| Variant | Eligible lapsed users receiving the intervention |
| Readout | Incremental return and value, with descriptive new-user comparisons separate |
| Stop or reject | Reject a causal claim based only on returning-versus-new-player retention |
Download the populated case worksheet. Its control, variant and decision rules are a proposed experiment, not an account of results already measured. Keep the source URL with the worksheet when circulating it so reported findings cannot become unattributed team benchmarks.
Evidence boundary
The source's comparison is meaningful only within its stated populations. We did not inspect the audience definitions, allocation or underlying cohort data, and no claim is made that the original campaign lacked other analysis elsewhere.
For background, see reading an mmp dashboard without fooling yourself and cohort quality checklist ua spend. No first-hand campaign result is asserted.
