A cohort quality checklist before you scale UA spend

A UA manager who scales spend on a campaign because CPI came in under target is answering the wrong question. CPI tells you what a cohort cost to acquire. It says nothing about whether that cohort will retain, or convert, or pay back inside a timeframe the business can tolerate. Scaling on CPI alone is how a studio ends up doubling spend into a channel that is quietly recruiting the wrong players, and does not find out until the cohort has had six weeks to prove it.

Why the mistake compounds

The reason this particular error is expensive rather than merely wrong is timing. A CPI signal lands within hours of a campaign going live. A genuine read on cohort quality, meaning retention, payer conversion, early revenue per user against a comparable benchmark, takes two to six weeks depending on the game's monetisation cycle. A team under pressure to hit a spend target will scale on the fast signal because the slow one is not ready yet, and by the time the slow signal arrives showing the cohort underperforms, the studio has already committed several more weeks of budget to the same channel and the same creative. Cohort quality problems do not show up as a spike. They show up as a slow, hard-to-diagnose decline in blended LTV that gets misattributed to the game or the season or the market, when the actual cause was a scaling decision made on incomplete information a month earlier.

As we set out in A Cohort Quality Scorecard for the Weekly UA Review, the fix is not more data. It is a small set of checks applied consistently before a scaling decision, rather than a large dashboard reviewed occasionally.

The checklist before you scale

Run this before increasing budget on any campaign or channel or creative that has cleared its CPI target:

• Day 1 and Day 7 retention against your game's own historical baseline, not an industry benchmark. A cohort that retains two points below your own historical average is a quality signal, even if it beats a published benchmark from a different genre.
• Payer conversion rate by day 7, compared to the blended average across your other live channels. A channel converting payers at half the blended rate is recruiting a cheaper but structurally different audience, and CPI alone will never reveal that.
• Early ARPDAU trend across the first 14 days, watching for a cohort that starts strong and decays faster than your baseline cohort, which usually indicates a creative or targeting mismatch rather than a genuine engaged audience.
• Creative-to-product expectation gap, meaning a manual check of whether the ad creative that drove the cohort promised a game experience the actual product delivers in its first session. Mismatched expectations show up in cohort quality before they show up in reviews.
• Fraud and click-injection indicators from your MMP, particularly for a channel scaling unusually fast, since inflated installs at attractive CPIs are one of the most common causes of a cohort quality collapse that looks like a targeting problem.
• Source and sub-source concentration, checking whether the volume is coming from a narrow slice of placements or publishers rather than a broad, diversified mix, since concentrated volume is more exposed to a single bad-actor placement.

Reading the signals together

No single item on that list should trigger a scaling decision on its own. A channel with slightly soft Day 7 retention but strong payer conversion and clean fraud indicators is usually fine to scale cautiously; a channel with excellent retention but a narrow sub-source concentration deserves a pause no matter how good the other numbers look, because concentration risk tends to surface later as a fraud or policy problem rather than a quality one. The checklist works as a set of cross-checks, not as a scorecard with a pass threshold. Cohort quality failures rarely show up in every metric at once.

Building the checklist into the approval workflow

A checklist that only exists as a document nobody opens under deadline pressure will not change behaviour. It needs an owner and a trigger point: whoever approves a budget increase request should require the checklist result attached to that request, not reviewed afterward as part of a monthly retrospective. Make the checklist a mandatory field in the scaling approval process rather than a best practice you encourage teams to follow, and you have a genuine gate instead of a document someone opens only after a scaling decision has already gone wrong.

It is also worth being honest about the checklist's limits. It catches quality problems that surface within two to three weeks of a cohort's acquisition. Retention, payer mix, fraud: most of what goes wrong lands in that window. It will not catch a quality problem that only emerges after 60 or 90 days, such as a cohort that retains and pays normally at first but churns unusually fast once past a game's first content update. Longer-horizon quality issues need their own slower review cadence layered on top of this checklist, not a replacement for it.

What to do when the checklist fails

A channel that fails two or more checks should not be cut immediately, particularly if CPI is genuinely strong. The correct next step is a controlled scale-back: reduce budget by a third while holding creative and targeting constant, then re-run the checklist against the smaller, fresher cohort two weeks later. That isolates whether the quality problem was a function of scale, meaning the algorithm reached into lower-quality inventory to hit volume, or a structural problem with the channel or creative that will persist at any spend level. Cutting a channel outright on the first sign of trouble wastes the diagnostic value of a controlled pull-back. Re-scaling straight after one clean checklist pass repeats the original mistake with slightly better information.

Building this into the weekly review

Teams that treat cohort quality as a scaling gate rather than a retrospective report tend to catch problems while they are still cheap to fix, which is the entire point of running the checklist before a spend increase rather than after one has already gone out the door.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• A Cohort Quality Scorecard for the Weekly UA Review
• Cohort quality checks before you scale a channel