Cohort quality checks before you scale a channel

A UA team runs a test budget on a new channel or a new creative concept, watches the early numbers, and eventually decides whether to move from test spend to scale spend. That decision point, not the weekly review that follows it, is where a bad cohort does the most damage, because scaling multiplies whatever quality problem was already present in the test cohort. A gate checklist applied specifically at that moment, separate from the ongoing weekly scorecard a team already runs, catches problems before they get expensive rather than after.

What the gate actually checks

Session length in the first few sessions is the fastest available signal and the easiest to misread on its own. A short first session is normal for many casual genres and alarming for a mid-core title with a longer onboarding. The useful comparison is not an absolute threshold but session length against the studio's own established baseline for that genre and channel, since a channel producing sessions half the length of an equivalent Meta or Google cohort is very likely producing users who installed for the wrong reason.

Tutorial completion is a cleaner signal than session length because it ties directly to whether a user engaged with the core loop rather than simply opening the app. A cohort with markedly lower tutorial completion than the studio's baseline usually indicates a creative-audience mismatch, an ad promising a different game to the one that installs, rather than a channel problem in the technical sense.

Day-three retention is the point at which most of the false positives from the first two signals resolve themselves. A cohort that survives to day three with retention close to baseline has usually cleared the curiosity-click risk that shows up in session length and tutorial completion. A cohort that looked fine on day one and falls sharply by day three is the classic pattern of a channel that is good at getting an install and bad at getting a player who stays.

Payer rate at day seven is the metric finance actually cares about, and it is also the one most vulnerable to small sample noise at the test-budget stage. A test cohort in the low hundreds can show a payer rate that looks strong or weak almost entirely by chance. The gate should treat a day-seven payer rate from a small test cohort as directional, not conclusive, and should specify in advance how large the cohort needs to be before that number is allowed to drive the scale decision on its own.

Fraud flags belong in the same gate rather than in a separate compliance process, because a channel producing a cohort that passes every quality signal above but shows an unusually high proportion of flagged installs, device ID anomalies, install-to-open times too fast to be plausible, is not a channel to scale regardless of how the revenue numbers read, since a fraud problem tends to worsen rather than resolve once spend increases.

Making the gate a real decision point

The value of formalising these five checks as a gate, rather than leaving them scattered across a weekly dashboard, is that it forces a single explicit decision moment: pass all five at the agreed thresholds and move to scale budget, fail any one and either extend the test at the current spend level or kill the channel outright. Teams that skip the formal gate tend to scale on the strength of whichever metric looked best that week, which is usually cost per install or installs per mille, precisely the two metrics least correlated with whether the cohort will still be worth anything a month later.

The gate does not replace the ongoing cohort quality review a team runs once a channel is live at scale. It exists specifically to prevent a channel from reaching that ongoing review with a quality problem already baked into a larger, more expensive cohort. A channel that fails the gate and gets scaled anyway under budget pressure rarely improves once spend increases; the same audience mismatch or fraud exposure that showed up in the test cohort simply repeats at a larger, costlier size.

Who owns the decision

A gate is only as reliable as the discipline that enforces it, which is why the decision at each stage should sit with a named owner rather than emerge informally from a group chat once the numbers look decent. A media buyer under pressure to hit a spend target has an obvious incentive to read a borderline result generously. Giving the actual scale decision to whoever also owns the cohort quality review, rather than to the person closest to the spend target, keeps the gate a genuine check rather than a formality that gets waved through whenever budget needs a new place to go.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• A cohort quality checklist before you scale UA spend
• How the mobile UA funnel works, step by step