A Cohort Quality Scorecard for the Weekly UA Review

Most weekly UA reviews open with CPI and close with blended ROAS, and everything discussed in between is really just an argument about whether the number in the middle went up or down. That format misses the question that actually predicts whether a channel is worth scaling: is the cohort arriving this week the same quality as the cohort that arrived last month, or has a channel started buying volume by lowering the bar on who counts as a user.

A cohort quality scorecard is not a replacement for CPI or ROAS. It is the check that stops a team from scaling a channel that is quietly degrading, three weeks before the degradation shows up in blended revenue.

Why CPI and D1 alone miss the drift

CPI answers what a user costs. D1 retention answers whether a user opened the app twice. Neither answers whether the user who opened it twice is the kind of user who converts to a payer, refers a friend, or plays long enough to be worth the media spent acquiring them. A channel can hold CPI flat and D1 retention flat while shifting its delivery toward an audience segment with materially worse downstream value, and a dashboard built only on those two metrics will not flag it until the revenue gap shows up weeks later, by which point the budget has already scaled into the problem.

This is a known failure mode with broad-match and automated bidding products, including Meta's Advantage+ app campaigns and Google's Performance Max for apps, both of which optimise toward the install or event target a team gives them without any inherent preference for the kind of user who retains. The automation is not doing anything wrong. It is doing exactly what it was told, which is why the check has to sit outside the platform's own reporting.

The five-line scorecard

Run this alongside CPI and ROAS in the weekly review, per channel, per week:

• D7 retention as a ratio to the account's trailing 90-day average, not an absolute number. A channel sitting persistently below its own historical baseline is degrading even if the absolute figure still looks acceptable next to industry benchmarks.
• Day 7 ARPDAU for the cohort against the same trailing baseline. This catches quality drift that retention alone misses, since a cohort can retain at a normal rate while spending at a below-normal rate.
• Share of cohort reaching a defined core-loop milestone (for a mid-core game, typically a first meaningful progression event; for hyper-casual, a session-count threshold) by day 3. A falling share here usually means creative or targeting is pulling in users who were never a fit for the mechanic.
• Payer conversion rate by day 14 against baseline. This is the slowest signal to arrive and the hardest to fake, which makes it the tie-breaker when the faster signals disagree.
• Refund and chargeback rate in the first 30 days, where applicable. A channel with clean CPI, retention and ARPDAU numbers but a rising refund rate is often buying users through misleading creative, a problem that shows up here before it shows up anywhere else.

A worked example

Take a hypothetical mid-core strategy title running two channels at similar CPI, around $4.20 for both, with D7 retention also close, 13% and 12% respectively. On CPI and retention alone, the channels look interchangeable. Running the scorecard, Channel A's day 7 ARPDAU sits at 94% of the account baseline and its day 14 payer conversion is flat against trailing average. Channel B's day 7 ARPDAU sits at 61% of baseline, and its core-loop milestone completion by day 3 has fallen from a trailing average of 48% to 34% over the past three weeks. Blended ROAS has not yet caught the difference because Channel B's volume is still small relative to the account, but the scorecard shows the direction clearly: Channel B is heading toward a quality problem that will show up in aggregate revenue in another month if spend keeps scaling on the current creative and targeting mix.

What to do when the scorecard disagrees with CPI

The scorecard is a leading indicator, not a verdict. A channel that scores poorly on one line but well on the other four is not automatically a channel to cut; it is a channel to investigate, usually by isolating the specific creative or audience segment driving the weak line before making an account-level decision. A channel that scores poorly on three or more lines simultaneously, as in the Channel B example above, is a stronger case for pausing scale until the cause is identified, since a compounding quality problem across multiple metrics rarely resolves on its own.

The habit worth building is treating the scorecard as a standing agenda item, not a diagnostic reserved for when something already looks wrong. By the time blended ROAS moves enough to trigger a review, a degrading channel has usually already absorbed several weeks of budget it should not have received. A five-line check that takes ten minutes in a weekly meeting is cheap insurance against that lag, and it is the difference between catching a channel's decline in week three and explaining it to a studio head in week eight.

The scorecard also has a quieter benefit beyond catching decay: it gives a media buyer language for defending a channel that looks weak on CPI alone but strong on the underlying cohort. A channel with a slightly higher CPI than its peers but a materially better day 14 payer conversion rate is not a channel to cut on a first glance at the top-line number, and a scorecard that surfaces that trade-off in the same weekly review as the CPI figure stops a good channel from getting cut for the wrong reason.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• A cohort quality checklist before you scale UA spend
• Cohort quality versus volume: a scorecard for weekly UA reviews