What Cyber Week Cohort Quality Analysis Actually Shows

Cyber Week cohort quality analysis has to start from a simple fact: a cohort acquired during Cyber Week will not look like a normal week's cohort, and the mistake most measurement teams make is comparing the two as though they should. Retention, payer conversion and early revenue per user all move during a period of unusual auction pressure and unusual player behaviour at once, which means a straight comparison against a standard baseline will flag problems that are not real and miss ones that are.

Why the baseline itself needs to change

During a CPM spike, an algorithm under pressure to hit volume targets at a worse price will often reach into different inventory than it does in a normal week, sometimes lower-quality placements, sometimes simply a different audience segment that happened to be cheaper to reach that particular week. Separately, player behaviour itself changes during a heavy shopping period: attention is split across more competing demands, first-session engagement can dip for reasons that have nothing to do with creative or targeting quality, and payer conversion in the first 48 hours sometimes lags simply because players are distracted rather than uninterested. A cohort quality read that does not account for both of these effects will attribute a temporary, expected dip entirely to acquisition quality, which leads teams to cut channels that were never actually underperforming once the holiday noise clears.

Building a Cyber Week-specific comparison

The fix is not a different metric. It is a different comparison group. Rather than measuring a Cyber Week cohort against your rolling 90-day baseline, measure it against the same week from the prior year if you have it, or against the closest comparable high-volume week your game has run, such as a previous major UA push at similar scale. This desk's Cohort Quality Scorecard for the Weekly UA Review sets out the standing metrics to track; the adjustment for a spike week is simply to swap the comparison baseline those metrics run against, not to invent new ones.

A worked example

Consider a hypothetical puzzle game that normally sees 38 percent Day 1 retention and a 4.5 percent Day 7 payer conversion rate against its rolling baseline. During Cyber Week, the same channel mix produces a cohort with 34 percent Day 1 retention and 3.8 percent Day 7 payer conversion, a result that looks like a meaningful quality drop against the standing baseline. But if the same game's Cyber Week cohort last year came in at 33 percent Day 1 retention and 3.6 percent payer conversion against that year's own rolling baseline of 37 percent and 4.3 percent respectively, the actual year-over-year comparison shows a channel performing in line with, or slightly better than, the prior year's holiday period. The standing baseline alone would have triggered a scale-back decision that the holiday-adjusted comparison shows was unnecessary.

What Cyber Week cohort quality analysis still flags

None of this means Cyber Week gets a blanket pass on quality checks. A channel whose Day 7 payer conversion falls further below its own holiday-adjusted baseline than the broader pattern would predict, or whose source and sub-source concentration narrows sharply during the spike, is still showing a genuine signal worth acting on. The distinction to hold onto is between an expected seasonal dip, which should not trigger a scaling decision, and a dip that exceeds what the seasonal pattern would predict, which should be treated with the same seriousness as any other quality flag raised outside the holiday window.

Fraud looks different during a spike week too

A high-volume week is also the week fraud and click-injection indicators are hardest to read correctly, for the same underlying reason: the normal baseline for what counts as suspicious volume shifts when every channel is scaling install counts at once. A sub-source that suddenly contributes a much larger share of installs than usual might be a fraud signal, or it might simply be a placement that performs well under the specific auction conditions Cyber Week creates. The safer approach is to hold fraud checks to the same absolute thresholds your MMP uses in a normal week rather than loosening them to account for the higher volume, and to flag anything that crosses those thresholds for manual review rather than assuming scale explains it away. A studio that quietly relaxes its fraud tolerance during the highest-spend week of the quarter is doing so at exactly the moment bad actors have the most incentive to test that tolerance.

Timing the read correctly

Cyber Week cohort quality analysis also needs patience that the rest of the business will not want to give it. The instinct after a heavy spend week is to read results within 48 to 72 hours and make a fast call, but the seasonal distraction effect on payer conversion tends to resolve over the following one to two weeks as players return to normal routines, meaning an early read during Cyber Week itself is more likely than usual to understate eventual cohort quality. Holding the scaling decision until at least the two-week mark, even under pressure to react immediately to a strong or weak early signal, produces a materially more accurate picture than a same-week verdict will.

Feeding the result back into next year's plan

The most useful output of a Cyber Week cohort quality analysis is not this year's scaling decision. It is a documented, holiday-adjusted baseline that next year's team can compare against directly, removing the guesswork of reconstructing last year's numbers under time pressure during the next spike. A studio that treats this analysis as a one-off exercise repeats the same baseline confusion every November; one that banks the comparison each year builds a genuinely useful holiday-specific reference that gets more reliable with each cycle it survives.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• Anatomy of a Q4 CPM spike
• Cohort quality versus volume: a scorecard for weekly UA reviews