More creative testing can make you worse

There is a volume of creative testing beyond which each additional test makes a team's decisions worse, not better. Most studios that have adopted high-velocity testing are past that point and do not know it, because the symptom looks like success: more winners declared every week.

This is not an argument against testing. It is an argument that testing has a capacity constraint set by spend, by the noise in the signal and by the team's ability to understand what it learned, and that the industry has spent two years pushing volume without ever asking where the constraint sits.

The statistical mechanism

Every creative test is a comparison against a control with some threshold for declaring a winner. Every comparison carries a chance of a false positive: a variant that looks better by luck. That chance does not shrink because you run more tests. It compounds.

Suppose an illustrative team tests 200 concepts a month with a threshold that gives each a one-in-twenty chance of a false winner. Ten false winners a month enter the portfolio. If the true rate of genuinely better concepts is around 3%, which is generous, six real winners arrive alongside them. More than half of what the team ships as "proven" is noise. Double the test volume at the same spend and each test gets half the impressions, the threshold gets easier to cross by chance, and the ratio gets worse.

The impressions problem is the one that bites hardest. Spend is roughly fixed. Tests divide it. At 50 tests a month, each concept might reach a sample where a 15% difference is detectable. At 300, most concepts are being judged on a sample where only a 60% difference would be reliable, and nobody's variant is 60% better. So the team either declares nothing, which feels like failure, or lowers the bar, which manufactures winners.

There is a second effect that compounds the first. Platforms' automated delivery decides which variants get impressions, and it starves the ones that start slowly. A concept that would have won given a fair sample never gets one. High volume increases the number of concepts judged on a few hundred impressions by an algorithm that had already given up on them.

The strategic damage

If the only cost were wasted production, high volume would be defensible. The larger cost is what false winners do to the creative strategy.

Each declared winner becomes a data point in the team's model of what works. Ten noise winners a month teach the team ten false lessons. Over a year that is a strategy built substantially on coincidence. The team develops confident beliefs about hooks, colour palettes and opening seconds that were never true, and briefs the next quarter's production against them.

Worse, the practice selects for a particular kind of concept. Small variations on a proven ad are cheap to make and just as likely to produce a false winner as a genuinely new idea. So a high-volume programme fills itself with variants of variants. The portfolio narrows while the team believes it is exploring. UA Ledger's February piece "Concept art is not a creative test" made a neighbouring point about what counts as a test at all; the argument here is that even proper tests, in excess, corrode judgement.

And there is fatigue. Shipping ten false winners a month into live campaigns means the audience sees more near-identical ads faster, which accelerates the fatigue of the real winners sitting beside them.

The trade-off most coverage skips

Reducing test volume has a real cost, and honesty requires saying so. Fewer tests means slower discovery of genuine breakthroughs, and in a category where a new hook can reset a studio's CPI for a quarter, speed of discovery matters. The teams with the largest budgets can run high volume without hitting the constraint because each test still gets a clean sample. Advice to test less is really advice to test at the volume your spend can support, and that is a smaller number than most teams are running.

The other cost is cultural. A creative team measured on output will resist a programme that ships less. The measure has to change with the volume.

A decision rule for setting volume

A workable framework starts from spend and works back to a number of tests, rather than starting from a desired number of tests and dividing spend.

• Decide the smallest improvement over control that would be worth acting on. For most teams this is somewhere between 15% and 25% on the metric that matters, usually cost per early payer or cost per retained player, not click-through rate.
• Work out the impressions each variant needs to detect that difference with reasonable confidence. Any analyst can do this; the answer is usually larger than the team expects.
• Divide the monthly test budget by that number. The result is the maximum tests per month the programme can support. It is a ceiling, not a target.
• Reserve a fixed share of that capacity, perhaps a quarter, for concepts that are genuinely different from anything in the portfolio, and protect that share from the pressure to iterate on winners.

The rule for declaring winners is then simple: no variant is promoted to the portfolio on the basis of a test that did not reach its required sample, regardless of how good the early numbers look.

An illustrative example

Consider an illustrative hybrid-casual studio testing 180 concepts a month, declaring about 25 winners, and puzzled that portfolio CPI keeps drifting up despite a steady stream of new "proven" creative.

Applying the framework, the team finds its spend supports about 40 properly powered tests a month. It cuts production to 40, with 10 reserved for new directions. Declared winners fall to around five a month. Over the following quarter the portfolio becomes more stable, the winners hold longer before fatigue, and the team's briefs get sharper because they are built on five reliable lessons instead of 25 mixed with noise.

The production budget saved is not returned to finance. It goes into making the 40 concepts better and into holdout measurement of whether the winners are incremental, which is the question a creative test cannot answer no matter how many of them you run.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• Why match-3 creative has hit a saturation point
• Dashboards make UA teams slower