The creative testing statistics most teams get wrong

Most creative tests in mobile UA are not underpowered. They are misread. Teams spend real budget generating enough impressions per variant to see a difference, then throw the result away by picking the top variant from a table of ten, declaring it the winner because its CPI is lowest, and being surprised when it performs like the average in production. The test was fine. The reading was the problem.

The argument here is that the statistical mistakes in creative testing are mostly mistakes of selection and framing, not of sample size, and that fixing them costs nothing in media spend. Some practitioners will push back that the platforms' own algorithms handle this. They do not. The auction optimises delivery, not inference.

Mistake one: the winner is the top of a ranked list

Rank ten variants by CPI and the one at the top is partly good and partly lucky. The expected performance of the best-looking option in any ranked set is worse than its observed performance, and the gap grows with the number of options. Statisticians call this the winner's curse. Creative teams call it "the winner did not scale".

The mechanism is simple. Each variant's observed CPI is its true CPI plus noise. Selecting the minimum selects for favourable noise. With three variants the effect is mild. With fifteen, and the volume of variants reported by the large advertisers in AppsFlyer's State of Gaming for Marketers 2026 suggests many teams are far beyond fifteen per month, the top slot is frequently occupied by an average asset that had a good week.

The fix is not more impressions. It is a second, smaller confirmation run for the top two or three, read separately from the exploration round. Treat the first round as a filter and the second as the measurement. A team that adopts this will find fewer winners and keep more of them.

Mistake two: reading significance on the wrong metric

Creative tests are usually judged on install-side metrics because those arrive fast and in volume. The decision they inform is a revenue decision. This is a substitution, and the significance calculation inherits the substitution.

A variant can be significantly better on CPI and worse on day-seven payer rate, and the second effect is what matters. It is also the one with the smaller sample and the longer delay, which is why teams do not wait for it. The honest position is that most creative tests are conclusive about clicks and installs and inconclusive about value, and a decision rule should say so rather than pretend the install verdict settles the matter.

A workable rule: promote on install metrics, but cap the promoted variant's share of spend until an early value signal, such as day-three retention or first-purchase rate, has cleared a pre-agreed floor. The floor does not need to be significant. It needs to be a tripwire that stops a cheap-install variant from consuming the budget before its cohort quality is known.

Mistake three: peeking, then stopping when it looks good

Dashboards refresh hourly. A test that is checked ten times before its planned end has ten chances to cross a significance threshold by chance, and the person watching will stop it at the first crossing. The nominal five per cent false-positive rate becomes something much larger, and the team never knows because the stopped tests all look like wins.

Two habits close the hole. Decide the end condition before the test starts, either a date or an impression count, and write it down where the team can see it. Then, if early stopping is genuinely needed for budget reasons, use a stricter threshold for early looks than for the final read. The exact adjustment matters less than the discipline of having one.

The second-order effect worth noticing is that peeking bias is worse for creative than for most experiments because creative performance drifts over its life. An early crossing is often a novelty effect, and stopping at the crossing locks in a verdict taken at the variant's best moment. The confirmation run from mistake one is also the cure here.

Mistake four: treating variants as independent when they share a parent

Ten variants cut from the same hook with different end cards are not ten independent tests. They share most of their content and therefore most of their noise. A concept that gets lucky with the audience on a given week lifts all its children together. Reading the table, the team sees a cluster of good numbers and concludes the concept is strong. Sometimes it is. Sometimes the cluster is one lucky draw appearing ten times.

Group variants by parent concept before ranking. Compare concepts on their pooled numbers, then compare executions inside the winning concept. This is a two-level read, and it aligns with what the creative team actually decides: which idea to invest in, then which cut to run. UA Ledger's earlier piece "Concept art is not a creative test" made the production side of this argument. The measurement side is the same: the concept is the unit that generalises, and the execution is the unit that gets lucky.

A worked illustration

Take an illustrative team running twelve variants across three concepts, four cuts each, for one week at equal budget. The raw table shows a spread of CPI from a low around two thirds of the mean to a high around one and a third of it. The top variant belongs to concept B.

Pooled by concept, A and B are close and C is clearly behind. Inside B, the four cuts are within a range that the week's volume cannot separate. The correct conclusions are: drop C, keep A and B alive, and run a confirmation week on the two best cuts from each of A and B at a slightly higher budget with a fixed end date. What the team would normally do is scale the single top variant from B and drop everything else, which throws away A on the strength of noise and bets the account on one lucky cut.

The confirmation week costs a fraction of the exploration week and typically changes the decision one time in three. That is the value of reading the statistics properly: not more certainty, but fewer confident mistakes.

The last thing to add to the test log is the number of variants that were compared to produce each winner. A winner from a field of three and a winner from a field of thirty are different objects, and the field size should travel with the result into the production decision.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• Writing creative briefs an agency can execute
• Anatomy of a network test that lied