Incrementality results do not transfer between channels

A common shortcut in UA measurement runs like this. The team runs a geo holdout on its largest channel, finds that attributed installs overstate incremental installs by some factor, and then applies that factor as a haircut across every channel on the plan. The result is a blended incremental ROAS that looks disciplined and is close to meaningless. Incrementality isn't a property of a channel. It's a property of the interaction between a channel, the rest of the media mix, and the organic demand for the game at the moment of the test; move any of the three and the number moves with them.

Practitioners who have run several tests know this in their bones and still apply the haircut, because the alternative is testing everything, and testing everything is expensive. The point of this piece is that the haircut isn't a cheap approximation. It's a wrong answer with a false precision attached, and there is a cheaper way to be approximately right.

The mechanism: three things a lift number is actually measuring

When a holdout shows that a channel drove an incremental lift of some size, that number is the sum of three things.

The first is genuinely new demand: players who would not have installed without the exposure. The second is the channel's position relative to the organic baseline. A channel that reaches people already searching for the game, or already exposed through creators or word of mouth, converts demand that existed and takes credit for it. The third is cross-channel cannibalisation. When the team holds out the test channel, other channels on the plan pick up some of the installs the test channel would have claimed, and their attributed numbers rise. The lift measured for the test channel is net of what its competitors absorbed.

Each of these varies by channel in ways that have nothing to do with the channel's quality. A rewarded video network and a social platform sit in completely different places relative to organic demand. A retargeting programme sits in a different place again, which is why an earlier UA Ledger piece, Is your retargeting incremental?, had to be a separate question rather than a footnote. Carrying a multiplier from one to another assumes the three components are in the same proportion, and there's no reason they would be.

The second-order effect: cutting one channel changes the others' numbers

The consequence most coverage misses is that incrementality results aren't even stable for the channel that produced them. They're stable only while the mix around that channel is stable.

Suppose a team tests channel A, finds it weakly incremental, and cuts its budget by half. Channel B, which was previously absorbing some of A's cannibalised installs, now has less to absorb. B's attributed installs fall, and if the team tests B a month later, its measured lift will be higher than it would have been before, because there is less overlap to net out. The team concludes B is strong and scales it, whereupon B's incrementality falls as it starts to reach the audience A used to cover.

None of this is a measurement error. It's the mix behaving as a system. But it means a channel-by-channel test programme that runs sequentially, changing budgets after each result, is measuring a moving target and can end up chasing its own tail. The direction of travel makes this worse. As paid share of installs grows, which every MMP dataset has shown for years, the organic baseline shrinks relative to paid, overlap between paid channels increases, and the interaction problem compounds rather than fading.

A framework: incrementality as a function, not a coefficient

The practical fix is to stop recording incrementality as a number attached to a channel and start recording it as a function of four inputs: the channel, its spend level, the mix around it, and the period. Every test result gets filed with all four. A result is only reused when all four are close to the conditions of the decision in front of the team.

That sounds like it makes reuse rare. It does; that's the correct outcome, and what it buys in exchange is a cheaper structure of tests.

• Anchor with a blended holdout. Once or twice a year, hold out all paid media in a small set of geos. This measures total paid incrementality with no cannibalisation problem, because there is nothing left to cannibalise.
• Treat that anchor as the only number safe to use as a portfolio-level haircut.
• Test the largest two or three channels individually, in the mix they actually run in, at the spend they actually run at. Don't test a channel at a token budget and scale the answer.
• For every other channel, use the blended anchor and the channel's attribution sensitivity as a proxy. A channel whose attributed volume swings sharply when the attribution window changes is more likely to be claiming existing demand than one whose volume is stable. That isn't a measurement of incrementality, but it's a useful ranking of who to test next.

A decision rule for expiry: a channel test result is stale when that channel's share of spend has moved by more than a set fraction, when the total paid share of installs has moved materially, or when the team has added or cut a channel adjacent to it in audience. Any of the three triggers a retest before the result goes into a budget decision.

An illustrative example

Take a hypothetical portfolio spending across a social platform, a rewarded video network and a DSP. A geo holdout on the social platform, run at full budget in the normal mix, shows an illustrative lift of 60% of attributed installs. The temptation is to apply 60% across the plan.

Instead, the blended holdout from the previous quarter showed that paid media overall was driving an illustrative 75% of attributed installs incrementally. That is the portfolio haircut. The social platform's 60% goes on file with its spend level and mix. The rewarded network, never tested, does not get assigned 60%. Its attribution sensitivity check shows its volume halves when the window shortens, which flags it for testing first, and the working assumption until then is the blended 75% with a note that it is probably generous for this channel.

The plan now carries three different confidence levels, visible to finance, rather than one false coefficient. The rewarded network goes to the front of the test queue, and its result will go on file against the mix that produced it, because by then the social platform's budget will have moved and the old number will already be approaching its expiry.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• AppLovin +66%, Unity −25%: two ad networks, one buyer's market
• Digital Turbine's On-Device Preload UA Channel in 2025