Incrementality tests you can afford: geo holdouts, ghost ads, and what breaks

The problem with attribution

Attribution assigns credit under a defined measurement rule. Incrementality asks what would have happened without the intervention. Those are different questions. Time-based regression models a counterfactual using historical treatment and control data. Experimental designs and causal models require explicit assumptions about the comparison group; a dashboard attribution total alone does not identify the counterfactual. This guide explains practical design questions, rather than promising that one method or fixed test duration can certify a campaign.

Two designs sit within reach of a mid-size team, and both break in ways worth knowing before you commit budget to either.

Geo holdouts

A geo holdout splits comparable markets or regions into a test group that keeps normal paid spend and a control group where you pause or cut it, then compares organic and total install or revenue trends between the two over a fixed window. Feasibility depends on whether spend can be changed in the selected markets and whether the available data supports a consistent outcome measure. Better still, the logic survives a conversation with a finance stakeholder who has never heard the word incrementality.

What breaks it is market comparability. Two regions rarely share the same seasonal curve or the same competitive intensity, their organic baselines drift apart for reasons nobody logs, and a holdout run over too short a window will read ordinary market noise as a treatment effect. Small markets break it too. Install volumes there sit too low for the difference between test and control to separate from noise at any confidence worth reporting. Choose duration and market allocation from baseline variance, expected effect, conversion lag and a power analysis. A longer run does not repair a systematically incomparable control.

Ghost ads and PSA tests

A ghost-ad experiment records ad opportunities where the advertiser would have served an impression to a control user, while excluding that advertiser from the actual control delivery. Another eligible ad can be shown. The original Ghost Ads research, introduction and methodology distinguishes this exposure logging from public-service-announcement controls, which substitute a different creative. A blank placeholder is not the defining mechanism.

Platform support and experimental implementation determine whether a ghost-ad approach is available. Ask what the platform logs, how treatment assignment is defined and what the estimator measures. Power depends on the design and outcome variance; do not assume a universal minimum spend. A result applies to the tested population, period and treatment and does not automatically transfer to another channel.

What to actually run

Start by writing the estimand: incremental installs, revenue or contribution over a stated horizon. Check historical variance and geographic spillovers, define the minimum effect worth detecting and agree the analysis before changing spend. Pilot the data pipeline first. A geo holdout and a platform experiment answer different scoped questions; do not treat either as universal truth or prescribe a six-week run without the supporting assumptions.

Related reading

• Ad fraud tooling: the market and what it catches
• AdAttributionKit re-engagement gets sharper after WWDC

Frequently asked questions

What should a UA operator know about: Incrementality tests you can afford: geo holdouts, ghost ads, and what breaks?

Attribution assigns credit under a defined measurement rule; incrementality asks what would have happened without the intervention — different questions. Dashboards that only total attributed installs do not identify a counterfactual. Two designs sit within reach of a mid-size team, and both break in ways worth knowing before budget is committed. A geo holdout splits comparable markets into a test group that keeps normal paid spend and a control where spend is paused or cut, then compares organic and total install or revenue trends over a fixed window. Feasibility depends on whether spend can actually be changed in the selected markets and whether the data supports a consistent outcome measure; the logic also survives a finance conversation with someone who has never heard the word incrementality. What breaks it is market comparability — regions rarely share the same seasonal curve or competitive intensity, organic baselines drift for unlogged reasons, and a window that is too short reads ordinary noise as treatment. Small markets break it too: install volumes sit too low for the test–control gap to separate from noise at any confidence worth reporting. Choose duration and market allocation from baseline variance, expected effect, conversion lag and a power analysis; a longer run does not repair a systematically incomparable control. A ghost-ad experiment records opportunities where the advertiser would have served an impression to a control user while excluding that advertiser from actual control delivery — another eligible ad can show; that exposure logging is distinct from public-service-announcement controls that substitute a different creative, and a blank placeholder is not the defining mechanism. Platform support and experimental implementation decide whether a ghost-ad approach is available at all — ask what the platform logs, how treatment assignment is defined and what the estimator measures. Power depends on design and outcome variance; do not assume a universal minimum spend. A result applies to the tested population, period and treatment and does not automatically transfer to another channel.

How do we act on this in the next sprint: Incrementality tests you can afford: geo holdouts, ghost ads, and what breaks?

This sprint: write the estimand first — incremental installs, revenue or contribution over a stated horizon — before changing any spend. Check historical variance and geographic spillovers, define the minimum effect worth detecting, and agree the analysis plan before the holdout starts. Pilot the data pipeline on a dry run so the comparison can actually be produced when the window closes. Pick one design for one scoped question: a geo holdout and a platform ghost-ad experiment answer different questions, so do not treat either as universal truth or prescribe a multi-week run without the supporting assumptions. If markets cannot be matched on pre-period behaviour, do not launch a geo holdout to tick a box; fix comparability or choose another design.