Measurement debt: how studios accumulate it

Engineering teams have a vocabulary for the shortcuts they take under deadline pressure and the cost of repaying them later. UA teams take exactly the same shortcuts in their measurement, and have no word for it. The claim here is that measurement debt is the largest unmanaged liability in most mobile game studios, and that it's invisible precisely because it never appears on a dashboard. The dashboard is where the debt lives.

The counter-argument is reasonable: UA measurement is inherently approximate, so chasing precision is a poor use of time compared with buying media and making creative. True of any single decision. False in aggregate, because the approximations don't average out; they accumulate in one direction, and that direction is always towards a picture that is easier to defend than the truth.

What the debt is made of

Measurement debt is any gap between what your reporting says and what actually happened that you know about, have decided to live with, and haven't written down. The last clause matters, because a known, documented approximation is a design choice. An undocumented one is debt.

Common forms in games studios, none of them exotic:

• Event definitions that drifted. The purchase event fires on a different trigger in the iOS and Android builds because two engineers implemented it a year apart.
• SKAN conversion value schemas set at launch and never revisited as the monetisation model changed. The piece SKAN 4 and AdAttributionKit, a year in, made the case that schemas age faster than teams expect.
• Attribution windows chosen by whoever set up the MMP account, copied to every new network, and never reconciled with the actual purchase cycle.
• Network-reported costs that stopped syncing for a partner after an API change, so someone now enters spend by hand from a weekly export.
• A blended ROAS formula in a spreadsheet that quietly excludes one geography because the currency conversion broke.
• Modelled conversions from a self-attributing network mixed into the same column as deterministic ones, with no flag.

Taken alone, each item is small; somebody knows about it. Together they mean nobody in the studio can state the true return on last quarter's spend within a defensible range.

Why it accumulates rather than averaging out

The mechanism is incentive-shaped. Fixing a measurement gap almost always produces a worse number in the short term, because teams discover gaps far more often when a number looks good than when it looks bad. Nobody investigates an underperforming campaign to check whether it is secretly fine. Everybody investigates an overperforming one, finds a tracking issue, and then faces a choice about whether to correct history.

The correction waits, because the quarter is closing and the number has already gone out, and the fix needs an engineer who is on the live-ops release. The gap becomes a known quirk. Six months later a new UA manager inherits a dashboard with a dozen known quirks and no list of them.

There's a second driver: the measurement stack changes under you. MMP migrations break something. So do network sunsets. So do SDK updates, along with every platform privacy change. The migration is the clearest case. Original attribution sources and install timestamps rarely survive the move unless the studio insists on it in the contract and checks the result, and once they're gone every historical cohort comparison rests on a reconstruction. Each of those events is a moment where the studio either repays debt, by rebuilding the affected measurement properly, or borrows more by patching it and moving on.

The second-order cost: decisions made on the debt

The direct cost is inaccurate reporting. The indirect cost is that the studio's institutional memory of what works rests on that inaccurate reporting.

A creative concept that tested well in a period when a network's cost sync had failed looks cheaper than it was. It becomes the reference concept, and the team judges later concepts against it and rejects them. A geography that appeared unprofitable because of a currency bug gets deprioritised, so the studio's knowledge of that market never develops. The debt doesn't just misstate the past; it shapes what the team believes and therefore what it tries afterwards.

Which is also why measurement debt is so hard to argue for fixing. The people who would need to authorise the work have seen the numbers that the debt produced, and those numbers were, on the whole, comfortable.

A repayment framework

Treat it the way a disciplined engineering team treats technical debt: make it visible, price it, then pay a fixed proportion of capacity towards it.

Make it visible with a register. One document, owned by the measurement lead, listing every known gap, when it started, which reports it affects, plus a rough estimate of the direction and size of the error. The act of writing the list usually surprises people; a studio that believes it has three known issues typically finds fifteen.

Price each item by the spend it touches. As an illustrative example, a broken cost sync on a partner receiving a tenth of the budget is a larger liability than a mislabelled event on a network receiving one per cent, even if the second is technically worse. Rank by spend exposure, not by how annoying the bug is.

Pay a fixed share of measurement and analytics capacity, a day a fortnight say, to the top of the ranked list, regardless of what else is happening. The moment repayment becomes discretionary, it stops.

Add a rule for new borrowing. Any campaign or network launched without complete tracking gets an entry in the register on day one, with a named owner and a date; the same goes for any new event. Borrowing is allowed. Undocumented borrowing isn't.

When the bill arrives

Studios usually repay measurement debt involuntarily, all at once, at the worst moment: during an MMP migration, during acquisition due diligence, or in the quarter when a network exits and spend has to move on a week's notice. In each case someone finally reconciles the numbers, discovers the accumulated gaps, and the studio spends weeks establishing what its true return has been while decisions wait.

The studios that come through those moments quickly aren't the ones with perfect measurement. They're the ones with the register, because a documented approximation takes an afternoon to correct and an undocumented one has to be rediscovered from scratch.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• A Clean Q3 to Q4 UA Handoff, Before the Volume Hits
• In-house or agency media buying: the decision