Post-mortems for failed campaigns, done honestly

The purpose of a campaign post-mortem is not to explain why the campaign failed. It is to work out whether the decision to run it was wrong, and those are different questions. A well-reasoned bet can lose. A reckless one can pay off. Most UA teams write post-mortems that grade the outcome, which means they learn the wrong lessons from both.

This matters more than it used to. With the ironSource Ads network gone since the end of April and a good deal of budget re-homed onto fewer, more automated channels, teams are placing bigger bets on fewer things. The cost of learning the wrong lesson from a failed launch or a failed channel test has gone up with the size of the cheque.

Why post-mortems drift toward self-protection

Start with the incentives in the room. The UA lead who approved the campaign is usually the person running the review. The buyer who executed it is present. The creative lead whose concepts underperformed is present. The finance partner who signed off the budget wants to know it will not happen again. Every one of those people has a stake in the story that gets written down.

The path of least resistance is a narrative that blames something external and unrepeatable: a platform change, a competitor's launch, a tracking gap, an unusual month. These explanations are often partly true, which is what makes them dangerous. They close the review without anyone having to say out loud that a forecast rested on a benchmark nobody checked, or that the kill criteria got written after the spend had started.

The second drift runs toward hindsight. Once the result lands, the warning signs look obvious. Day-three retention was soft in week one; why did nobody act? The honest answer, most of the time, is that soft day-three retention in week one is common and usually recovers, so the team was right not to panic on that signal alone. A post-mortem that punishes people for not reacting to noise trains them to overreact later, which costs more than the original failure did.

Separate the decision from the outcome

The framework that fixes this is old and borrowed from poker and from clinical medicine, but UA teams rarely apply it. Lay out a two-by-two: was the decision sound given what the team knew at the time, and did the outcome meet the target. Four boxes.

• Sound decision, good outcome: earned success. Document what the reasoning was so it can be reused.
• Sound decision, bad outcome: bad luck or unmodelled risk. The learning is about the risk, not the process.
• Poor decision, good outcome: the most dangerous box. The team got lucky and will now repeat the poor process with more confidence.
• Poor decision, bad outcome: the box everyone assumes they are in, and often they are not.

To place a campaign in a box honestly, you need a record of what the team believed before spend began. Which is why the single most valuable post-mortem habit happens before the campaign: a short pre-registration note stating the target; the assumptions that make the target plausible; the signals that would show those assumptions breaking; the date by which somebody has to call a stop. Without that note, the review is an argument about memory.

A worked illustration

Take an illustrative case. A mid-core studio budgets a six-week channel test on a new DSP at a level that would be material to its quarter. The pre-registration note says: we expect cost per payer to land within 20 percent of our blended figure by week four, because the DSP's own case studies in our genre suggest it and because our creative library is mature. The stop rule is a week-four cost per payer more than 40 percent above blended.

Week four arrives at 55 percent above blended and the team stops the campaign. Outcome: bad. The decision question is the harder one, and here at least the assumptions sat on paper in advance. One of them, the idea that vendor case studies in the genre made a reasonable prior, was weak; the team knew that at the time but had nothing better. The other, that a mature creative library would transfer, looked reasonable then. It failed for a specific reason: the DSP's inventory skewed to formats where the library was thin.

The honest classification is a partially sound decision with a bad outcome, and the learning is precise: audit format mix against creative inventory before any channel test, and discount vendor case studies to near zero as a prior. Neither of those learnings would surface from a review that concluded the DSP did not work for us.

The second-order effect nobody budgets for

There is a trade-off most guidance misses. Honest post-mortems make teams more willing to take defensible risks, because a sound decision that loses still goes down as sound, which is the point. They also make those teams slower to abandon a process that is failing for reasons the framework cannot see, because every individual loss gets filed under bad luck.

Guard against this with a rule at the portfolio level rather than the campaign level. If the same team classifies three consecutive failures as sound decisions with bad outcomes, the fourth review must be run by someone outside the team, and the question shifts from whether each decision was sound to whether the team's model of what makes a decision sound is itself broken. Runs of bad luck happen. Runs of four are usually a model problem.

What to write down, and who reads it

Keep the document short and shaped the same way every time, so reviews still line up a year later. Five sections: what we set out to do; what we believed, with the reasons; what the stop rule was; what happened against each assumption; which box it lands in, plus one paragraph of justification. Attach the pre-registration note unchanged.

Distribution is the part teams get wrong. If post-mortems only go upward to the person who owns the budget, they become defensive documents. If they go sideways to the other UA and creative teams in the studio, they become useful ones, and the author writes them differently knowing that. The most valuable reader of a failed 4X launch review is the puzzle team about to run a similar test with a different vendor's case studies as their prior.

One last discipline. Run the same review on the campaigns that succeeded, at least once a quarter. The box that rots a UA team from the inside is the poor decision with the good outcome, and you will never find it by only reviewing losses.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• The weekly UA review is where money is lost
• Black Friday Mobile Game CPMs: What to Budget For Now