Why players lie in surveys, and what their session data says instead

Ask a playtester whether they enjoyed a level and most will say yes. Saying no means justifying a negative opinion to a stranger who is running the session and writing things down, and that social cost runs a good deal higher than most survey designs account for. Stated preference data from surveys and focus sessions isn't useless. It measures something narrower than most teams think: what a player is willing to say out loud, rather than what a player actually did.

Session data versus survey data: not dishonesty, a measurement mismatch

Players aren't lying in any deliberate sense. They answer the question in front of them, which is usually some version of did you like this, filtered through politeness and a mild wish to be a useful participant, and filtered again through how little any of us can see of our own moment-to-moment engagement. Revealed preference carries none of those filters. Drop-off points, retry rates, seconds on screen before a quit, the number of times someone reopens the app: session data records behaviour instead of a reported opinion about behaviour, and those are different objects.

Where the two diverge most often

The clearest divergence sits around difficulty. Players rate levels they eventually beat as fun and fair in post-session surveys, even when the log shows six or seven failed attempts before the win. The stated verdict is about that win, the part they remember and the part they're happy to talk about. Session data keeps the frustration in between, and the frustration in between is what decides whether a real player, with no moderator watching and no obligation to finish, quits at attempt three rather than grinding on to attempt seven.

Pacing goes the same way: testers describe an onboarding sequence as quick, then the session data shows a completion time well past what those same testers would sit through in a live install funnel, because a lab session comes with an implicit agreement to see the thing out that a real install never carries.

What to trust instead

Retention curves, and specifically the shape of the drop-off across the first few sessions, beat any stated satisfaction score as a proxy for genuine engagement, because a player who comes back unprompted has revealed something a survey cannot reach. They wanted to return. Time-to-first-failure and time-to-first-quit inside a session run more honest than a post-session rating, since both land in the log before the player has had any chance to reframe the experience into something more flattering to describe.

Using both without letting one override the other

None of this makes surveys and moderated sessions worthless. They surface things session data cannot see on its own: confusion about what a mechanic is meant to do, a control scheme that feels wrong even after the player adapts to it, an art style that reads badly no matter how long someone plays. The mistake is treating a stated preference score as though it settles a question only behavioural data can answer. Difficulty and pacing are where that gap between what people say and what they do runs widest, and where it stays widest across studies.

A creative or design decision resting on a survey score alone, with no session data check on drop-off and retry behaviour around the same moment, rests on the version of the experience players were willing to describe rather than the version they actually had. The two are related. Treating them as interchangeable is where testing programmes go wrong most often.

Related archive reading

These articles provide related context and remain subject to their stated review status.

• First-party data is overrated for most studios
• Value optimisation selects for what you can measure