Friday 1 March 2013

Statistics under the influence

To anyone like myself who has worked in market research, it comes as little surprise that respondents to surveys have been under-reporting the amount they drink (see here). Come to think of it, to anyone with any experience of human nature, this should come as no surprise.

When quizzed in a survey, many people provide what they feel to be acceptable answers, rather than giving their best shot at the truth. This is the case even when the interviewer is a complete stranger (or when the survey is online) and they've been assured that their answers will be treated anonymously.

The alcohol stats are just one example of this. Other surveys routinely tell us that up to 70% of the UK population give to charity. I doubt the figure is so high.

But my favourite examples are surveys exploring personal relationships. They tend to tell us that men have had up to twice as many sexual partners as women. This is in fact mathematically impossible. If two groups are "interacting", then the average number of interactions is the same in each group*. The reality is that heterosexual men and heterosexual woman have, on average, the same number of sexual partners in a lifetime. All the surveys tell us is that, true to stereotype, women as a group prefer to underplay their promiscuity, men to exaggerate it.

The lesson of all this is not that we should be sceptical of all market research. But thinking carefully about where statistics come from and what they really mean should be standard practice for us all, market researchers included.

* OK, just in case there are any genuine detail freaks reading this.... I'll admit it's possible that other interactions outside UK man + UK woman (e.g. men having relations with other men or with non-UK women) could have some effect in boosting the figures for the male population. But that couldn't possibly explain such large discrepancies in the data. Similarly, we might acknowledge that some alcohol, once purchased, never actually gets consumed. But again the effect is not big enough to explain the big gap between booze sales and what people say they drink.