We are often more certain than the evidence really allows.
More quotes“Big data” has done marvellous work persuading executives that bias disappears once the file becomes too large to email.
Gallup’s soup story punctures the fantasy. A spoonful can tell you about the pot, provided the pot was stirred. Take a bucket from the greasy top and volume merely gives you greater confidence about the wrong layer.
Twenty carefully chosen conversations may reveal more than 20,000 survey clicks from enthusiasts who volunteered. A random set of support tickets can beat a lovingly curated parade of spectacular complaints. A modest poll may represent a country better than a million votes collected from visitors to one excitable website.
Row count is seductive because it looks like an asset. It has weight, infrastructure and specialist vocabulary. Representation is quieter. It asks the embarrassing question: who never had a chance to appear?
Picture a satisfaction survey sent after an easy, successful purchase. The furious customer who abandoned checkout cannot answer. Neither can the confused visitor who never understood the offer. Ten thousand glowing responses later, the company congratulates itself on delighting precisely the people capable of completing the journey.
Size reduces sampling noise. Selection decides what has been sampled. Another six zeros cannot summon the missing customers, employees or voters; they measure the people already present with gorgeous precision.
Small research can be rubbish too. Three convenient customers do not become “qualitative truth” because everybody spoke for an hour and received a voucher. The useful question begins upstream: which population does this decision concern, and what process allowed an observation to enter the data?
Trace that process and the omissions become visible. Ignore it and even beautiful mathematics will polish the wrong answer. A huge ladle from an unstirred pot is still a lousy taste test; the impressive part is the ladle, not the tasting.