Why ignoring small data sets can backfire

The ethnographer Tricia Wang even suggested in her 2016 TEDxCambridge talk that quantification bias created by big data led to the near death of Nokia as a handset manufacturer. All their data suggested that people would only spend a certain proportion of their salary on a phone handset, so the market for smartphones in the developing world would be correspondingly small. Wang noticed that, once people saw a smartphone, their readiness to spend on a handset soared. Her findings were ignored as she had ‘too few data points’. However, in reality, all valuable information starts with very little data – the lookout on the Titanic only had one data point… ‘Iceberg ahead’, but it was more important than any huge survey on iceberg frequency.

The lookout on the Titanic had one data point. Its sample size was appalling. Its relevance was excellent.

“Too few data points” can be a sensible warning or a bureaucratic anaesthetic. It prevents a strange observation from disturbing a large, reassuring dataset. The spreadsheet says people in this market will never spend that much on a phone. One person sees them change their mind when the phone appears. Guess which evidence fits neatly into the forecast.

Big data is superb at describing repeated behaviour inside the world that produced it. It becomes less impressive when the world changes, the product creates a new appetite or the interesting event has happened only once. The first customer using a tool in an unintended way is numerically insignificant and strategically priceless. The first crack in a dam has dreadful statistical power right up until the second one.

Dashboard culture makes this worse because abundance looks like authority. A million neat rows arrive in a polished chart; one uncomfortable observation arrives as a sentence from someone whose job title lacks “insights”. Guess which gets forwarded. Organisations do not merely count data. They rank the people carrying it.

Small evidence should not be promoted straight to universal truth. An anecdote is a flare, not a map. Investigate it. Look for the mechanism. Ask whether it predicts something the larger model missed, then test that prediction quickly.

The real stupidity is treating confidence as a function of row count alone. Ten million records answering yesterday’s question can lose to one observation that changes tomorrow’s. When the strange little fact contradicts the beautiful dashboard, do not squash it for spoiling the colour scheme. Walk over and have a look.