How too much data can make us overconfident in our predictions

The problem of more data was investigated by Paul Slovic, Professor of Psychology at the University of Oregon. He ran an experiment with professional horseracing handicap setters in which they were given a list of 88 variables that were useful in predicting a horse’s performance. The participants then had to predict the outcome of the race and their confidence in their prediction. They repeated these tasks with access to different levels of data: either 5, 10, 20, 30 or 40 of the variables.

The results were illuminating. Accuracy was the same regardless of the number of variables used. However, overconfidence grew as more data was harnessed. Experts overestimated the importance of factors that had a limited value. It was only when five data points were used that accuracy and confidence were well calibrated.

Marketers face a similar set of problems. They have access to more data than ever before and many believe that because the information exists they should use it. The Slovic experiment suggests otherwise. We shouldn’t harness data just because we can. Instead, as much time should be spent choosing which data sets to ignore as which to use.

Forty variables produced the same accuracy and a larger ego.

That result should be printed above every dashboard login.

More information feels like more control because the model develops knobs, labels and a diagram with arrows. Marketing attribution can add six touchpoints and three identity graphs without improving its answer. Hiring panels schedule another interview, then mistake the fresh adjectives for fresh signal. Project plans grow dependencies until uncertainty looks engineered.

Useful data does exist. The trick is making each addition compete for its chair. Begin with the smallest credible set, record the forecast, then add a variable and see whether performance improves somewhere outside the sample. Confidence must be measured too. A model becoming more certain while remaining equally wrong is not maturing; it is learning executive presence.

Accumulation has political advantages. A warehouse looks like an asset. Excluding a metric feels negligent. Nobody gets promoted for deleting seventeen dashboard tiles and admitting three numbers do the work. Yet selection is the judgement companies claim they hired analysts to provide.

Data cannot rescue a decision from the obligation to choose what matters. When detail adds explanation but no predictive power, it is office décor. Very expensive décor, admittedly, with excellent filters.