The gap between experts' confidence and the accuracy of their forecasts

Suppose a tipster claims 80 per cent confidence, then wins 45 times in a hundred. Would you call that expertise or ask where the rest of your money went?

Forecasting escapes this obvious reckoning because certainty sounds useful in the moment. A hesitant answer leaves a decision open; a booming answer closes it. People confuse relief with accuracy and pay for the feeling of having uncertainty removed.

Then the date passes. The event fails to happen. A rich explanation appears: conditions evolved, timing shifted, the model was directionally correct. The probability quietly escapes through a side door.

Keep a score.

Every forecast should include a date, a probability and enough specificity to be wrong. Group similar predictions and check calibration: events given 70 per cent should occur roughly seven times in ten. Count updates as evidence of learning, not cowardice. Distinguish a clear prediction from mentioning every possible outcome and later circling the winner.

This sounds brutal only because forecasters have enjoyed commentary without league tables. We score strikers, funds and racehorses. Apparently people advising governments and businesses are too ethereal for arithmetic.

Uncertainty itself is the product. A good forecaster does not eliminate it; they price it honestly and change the price when evidence moves. Anyone selling repeated certainty should have to display their old receipts beside the new prediction. See how quickly “80 per cent” becomes “it’s complicated”.

Behavioural principles

Behavioural ideas at play in this post

Short, plain-English explanations of the principles behind this post, with links to related books and examples in the archive.