Our tendency to underestimate the variance - or noise - in business

In a well-run insurance company, if you randomly selected two qualified underwriters or claims adjusters, how different would you expect their estimates for the same case to be? Specifically, what would be the difference between the two estimates, as a percentage of their average?

We asked numerous executives in the company for their answers, and in subsequent years, we have obtained estimates from a wide variety of people in different professions. Surprisingly, one answer is clearly more popular than all others. Most executives of the insurance company guessed 10% or less. When we asked 828 CEOs and senior executives from a variety of industries how much variation they expected to find in similar expert judgments, 10% was also the median answer and the most frequent one (the second most popular was 15%). A 10% difference would mean, for instance, that one of the two underwriters set a premium of $9,500 while the other quoted $10,500. Not a negligible difference, but one that an organization can be expected to tolerate.

Our noise audit found much greater differences. By our measure, the median difference in underwriting was 55%, about five times as large as was expected by most people, including the company’s executives.

£9,500 from one qualified underwriter. £10,500 from another. That was the kind of difference executives expected. Across the audit, the median gap was 55 per cent. Internally, each professional usually sees only their own answer, so the inconsistency remains hidden.

Commercial noise has no consistent direction. Bias can sometimes be found in the average. Noise means the same process treats comparable people differently depending on who picked up the work, what they saw first or whether lunch was disappointing. Replacing every professional with a rigid score would create different errors: standardisation can erase relevant differences and encourage blind obedience to poor data. First measure variation with repeated, independently assessed cases. Let people make their initial judgement before discussion. Then inspect where estimates separate and whether the stated reasons survive comparison.

Decision guides, shared reference cases and structured inputs can reduce arbitrary variance while preserving room for exceptions. Exceptions should require a reason somebody else can review. Executives guessed around 10 per cent. The measured median was 55 per cent. Before improving the process, the company has to tolerate seeing how inconsistent it already is.

Two new cases arrive while the audit findings are still being discussed. Both underwriters assess them independently, record the reasons and compare only afterwards. The gaps are smaller in one case and larger in the other.