Business decisions are noisier than we think

Before reading on, you may want to think of your own answer to the following questions: In a well-run insurance company, if you randomly selected two qualified underwriters or claims adjusters, how different would you expect their estimates for the same case to be? Specifically, what would be the difference between the two estimates, as a percentage of their average? We asked numerous executives in the company for their answers, and in subsequent years, we have obtained estimates from a wide variety of people in different professions. Surprisingly, one answer is clearly more popular than all others. Most executives of the insurance company guessed 10% or less. When we asked 828 CEOs and senior executives from a variety of industries how much variation they expected to find in similar expert judgments, 10% was also the median answer and the most frequent one (the second most popular was 15%). A 10% difference would mean, for instance, that one of the two underwriters set a premium of $9,500 while the other quoted $10,500. Not a negligible difference, but one that an organization can be expected to tolerate. Our noise audit found much greater differences. By our measure, the median difference in underwriting was 55%, about five times as large as was expected by most people, including the company’s executives.

Noise audit, case 17:

Underwriter A: £9,400.

Underwriter B: £15,100.

Both are qualified. Both have the same file. Each can explain the figure with calm authority.

That is the horror. An obviously incompetent answer creates a person to blame. Two plausible answers create an institution that can pretend discretion is functioning beautifully.

Executives expected differences around 10 per cent. The audit found a median difference of 55 per cent. Customers never see this comparison. Each receives one premium and assumes the number came from a stable process rather than whichever expert opened the case.

Averages conceal the lottery. The company may price accurately overall while treating similar people very differently. Aggregate success does not comfort the customer who drew the expensive assessor.

Two errors can even flatter the overall result by cancelling each other: one case priced far too high, another far too low, the mean sitting smugly near correct. Accuracy at company level can coexist with arbitrary treatment one decision at a time.

Structure helps where judgement adds little: the same input order, explicit criteria, reference cases and independent initial scores. Yet forcing every difficult case through one formula creates a different stupidity. Exceptions contain information, and experts should be able to depart from a guide when they can state why.

Case 18 produces close estimates. Case 19 splits even further. The disagreement discussion reveals that one underwriter treats an old claim as highly predictive while another nearly ignores it. That policy choice had been living inside individual judgement without a name.

The audit has created no villain and no triumphant fix. It has merely made professional consistency measurable, which is why some people suddenly become very fond of the word “art”.