Why survey answers change with the question

For example, a questionnaire on the number of headaches people experience in one week was given to two different groups of subjects. One group had to indicate whether the number was 1-5, 6-10, 11-15, and so on, while the other was presented with the numbers broken down into 1-3, 4-6, 7-9, etc. The first group reported many more headaches than the second. Moreover, almost everyone is influenced by the two end points of a scale, tending to pick a number that is near the middle.

“Seventy per cent of customers agree.” With what, exactly? The proposition, the flattering wording, or the answer range that quietly told them what normal people are supposed to say?

Surveys are too often treated as buckets lowered into a well of existing opinion. They are closer to cocktail shakers. Ask whether a buyer would pay between £10 and £50 and you have introduced one world; stretch the scale to £500 and you have introduced another. Offer an employee “one to three stressful days” as the low band and stress starts looking abundant. Force somebody to choose a job title from your dropdown and—miracle of miracles—the population now resembles your database. Then the chart arrives in a board deck wearing a tie and pretending none of this happened. I do not distrust surveys; I distrust the laundering of a designed interaction into “what people think”.

For any important claim, I want the exact question printed beside the result. I want plausible alternative wording tested, option order rotated, open text inspected and stated preference compared with behaviour where possible. If changing the scale changes the conclusion, the conclusion belongs partly to the scale. That is not an awkward methodological footnote. It is the finding. Publish the question or keep the triumphant percentage to yourself.