We judge things relative to nearby alternatives, not in isolation.
More quotesA platform does not become neutral because its prejudice is automated by strangers.
A name looks like innocent information. In the Airbnb research described here, it also acted as a cue from which hosts inferred race, with measurable consequences for price and acceptance. That is the brutal design lesson: identity can enter a decision before the information relevant to the transaction.
The same risk appears in CV screening, marketplace enquiries and requests for finance. A hiring manager sees a name before evidence of the work. A host sees a profile photograph before the booking history. A lender may encounter a postcode that carries more social meaning than it deserves. Nobody has to announce a discriminatory policy for a discriminatory pattern to emerge.
Companies love calling themselves platforms at this point, as though the word absolves the architecture. It does not. The order, prominence and necessity of identity cues are product decisions. So are audit logs, structured criteria and the ability to detect unequal outcomes. There are practical moves: delay irrelevant identity information; show qualifications or transaction evidence first; standardise what decision-makers assess; test outcomes by group; investigate differences rather than decorating them with a diversity statement.
Blindness is not always possible or desirable, and fairness is not solved by hiding every human detail. But “users did it” is a coward’s answer when the product chose the conditions.
Design decides which facts get to speak first.