When a prediction changes what it predicts: reflexivity in my day job
Last year my team built a model to predict which small-business customers would fall behind on repayments. It worked well on historical data. Then the business acted on it: flagged customers got tighter terms and more calls. Some of them, squeezed, did fall behind. Others, warned, got their affairs in order and paid early. Six months later the model's accuracy looked strange, and nobody could say whether it had been right.
This is an old problem with a new face. Robert Merton's essay on the self-fulfilling prophecy in 1948 used a bank run: a rumour that a sound bank will fail makes depositors withdraw, which makes it fail. Ian Hacking called a related thing the looping effect: classifying people changes them, and then the classification is out of date.
What strikes me philosophically:
- A social prediction cannot be stated independently of whether it is known. Planets do not read astronomy; customers do read their bank letters.
- So "was the model accurate?" may have no clean answer once it is used.
- And the choice of what to predict is not neutral, because the prediction becomes an intervention.
In data teams we call this "feedback" or "drift" and treat it as a technical nuisance. I'm starting to think it's the central fact about predicting people. Should social models be judged as descriptions at all, or as policies?