Thesis Tiny probabilities of vast payoffs should simply be ignored, however large the expected value
Its coordinates: 2.6 Rationality · 2.8 Formal Epistemology
Where it stands
Our effective altruism group had its usual Thursday argument, and I lost it, so I'm bringing it here.
The setup is Pascal's mugging, a puzzle named after Pascal's wager. A stranger with no weapon asks for your wallet. In return he promises, by powers he can't demonstrate, a thousand quadrillion years of bliss. You think he is almost certainly lying. But "almost certainly" isn't "certainly", and if your credence that he's honest is even one in a trillion trillion, the expected value of handing over the wallet beats keeping it. And he can always raise the number until it does.
My claim: a rational agent should ignore possibilities below some very small probability, whatever payoff is attached to them. That isn't a fudge. It's how every sane person already lives. Nobody insures against a meteorite landing on their breakfast.
The obvious objection, which my friends pressed hard, is that any cut-off is arbitrary, and that you could slice a serious risk into many tiny ones, each below the threshold, and then ignore the lot.
The stakes aren't academic for us. A good deal of longtermist reasoning multiplies very small chances of changing the far future by astronomically large numbers of future people. If I'm right, much of that reasoning goes too. Am I?