In that scenario I would predict that the thing I was told was wrong, i.e. it is simply not true that one of them is anti-optimizing for negative profit. I have strong priors that people are optimizing for things they want.
Perhaps it’s just a prior that people are relatively good at optimizing for things they want. But the impossibility theorem seems to indicate that there are lots of different planners you could hypothesize, and somehow humans just seize upon one. (Though we’re often wrong, eg. typical mind fallacy.)
TL;DR: we do surprisingly well at inferring goals, given this impossibility result, and I’m not sure why. Maybe it’s a prior we’re born with.
One hypothesis why we do so well: we “simulate” other people on a very similar hardware, and relatively similar mind (when compared to the abstract set of planners). Which is a sort of strong implicit prior. (Some evidence for that is we have much more trouble inferring goals of other people if their brains function far away from what’s usual on some dimension)
In that scenario I would predict that the thing I was told was wrong, i.e. it is simply not true that one of them is anti-optimizing for negative profit. I have strong priors that people are optimizing for things they want.
Perhaps it’s just a prior that people are relatively good at optimizing for things they want. But the impossibility theorem seems to indicate that there are lots of different planners you could hypothesize, and somehow humans just seize upon one. (Though we’re often wrong, eg. typical mind fallacy.)
TL;DR: we do surprisingly well at inferring goals, given this impossibility result, and I’m not sure why. Maybe it’s a prior we’re born with.
One hypothesis why we do so well: we “simulate” other people on a very similar hardware, and relatively similar mind (when compared to the abstract set of planners). Which is a sort of strong implicit prior. (Some evidence for that is we have much more trouble inferring goals of other people if their brains function far away from what’s usual on some dimension)