I think this is the key issue. There’s a difference between inaccessible-as-in-expensive, and inaccessible-as-in-there-is-no-unique-answer.
If it’s really impossible to tell what Alice is thinking, the safe bet not that Alice has some Platonic property of What She’s Really Thinking that we just can’t access. The safe bet is that the abstract model of the world we have where there’s some unique answer to what Alice is thinking doesn’t match reality well here. What we want isn’t an AI that accesses Alice’s Platonic properties, we want an AI that figures out what’s “relevant, helpful, useful, etc.”
I think this is the key issue. There’s a difference between inaccessible-as-in-expensive, and inaccessible-as-in-there-is-no-unique-answer.
If it’s really impossible to tell what Alice is thinking, the safe bet not that Alice has some Platonic property of What She’s Really Thinking that we just can’t access. The safe bet is that the abstract model of the world we have where there’s some unique answer to what Alice is thinking doesn’t match reality well here. What we want isn’t an AI that accesses Alice’s Platonic properties, we want an AI that figures out what’s “relevant, helpful, useful, etc.”