I think “unlearnable” is a level removed from equally-important questions (like un-modelable, or sufficiently-precise model is not computable by Alice’s means, even if she could learn/find it). And that this treatment gives FAR too binary a focus on what a “theory of mind” actually does, and how precise and correct it is.
I think we have to start with “how good is Stuart Armstrong’s theory of mind for humans (both for their best friend and for a randomly-selected person from a different culture and generation, and for a population making collective decisions)”, as a an upper bound for how good Alice’s theories of mind can be can be at all. Identify specific predictions we want from the theory, and figure out how we’ll evaluate it for humans and for aliens.
I’d expect that it’s not very good to start with, but still better than random and we can do some useful modeling. For aliens, you’re correct that our assumptions matter a lot (well, what matters is the aliens’ specifics, but our assumptions control our modeling of our ability to model). For aliens much more complex than ourselves, our theory of mind will be less accurate, as there’s less of their computations that we can simplify to fit in our heads/computers.
I think “unlearnable” is a level removed from equally-important questions (like un-modelable, or sufficiently-precise model is not computable by Alice’s means, even if she could learn/find it). And that this treatment gives FAR too binary a focus on what a “theory of mind” actually does, and how precise and correct it is.
I think we have to start with “how good is Stuart Armstrong’s theory of mind for humans (both for their best friend and for a randomly-selected person from a different culture and generation, and for a population making collective decisions)”, as a an upper bound for how good Alice’s theories of mind can be can be at all. Identify specific predictions we want from the theory, and figure out how we’ll evaluate it for humans and for aliens.
I’d expect that it’s not very good to start with, but still better than random and we can do some useful modeling. For aliens, you’re correct that our assumptions matter a lot (well, what matters is the aliens’ specifics, but our assumptions control our modeling of our ability to model). For aliens much more complex than ourselves, our theory of mind will be less accurate, as there’s less of their computations that we can simplify to fit in our heads/computers.