One difficulty is that, since the agent is part of the environment, modeling the environment in every detail would require the agent to model itself in every detail, which would require the agent’s self-model to be as “big” as the whole agent. An agent can’t fit inside its own head.
Nitpick: This isn’t necessarily true. Many fractals contain parts which are exact miniature duplicates of the whole. Thus it is in principle possible for an agent to contain a perfectly detailed, perfectly exact representation of a world bigger than itself, even with the most central/paradigmatic/literal sort of representation. Whether it is in practice possible is a different story… But still, it seems not obvious that this is a big problem, especially since the agent’s self-model doesn’t have to be literally a tiny image of the world. (I trust your expertise that there is a real problem here, I just think that this quoted passage does a poor job of motivating the problem.)
In real world computers, we have finite memory, so my reading of this was assuming a finite state space. The fractal stuff requires an infinite sets, where two notions of smaller (‘is a subset of’ and ‘has fewer elements’) disagree—the mini-fractal is a subset of the whole fractal, but it has the same number of elements and hence corresponds perfectly.
Nitpick: This isn’t necessarily true. Many fractals contain parts which are exact miniature duplicates of the whole. Thus it is in principle possible for an agent to contain a perfectly detailed, perfectly exact representation of a world bigger than itself, even with the most central/paradigmatic/literal sort of representation. Whether it is in practice possible is a different story… But still, it seems not obvious that this is a big problem, especially since the agent’s self-model doesn’t have to be literally a tiny image of the world. (I trust your expertise that there is a real problem here, I just think that this quoted passage does a poor job of motivating the problem.)
In real world computers, we have finite memory, so my reading of this was assuming a finite state space. The fractal stuff requires an infinite sets, where two notions of smaller (‘is a subset of’ and ‘has fewer elements’) disagree—the mini-fractal is a subset of the whole fractal, but it has the same number of elements and hence corresponds perfectly.
What about quines?