“But I have this nagging sense that even in that case the approximations may not do everything we want because they are trying to optimize on features that are unachievable for real agents”—This seems to be the central plank of your case. I don’t suppose you could spec it out some more? I have some idea of what you might be thinking, but I don’t want to put words in your mouth.
Well since I was confused my reasons for concern have dissolved and timelessness now feels like aiming in the direction of what you want, although it would probably make sense to consider stepping back further such that you are not trapped by, say, assumptions about the metaphysics of the world (think, allow yourself to go up a Tegmark level). So that aside I now wonder what you had in mind when I said this, since whatever i was thinking was not relevant.
Well since I was confused my reasons for concern have dissolved and timelessness now feels like aiming in the direction of what you want, although it would probably make sense to consider stepping back further such that you are not trapped by, say, assumptions about the metaphysics of the world (think, allow yourself to go up a Tegmark level). So that aside I now wonder what you had in mind when I said this, since whatever i was thinking was not relevant.