I do not see how the agent ‘seeks’ out powerful states because, as you say, the agent is fixed.
I do think this is mostly a matter of translation of math to English being hard. Like, when Alex says “optimal agents seek power”, I think you should translate it as “when we don’t know what goal an optimal agent has, we should assign higher probability that it will go to states that have higher power”, even though the agent itself is not thinking “ah, this state is powerful, I’ll go there”.
[Deleted]
I do think this is mostly a matter of translation of math to English being hard. Like, when Alex says “optimal agents seek power”, I think you should translate it as “when we don’t know what goal an optimal agent has, we should assign higher probability that it will go to states that have higher power”, even though the agent itself is not thinking “ah, this state is powerful, I’ll go there”.