A background piece of my own models (which could be right/wrong independent of the content of this post) is that bounded rationality constraints basically just don’t bind at all in practice once a mind passes a certain threshold, and in particular will not bind for even moderately-superhuman general AI.
The canonical example illustrating how this could happen is inference involving an ideal gas. Roughly speaking, once an agent is smart enough to use the Boltzmann distribution, even a Jupiter brain will not be able to do significantly better in practice, because chaotic dynamics wipe out all the other signal. In that case, bounded rationality constraints cease to bind once the agent is smart enough to use a Boltzmann distribution (again, roughly speaking).
A background piece of my own models (which could be right/wrong independent of the content of this post) is that bounded rationality constraints basically just don’t bind at all in practice once a mind passes a certain threshold, and in particular will not bind for even moderately-superhuman general AI.
The canonical example illustrating how this could happen is inference involving an ideal gas. Roughly speaking, once an agent is smart enough to use the Boltzmann distribution, even a Jupiter brain will not be able to do significantly better in practice, because chaotic dynamics wipe out all the other signal. In that case, bounded rationality constraints cease to bind once the agent is smart enough to use a Boltzmann distribution (again, roughly speaking).