Wait, but can’t the AI also choose to adopt the strategy “build another computer with a larger largest computable number”?
If the computer has a finite amount of memory and can’t build more, this puts a 2n bound on how long it can weight. If it can build more, it will.
The point is that it needs to pick some long running computation that it can be fairly sure halts eventually.
This gets into details about exactly how the AI is handling logical uncertainty.
Wait, but can’t the AI also choose to adopt the strategy “build another computer with a larger largest computable number”?
If the computer has a finite amount of memory and can’t build more, this puts a 2n bound on how long it can weight. If it can build more, it will.
The point is that it needs to pick some long running computation that it can be fairly sure halts eventually.
This gets into details about exactly how the AI is handling logical uncertainty.