I see no contradictions with a superintelligent being mostly motivated to optimize virtual worlds, and it seems an interesting hypothesis of yours that this may be a common attractor. I expect this to be more likely if these simulations are rich enough to present a variety of problems, such that optimizing them continues to provide challenges and discoveries for a very long time.
Of course even a being that only cares about this simulated world may still take actions in the real-world (e.g. to obtain more compute power), so this “wire-heading” may not prevent successful power-seeking behavior.
The key thing to notice is that in order to exploit this scenario, we have to have a world-model that is precise enough to model reality much better than humans, but not be so good at modelling a reality that it’s world models are isomorphic to a reality.
This might be easy or challenging, but it does mean we probably can’t crank up the world-modeling part indefinitely while still trapping it via wireheading.
I see no contradictions with a superintelligent being mostly motivated to optimize virtual worlds, and it seems an interesting hypothesis of yours that this may be a common attractor. I expect this to be more likely if these simulations are rich enough to present a variety of problems, such that optimizing them continues to provide challenges and discoveries for a very long time.
Of course even a being that only cares about this simulated world may still take actions in the real-world (e.g. to obtain more compute power), so this “wire-heading” may not prevent successful power-seeking behavior.
The key thing to notice is that in order to exploit this scenario, we have to have a world-model that is precise enough to model reality much better than humans, but not be so good at modelling a reality that it’s world models are isomorphic to a reality.
This might be easy or challenging, but it does mean we probably can’t crank up the world-modeling part indefinitely while still trapping it via wireheading.