Whether and how well an intelligence “steer[s] reality into regions that are higher in [its] preference ordering” is dependent not just on its intelligence but also on it’s “power”. A vastly intelligent AI that is running on a disk connected to all kinds of sensors and a power source, but not connected to anything that lets it effect the world outside of itself (internet, contact with humans, a robot arm, a nano-factory, etc.) won’t steer reality anywhere. All it can do is think about what it would do if ti could do ANYthing.
Defining intelligence in this way (as an optimization process), would lead us to disregard this hyper-intelligent entity as not an intelligence at all. It seems like we have account for power (the ability to actualize one’s will) somewhere. Granted, intelligence makes one powerful, but it is not the only attribute that does so.
Whether and how well an intelligence “steer[s] reality into regions that are higher in [its] preference ordering” is dependent not just on its intelligence but also on it’s “power”. A vastly intelligent AI that is running on a disk connected to all kinds of sensors and a power source, but not connected to anything that lets it effect the world outside of itself (internet, contact with humans, a robot arm, a nano-factory, etc.) won’t steer reality anywhere. All it can do is think about what it would do if ti could do ANYthing.
Defining intelligence in this way (as an optimization process), would lead us to disregard this hyper-intelligent entity as not an intelligence at all. It seems like we have account for power (the ability to actualize one’s will) somewhere. Granted, intelligence makes one powerful, but it is not the only attribute that does so.