Informally, it’s the kind of intelligence (usually understood as something like ” the capacity to achieve goals in a wide variety of environments”) which is capable of doing that which is instrumental to achieving the goal. Given a goal, it is the capacity to achieve that goal, to do what is instrumental to achieving that goal.
Bostrom, in Superintelligence (2014), speaks of it as “means-end reasoning”.
So, strictly speaking, it does not involve reasoning about the ends or goals in service of which the intelligence/optimisation is being pressed.
Example: a chess-playing system will have some pre-defined goal and optimise instrumentally toward that end, but will not evaluate the goal itself.
Informally, it’s the kind of intelligence (usually understood as something like ” the capacity to achieve goals in a wide variety of environments”) which is capable of doing that which is instrumental to achieving the goal. Given a goal, it is the capacity to achieve that goal, to do what is instrumental to achieving that goal.
Bostrom, in Superintelligence (2014), speaks of it as “means-end reasoning”.
So, strictly speaking, it does not involve reasoning about the ends or goals in service of which the intelligence/optimisation is being pressed.
Example: a chess-playing system will have some pre-defined goal and optimise instrumentally toward that end, but will not evaluate the goal itself.