[link] Simplifying the environment: a new convergent instrumental goal

http://​​ka­j­so­tala.fi/​​2016/​​04/​​sim­plify­ing-the-en­vi­ron­ment-a-new-con­ver­gent-in­stru­men­tal-goal/​​

Con­ver­gent in­stru­men­tal goals (also ba­sic AI drives) are goals that are use­ful for pur­su­ing al­most any other goal, and are thus likely to be pur­sued by any agent that is in­tel­li­gent enough to un­der­stand why they’re use­ful. They are in­ter­est­ing be­cause they may al­low us to roughly pre­dict the be­hav­ior of even AI sys­tems that are much more in­tel­li­gent than we are.

In­stru­men­tal goals are also a strong ar­gu­ment for why suffi­ciently ad­vanced AI sys­tems that were in­differ­ent to­wards hu­man val­ues could be dan­ger­ous to­wards hu­mans, even if they weren’t ac­tively mal­i­cious: be­cause the AI hav­ing in­stru­men­tal goals such as self-preser­va­tion or re­source ac­qui­si­tion could come to con­flict with hu­man well-be­ing. “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for some­thing else.

I’ve thought of a can­di­date for a new con­ver­gent in­stru­men­tal drive: sim­plify­ing the en­vi­ron­ment to make it more pre­dictable in a way that al­igns with your goals.