Reframing the Problem of AI Progress

“Fas­ci­nat­ing! You should definitely look into this. For­tu­nately, my own re­search has no chance of pro­duc­ing a su­per in­tel­li­gent AGI, so I’ll con­tinue. Good luck son! The gov­ern­ment should give you more money.”

Stu­art Arm­strong para­phras­ing a typ­i­cal AI researcher

I for­got to men­tion in my last post why “AI risk” might be a bad phrase even to de­note the prob­lem of UFAI. It brings to mind analo­gies like physics catas­tro­phes or as­tro­nom­i­cal dis­asters, and lets AI re­searchers think that their work is ok as long as they have lit­tle chance of im­me­di­ately de­stroy­ing Earth. But the real prob­lem we face is how to build or be­come a su­per­in­tel­li­gence that shares our val­ues, and given that this seems very difficult, any progress that doesn’t con­tribute to the solu­tion but brings for­ward the date by which we must solve it (or be stuck with some­thing very sub­op­ti­mal even if it doesn’t kill us), is bad. The word “risk” con­notes a small chance of some­thing bad sud­denly hap­pen­ing, but slow steady progress to­wards los­ing the fu­ture is just as wor­ri­some.

The usual way of stat­ing the prob­lem also in­vites lots of de­bate that are largely beside the point (as far as de­ter­min­ing how se­ri­ous the prob­lem is), like whether in­tel­li­gence ex­plo­sion is pos­si­ble, or whether a su­per­in­tel­li­gence can have ar­bi­trary goals, or how sure we are that a non-Friendly su­per­in­tel­li­gence will de­stroy hu­man civ­i­liza­tion. If some­one wants to ques­tion the im­por­tance of fac­ing this prob­lem, they re­ally in­stead need to ar­gue that a su­per­in­tel­li­gence isn’t pos­si­ble (not even a mod­est one), or that the fu­ture will turn out to be close to the best pos­si­ble just by ev­ery­one push­ing for­ward their own re­search with­out any con­cern for the big pic­ture, or per­haps that we re­ally don’t care very much about the far fu­ture and dis­tant strangers and should pur­sue AI progress just for the im­me­di­ate benefits.

(This is an ex­panded ver­sion of a pre­vi­ous com­ment.)