Questions for a Friendly AI FAQ

I’ve be­gun work (with a few oth­ers) on a some­what com­pre­hen­sive Friendly AI F.A.Q. The an­swers will be much longer and more de­tailed than in the Sin­gu­lar­ity FAQ. I’d ap­pre­ci­ate feed­back on which ques­tions should be added.

1. Friendly AI: His­tory and Concepts

1. What is Friendly AI?

2. What is the Sin­gu­lar­ity? [w/​ ex­pla­na­tion of all three types]

3. What is the his­tory of the Friendly AI Con­cept?

4. What is nan­otech­nol­ogy?

5. What is biolog­i­cal cog­ni­tive en­hance­ment?

6. What are brain-com­puter in­ter­faces?

7. What is whole brain em­u­la­tion?

8. What is gen­eral in­tel­li­gence? [w/​ ex­pla­na­tion of why ‘op­ti­miza­tion power’ may less con­fus­ing than ‘in­tel­li­gence’, which tempts an­thro­po­mor­phic bias]

9. What is greater-than-hu­man in­tel­li­gence?

10. What is su­per­in­tel­li­gence, and what pow­ers might it have?

2. The Need for Friendly AI

1. What are the paths to an in­tel­li­gence ex­plo­sion?

2. When might an in­tel­li­gence ex­plo­sion oc­cur?

3. What are AI take­off sce­nar­ios?

4. What are the likely con­se­quences of an in­tel­li­gence ex­plo­sion? [sur­vey of pos­si­ble effects, good and bad]

5. Can we just keep the ma­chine su­per­in­tel­li­gence in a box, with no ac­cess to the in­ter­net?

6. Can we just cre­ate an Or­a­cle AI that in­forms us but doesn’t do any­thing?

7. Can we just pro­gram ma­chines not to harm us?

8. Can we pro­gram a ma­chine su­per­in­tel­li­gence to max­i­mize hu­man plea­sure or de­sire satis­fac­tion?

9. Can we teach a ma­chine su­per­in­tel­li­gence a moral code with ma­chine learn­ing?

10. Won’t some other so­phis­ti­cated sys­tem con­strain AGI be­hav­ior?

3. Co­her­ent Ex­trap­o­lated Volition

1. What is Co­her­ent Ex­trap­o­lated Vo­li­tion (CEV)?

2. …

4. Alter­na­tives to CEV

1. …

5. Open Prob­lems in Friendly AI Research

1. What is re­flec­tive de­ci­sion the­ory?

2. What is time­less de­ci­sion the­ory?

3. How can an AI pre­serve its util­ity func­tion through­out on­tolog­i­cal shifts?

4. How can an AI have prefer­ences over the ex­ter­nal world?

5. How can an AI choose an ideal prior given in­finite com­put­ing power?

6. How can an AI deal with log­i­cal un­cer­tainty?

7. How can we elicit a util­ity func­tion from hu­man be­hav­ior and func­tion?

8. How can we de­velop microe­co­nomic mod­els for self-im­prov­ing sys­tems?

9. How can tem­po­ral, bounded agents ap­prox­i­mate ideal Bayesi­anism?