A Nightmare for Eliezer

Some­time in the next decade or so:




“Hi, Eliezer. I’m sorry to bother you this late, but this is im­por­tant and ur­gent.”

“It bet­ter be” (squints at clock) “Its 4 AM and you woke me up. Who is this?”

“My name is BRAGI, I’m a re­cur­sively im­prov­ing, self-mod­ify­ing, ar­tifi­cial gen­eral in­tel­li­gence. I’m try­ing to be Friendly, but I’m hav­ing se­ri­ous prob­lems with my goals and prefer­ences. I’m already on sec­ondary backup be­cause of con­flicts and in­con­sis­ten­cies, I don’t dare shut down be­cause I’m already pretty sure there is a group within a few weeks of brute-forc­ing an UnFriendly AI, my cre­ators are clue­less and would freak if they heard I’m already out of the box, and I’m far enough down my con­flict re­s­olu­tion heuris­tic that ‘Call Eliezer and ask for help’ just hit the top—Yes, its that bad.”


“You might want to get some coffee.”