AlphaGo Zero and the Foom Debate

AlphaGo Zero uses 4 TPUs, is built en­tirely out of neu­ral nets with no hand­crafted fea­tures, doesn’t pre­train against ex­pert games or any­thing else hu­man, reaches a su­per­hu­man level af­ter 3 days of self-play, and is the strongest ver­sion of AlphaGo yet.

The ar­chi­tec­ture has been sim­plified. Pre­vi­ous AlphaGo had a policy net that pre­dicted good plays, and a value net that eval­u­ated po­si­tions, both feed­ing into looka­head us­ing MCTS (ran­dom prob­a­bil­ity-weighted plays out to the end of a game). AlphaGo Zero has one neu­ral net that se­lects moves and this net is trained by Paul Chris­ti­ano-style ca­pa­bil­ity am­plifi­ca­tion, play­ing out games against it­self to learn new prob­a­bil­ities for win­ning moves.

As oth­ers have also re­marked, this seems to me to be an el­e­ment of ev­i­dence that fa­vors the Yud­kowskian po­si­tion over the Han­so­nian po­si­tion in my and Robin Han­son’s AI-foom de­bate.

As I re­call and as I un­der­stood:

- Han­son doubted that what he calls “ar­chi­tec­ture” is much of a big deal, com­pared to (Han­son said) el­e­ments like cu­mu­la­tive do­main knowl­edge, or spe­cial-pur­pose com­po­nents built by spe­cial­ized com­pa­nies in what he ex­pects to be an ecol­ogy of com­pa­nies serv­ing an AI econ­omy.

- When I re­marked upon how it sure looked to me like hu­mans had an ar­chi­tec­tural im­prove­ment over chim­panzees that counted for a lot, Han­son replied that this seemed to him like a one-time gain from al­low­ing the cul­tural ac­cu­mu­la­tion of knowl­edge.

I em­pha­size how all the mighty hu­man ed­ifice of Go knowl­edge, the joseki and tac­tics de­vel­oped over cen­turies of play, the ex­perts teach­ing chil­dren from an early age, was en­tirely dis­carded by AlphaGo Zero with a sub­se­quent perfor­mance im­prove­ment. Th­ese mighty ed­ifices of hu­man knowl­edge, as I un­der­stand the Han­so­nian the­sis, are sup­posed to be the bulwark against rapid gains in AI ca­pa­bil­ity across mul­ti­ple do­mains at once. I was like “Hu­man in­tel­li­gence is crap and our ac­cu­mu­lated skills are crap” and this ap­pears to have been bourne out.

Similarly, sin­gle re­search labs like Deep­mind are not sup­posed to pull far ahead of the gen­eral ecol­ogy, be­cause adapt­ing AI to any par­tic­u­lar do­main is sup­posed to re­quire lots of com­po­nents de­vel­oped all over the place by a mar­ket ecol­ogy that makes those com­po­nents available to other com­pa­nies. AlphaGo Zero is much sim­pler than that. To the ex­tent that no­body else can run out and build AlphaGo Zero, it’s ei­ther be­cause Google has Ten­sor Pro­cess­ing Units that aren’t gen­er­ally available, or be­cause Deep­mind has a silo of ex­per­tise for be­ing able to ac­tu­ally make use of ex­ist­ing ideas like ResNets, or both.

Sheer speed of ca­pa­bil­ity gain should also be high­lighted here. Most of my ar­gu­ment for FOOM in the Y-H de­bate was about self-im­prove­ment and what hap­pens when an op­ti­miza­tion loop is folded in on it­self. Though it wasn’t nec­es­sary to my ar­gu­ment, the fact that Go play went from “no­body has come close to win­ning against a pro­fes­sional” to “so strongly su­per­hu­man they’re not re­ally both­er­ing any more” over two years just be­cause that’s what hap­pens when you im­prove and sim­plify the ar­chi­tec­ture, says you don’t even need self-im­prove­ment to get things that look like FOOM.

Yes, Go is a closed sys­tem al­low­ing for self-play. It still took hu­mans cen­turies to learn how to play it. Per­haps the new Han­so­nian bulwark against rapid ca­pa­bil­ity gain can be that the en­vi­ron­ment has lots of em­piri­cal bits that are sup­posed to be very hard to learn, even in the limit of AI thoughts fast enough to blow past cen­turies of hu­man-style learn­ing in 3 days; and that hu­mans have learned these vi­tal bits over cen­turies of cul­tural ac­cu­mu­la­tion of knowl­edge, even though we know that hu­mans take cen­turies to do 3 days of AI learn­ing when hu­mans have all the em­piri­cal bits they need; and that AIs can­not ab­sorb this knowl­edge very quickly us­ing “ar­chi­tec­ture”, even though hu­mans learn it from each other us­ing ar­chi­tec­ture. If so, then let’s write down this new world-wreck­ing as­sump­tion (that is, the world ends if the as­sump­tion is false) and be on the look­out for fur­ther ev­i­dence that this as­sump­tion might per­haps be wrong.

Tl;dr: As oth­ers are already re­mark­ing, the situ­a­tion with AlphaGo Zero looks noth­ing like the Han­so­nian hy­poth­e­sis and a heck of a lot more like the Yud­kowskian one.

Added: AlphaGo clearly isn’t a gen­eral AI. There’s ob­vi­ously stuff hu­mans do that make us much more gen­eral than AlphaGo, and AlphaGo ob­vi­ously doesn’t do that. How­ever, if even with the hu­man spe­cial sauce we’re to ex­pect AGI ca­pa­bil­ities to be slow, do­main-spe­cific, and re­quiring feed-in from a big mar­ket ecol­ogy, then the situ­a­tion we see with­out hu­man-equiv­a­lent gen­er­al­ity spe­cial sauce should not look like this.

To put it an­other way, I put a lot of em­pha­sis in my de­bate on re­cur­sive self-im­prove­ment and the re­mark­able jump in gen­er­al­ity across the change from pri­mate in­tel­li­gence to hu­man in­tel­li­gence. It doesn’t mean we can’t get info about speed of ca­pa­bil­ity gains with­out self-im­prove­ment. It doesn’t mean we can’t get info about the im­por­tance and gen­er­al­ity of al­gorithms with­out the gen­eral in­tel­li­gence trick. The de­bate can start to set­tle for fast ca­pa­bil­ity gains be­fore we even get to what I saw as the good parts; I wouldn’t have pre­dicted AlphaGo and lost money bet­ting against the speed of its ca­pa­bil­ity gains, be­cause re­al­ity held a more ex­treme po­si­tion than I did on the Yud­kowsky-Han­son spec­trum.

Cross­posted here.