Lone Genius Bias and Returns on Additional Researchers

One thing that most puz­zles me about Eliezer’s writ­ings on AI is his ap­par­ent be­lief that a small or­ga­ni­za­tion like MIRI is likely to be able to beat larger or­ga­ni­za­tions like Google or the US Depart­ment of Defense to build­ing hu­man-level AI. In fact, he seems to be­lieve such larger or­ga­ni­za­tions may have no ad­van­tage at all over a smaller one, and per­haps will even be at a dis­ad­van­tage. In his 2011 de­bate with Robin Han­son, he said:

As far as I can tell what hap­pens when the gov­ern­ment tries to de­velop AI is noth­ing. But that could just be an ar­ti­fact of our lo­cal tech­nolog­i­cal level and it might change over the next few decades. To me it seems like a deeply con­fus­ing is­sue whose an­swer is prob­a­bly not very com­pli­cated in an ab­solute sense. Like we know why it’s difficult to build a star. You’ve got to gather a very large amount of in­ter­stel­lar hy­dro­gen in one place. So we un­der­stand what sort of la­bor goes into a star and we know why a star is difficult to build. When it comes to build­ing a mind, we don’t know how to do it so it seems very hard. We like query our brains to say “map us a strat­egy to build this thing” and it re­turns null so it feels like it’s a very difficult prob­lem. But in point of fact we don’t ac­tu­ally know that the prob­lem is difficult apart from be­ing con­fus­ing. We un­der­stand the star-build­ing prob­lem so we know it’s difficult. This one we don’t know how difficult it’s go­ing to be af­ter it’s no longer con­fus­ing.

So to me the AI prob­lem looks like a—it looks to me more like the sort of thing that the prob­lem is find­ing bright enough re­searchers, bring­ing them to­gether, let­ting them work on that prob­lem in­stead of de­mand­ing that they work on some­thing where they’re go­ing to pro­duce a progress re­port in two years which will val­i­date the per­son who ap­proved the grant and ad­vance their ca­reer. And so the gov­ern­ment has his­tor­i­cally been tremen­dously bad at pro­duc­ing ba­sic re­search progress in AI, in part be­cause the most se­nior peo­ple in AI are of­ten peo­ple who got to be very se­nior by hav­ing failed to build it for the longest pe­riod of time. (This is not a uni­ver­sal state­ment. I’ve met smart se­nior peo­ple in AI.)

But nonethe­less, ba­si­cally I’m not very afraid of the gov­ern­ment be­cause I don’t think it’s a throw warm bod­ies at the prob­lem and I don’t think it’s a throw warm com­put­ers at the prob­lem. I think it’s a good method­ol­ogy, good peo­ple se­lec­tion, let­ting them do suffi­ciently blue sky stuff, and so far his­tor­i­cally the gov­ern­ment has been tremen­dously bad at pro­duc­ing that kind of progress. (When they have a great big pro­ject to try to build some­thing it doesn’t work. When they fund long-term re­search it works.)

I ad­mit, I don’t feel like I fully grasp all the rea­sons for the dis­agree­ment be­tween Eliezer and my­self on this is­sue. Some of the dis­agree­ment, I sus­pect, comes from slightly differ­ent views on the na­ture of in­tel­li­gence, though I’m hav­ing the trou­ble pin­point­ing what those differ­ences might be. But some of the differ­ence, I’m think, comes from the fact that I’ve be­come con­vinced hu­mans suffer from a Lone Ge­nius Bias—a ten­dency to over-at­tribute sci­en­tific and tech­nolog­i­cal progress to the efforts of lone ge­niuses.

Dis­claimer: My un­der­stand­ing of Luke’s cur­rent strat­egy for MIRI is that it does not hinge on whether or not MIRI it­self even­tu­ally builds AI or not. It seems to me that as long as MIRI keeps pub­lish­ing re­search that could po­ten­tially help other peo­ple build FAI, MIRI is do­ing im­por­tant work. There­fore, I wouldn’t ad­vo­cate any­thing in this post be­ing taken as a rea­son not to donate to MIRI. I’ve donated re­cently, and will prob­a­bly [edit: see be­low] con­tinue to do so in the fu­ture.

In­tel­li­gence Ex­plo­sion Microe­co­nomics has an in­ter­est­ing sec­tion la­beled “Re­turns on Pop­u­la­tion” (sec­tion 3.4) where, among other things, Eliezer says:

Although I ex­pect that this sec­tion of my anal­y­sis will not be with­out con­tro­versy, it ap­pears to the au­thor to also be an im­por­tant piece of data to be ex­plained that hu­man sci­ence and en­g­ineer­ing seem to scale over time bet­ter than over pop­u­la­tion—an ex­tra decade seems much more valuable than adding warm bod­ies.

In­deed, it ap­pears to the au­thor that hu­man sci­ence scales lu­dicrously poorly with in­creased num­bers of sci­en­tists, and that this is a ma­jor rea­son there hasn’t been more rel­a­tive change from 1970–2010 than from 1930–1970 de­spite the vastly in­creased num- ber of sci­en­tists. The rate of real progress seems mostly con­stant with re­spect to time, times a small fac­tor more or less. I ad­mit that in try­ing to make this judg­ment I am try­ing to sum­ma­rize an over­whelm­ingly dis­tant grasp on all the fields out­side my own hand­ful. Even so, a com­plete halt to sci­ence or a truly ex­po­nen­tial (or even quadratic) speedup of real progress both seem like they would be hard to miss, and the ex­po­nen­tial in­crease of pub­lished pa­pers is mea­surable. Real sci­en­tific progress is con­tin­u­ing over time, so we haven’t run out of things to in­ves­ti­gate; and yet some­how real sci­en­tific progress isn’t scal­ing any­where near as fast as pro­fes­sional sci­en­tists are be­ing added.

The most char­i­ta­ble in­ter­pre­ta­tion of this phe­nomenon would be that sci­ence prob­lems are get­ting harder and fields are adding sci­en­tists at a com­bined pace which pro­duces more or less con­stant progress. It seems plau­si­ble that, for ex­am­ple, In­tel adds new re­searchers at around the pace re­quired to keep up with its ac­cus­tomed ex­po­nen­tial growth...

Eliezer goes on to sug­gest, how­ever, that In­tel is not at all typ­i­cal, and pro­poses some other ex­pla­na­tions, two of which (“sci­ence is in­her­ently bounded by se­rial causal depth” and that sci­en­tific progress is limited by the need to wait for the last gen­er­a­tion to die) sug­gest that progress doesn’t scale at all with added re­searchers, at least past a cer­tain point.

I’m in­clined to think that that Eliezer’s ba­sic claim here—that re­search progress scales bet­ter with time than pop­u­la­tion—is prob­a­bly cor­rect. Dou­bling the num­ber of re­searchers work­ing on a prob­lem rarely means solv­ing the prob­lem twice as fast. How­ever, I doubt the scal­ing is as lu­dicrously bad as Eliezer sug­gests. I sus­pect the case of In­tel is fairly typ­i­cal, and the “sci­ence prob­lems are get­ting harder” the­ory of the his­tory of sci­ence has a lot more go­ing for it than Eliezer wants to grant.

For one thing, there seems to be a hu­man bias in fa­vor of at­tribut­ing sci­en­tific and tech­nolog­i­cal progress to lone ge­niuses—call it the Lone Ge­nius Bias. In fic­tion, it’s com­mon for the cast to have a sin­gle “smart guy,” a Reed Richards type, who does ev­ery­thing im­por­tant in the the sci­ence and tech­nol­ogy area, pul­ling off mirac­u­lous achieve­ments all by him­self. (If you’re lucky, this role will be shared by two char­ac­ters, like Fitz-Sim­mons on Joss Whe­don’s new S.H.I.L.D. TV show.) Similarly, villain­ous plots of­ten hinge on kid­nap­ping one sin­gle sci­en­tist who will be able to fulfill all the villain with all the villain’s tech­ni­cal know-how needs.

There’s some rea­son to chalk this up to pe­cu­liar­i­ties of fic­tion (see TVTtropes ar­ti­cles on the Om­ni­dis­ci­plinary Scien­tist and The Main Char­ac­ters Do Every­thing gen­er­ally). But it of­ten seems to bleed over into per­cep­tions of real-life sci­en­tists and en­g­ineers. Saul Kripke, in the course of mak­ing a point about proper names, once claimed that he of­ten met peo­ple who iden­ti­fied Ein­stein as the in­ven­tor of the atom bomb.

Of course, in re­al­ity, Ein­stein just pro­vided the ini­tial the­o­ret­i­cal ba­sis for the atom bomb. Not only did the bomb it­self re­quire the Man­hat­tan Pro­ject (which in­volved over 100,000 peo­ple) to build, but there there was a fair amount of ba­sic sci­ence that had to take place af­ter Ein­stein’s origi­nal state­ment of mass-en­ergy equiv­alence in 1905 be­fore the Man­hat­tan Pro­ject could even be con­ceived of.

Or: in the pop­u­lar imag­i­na­tion, Thomas Edi­son was an amaz­ingly brilli­ant in­ven­tor, al­most on par with Reed Richards. A con­trar­ian view, pop­u­lar among tech geeks, says that ac­tu­ally Edi­son was a jerk who got fa­mous tak­ing credit for other peo­ple’s work, and also he de­pended on hav­ing a lot of other peo­ple work­ing for him at Menlo Park. But then there’s a meta-con­trar­ian view that ar­gues that Menlo Park was “the first in­dus­trial re­search lab,” and in­dus­trial re­search labs are very im­por­tant, to the point that Menlo Park it­self was Edi­son’s “ma­jor in­no­va­tion.” On this view, it’s not Edi­son’s fault that Lone Ge­nius Bias leads peo­ple to mi­s­un­der­stand what his true con­tri­bu­tion was.

It’s easy to see, in evolu­tion­ary terms, why hu­mans might suffer from Lone Ge­nius Bias. In the an­ces­tral en­vi­ron­ment, ma­jor achieve­ments would of­ten have been the work of a sin­gle in­di­vi­d­ual. The­o­ret­i­cally, there might have been the oc­ca­sional achieve­ment that re­quired the co­op­er­a­tion of a whole en­tire hunter-gath­erer band, but ma­jor achieve­ments were never the work of In­tel-sized R&D de­part­ments or 100,000 per­son Man­hat­tan Pro­jects. (The is an in­stance of the more gen­eral prin­ci­ple that hu­mans have trou­ble fully grokking com­plex mod­ern so­cieties.)

Once you know about Lone Ge­nius Bias, you should be sus­pi­cious when you find your­self grav­i­tat­ing to­wards fu­ture sce­nar­ios where the key in­no­va­tions are the work of a few ge­niuses. Fur­ther­more, it’s not just that big pro­jects are more com­mon now than they were in the an­ces­tral en­vi­ron­ment. The ten­dency of ma­jor ad­vances to be the work of large groups seems to have no­tice­ably in­creased over just the last cen­tury or so, and that trend may only con­tinue even fur­ther in the fu­ture.

Con­sider No­bel Prizes. The first No­bel Prizes were awarded in 1901. When peo­ple think of No­bel Prize win­ners they tend to think of un­shared No­bel Prizes, like Ein­stein’s, but in fact a No­bel Prize can be shared by up to three peo­ple. And when you look at the list of No­bel Prize win­ners over the years, the ten­dency to­wards giv­ing out more and more shared prizes as time goes on is ob­vi­ous.

In fact, given the way sci­ence cur­rently works, many peo­ple find the rule rule that no more than three peo­ple can share a prize too re­stric­tive. The No­bel for the dis­cov­ery of the Higgs Bo­son, for ex­am­ple, went to two the­o­ret­i­ci­ans who pre­dicted the par­ti­cle decades ago, while ig­nor­ing the con­tri­bu­tions of the large num­ber of ex­per­i­men­tal sci­en­tists whose work was re­quired to con­firm the par­ti­cle’s ex­is­tence. An IEEE Spec­trum head­line went as far as to state the prize “ig­nores how mod­ern sci­ence works.”

You can reach the same con­clu­sion just look­ing at the bylines on sci­en­tific pa­pers. The sin­gle-au­thor sci­en­tific pa­per “has all but dis­ap­peared.” Some of that may be due to peo­ple gam­ing the cita­tion-count-as-mea­sure-of-sci­en­tific-pro­duc­tivity sys­tem, but my im­pres­sion is that the typ­i­cal uni­ver­sity sci­ence lab’s PI (prin­ci­ple in­ves­ti­ga­tor) re­ally couldn’t be nearly as pro­duc­tive with­out their mi­ni­a­ture army of post­docs, grad stu­dents, and paid staff. (Con­sider also that gam­ing of cita­tion counts hasn’t led to an ex­plo­sion of au­thors-per-pa­per in fields like philos­o­phy, where there are ob­vi­ously fewer benefits to col­lab­o­ra­tion.)

And if you need one more ar­gu­ment that sci­en­tific prob­lems are get­ting harder, and in­creas­ingly un­likely to be solved by lone ge­niuses… what does any­one hon­estly think the chances are that the Next Big Thing in sci­ence will come in the form some 26 year old pub­lish­ing a few sin­gle-au­thor pa­pers in the same year he got his PhD?

Up­date: Luke’s com­ments on this post are awe­some and I recom­mend peo­ple read them.