[Altruist Support] LW Go Foom

In which I worry that the Less Wrong pro­ject might go hor­ribly right. This post be­longs to my Altru­ist Sup­port se­quence.

Every pro­ject needs a risk as­sess­ment.

There’s a feel­ing, just bub­bling un­der the sur­face here at Less Wrong, that we’re just play­ing at ra­tio­nal­ity. It’s ra­tio­nal­ity kinder­garten. The prob­lem has been ex­pressed in var­i­ous ways:

And peo­ple are start­ing to look at fix­ing it. I’m not wor­ried that their at­tempts—and mine—will fail. At least we’d have fun and learn some­thing.

I’m wor­ried that they will suc­ceed.

What would such a Su­per Less Wrong com­mu­nity do? Its mem­bers would self-im­prove to the point where they had a good chance of suc­ceed­ing at most things they put their mind to. They would re­cruit new ra­tio­nal­ists and then op­ti­mize that re­cruit­ment pro­cess, un­til the com­mu­nity got big. They would de­velop meth­ods for rapidly gen­er­at­ing, clas­sify­ing and eval­u­at­ing ideas, so that the only ideas that got tried would be the best that any­one had come up with so far. The group would struc­ture it­self so that peo­ple’s ba­sic so­cial drives—such as their de­sire for sta­tus—worked in the in­ter­ests of the group rather than against it.

It would be pretty formidable.

What would the prod­ucts of such a com­mu­nity be? There would prob­a­bly be a self-help book that works. There would be an effec­tive, prac­ti­cal guide to set­ting up effec­tive com­mu­ni­ties. There would be an in­tu­itive, prac­ti­cal guide to hu­man be­hav­ior. There would be books, sem­i­nars and classes on how to re­ally achieve your goals—and only the ma­te­ri­als which ac­tu­ally got re­sults would be kept. There would be a bunch of stuff on the Dark Arts too, no doubt. Pos­si­bly some AI re­search.

That’s a whole lot of ma­te­rial that we wouldn’t want to get into the hands of the wrong peo­ple.

Dangers in­clude:

  • Half-ra­tio­nal­ists: peo­ple who pick up on enough memes to be re­ally dan­ger­ous, but not on enough to re­al­ise that what they’re do­ing might be fool­ish. For ex­am­ple, build­ing an AI with­out adding the friendli­ness fea­tures.

  • Ra­tion­al­ists with bad goals: Some­one could ra­tio­nally set about try­ing to de­stroy hu­man­ity, just for the lulz.

  • Danger­ous in­for­ma­tion dis­cov­ered: e.g. the ra­tio­nal­ist com­mu­nity de­vel­ops a The­ory of Every­thing that re­veals a recipe for a physics dis­aster (e.g. a cheap way to turn the Earth into a block hole). A non-ra­tio­nal­ist de­cides to ex­ploit this.

If this is a prob­lem we should take se­ri­ously, what are some pos­si­ble strate­gies for deal­ing with it?

  1. Just go ahead and ig­nore the is­sue.

  2. The Bayesian Con­spir­acy: only those who can be trusted are al­lowed ac­cess to the se­cret knowl­edge.

  3. The Good Word: mix in ra­tio­nal­ist ideas with do-good and stay-safe ideas, to the ex­tent that they can’t be eas­ily sep­a­rated. The idea be­ing that any­one who un­der­stands ra­tio­nal­ity will also un­der­stand that it must be used for good.

  4. Ra­tion­al­ity cap: we de­velop enough ra­tio­nal­ity to achieve our goals (e.g. friendly AI) but de­liber­ately stop short of de­vel­op­ing the ideas too far.

  5. Play at ra­tio­nal­ity: cre­ate a com­mu­nity which ap­pears ra­tio­nal enough to dis­tract peo­ple who are that way in­clined, but which does not dra­mat­i­cally in­crease their per­sonal effec­tive­ness.

  6. Risk man­age­ment: ac­cept that each new idea has a po­ten­tial pay­off (in terms of helping us avoid ex­is­ten­tial threats) and a po­ten­tial cost (in terms of helping “bad ra­tio­nal­ists”). Im­ple­ment the ideas which come out pos­i­tive.

In the post ti­tle, I have sug­gested an anal­ogy with AI take­off. That’s not en­tirely fair; there is prob­a­bly an up­per bound to how effec­tive a com­mu­nity of hu­mans can be, at least un­til brain im­plants come along. We’re prob­a­bly talk­ing two or­ders of mag­ni­tude rather than ten. But given that hu­man­ity already has tech­nol­ogy with slight ex­is­ten­tial threat im­pli­ca­tions (nu­clear weapons, rudi­men­tary AI re­search), I would be wor­ried about a move­ment that aims to make all of hu­man­ity more effec­tive at ev­ery­thing they do.