# Forum participation as a research strategy

Re­cently I’ve no­ticed a cog­ni­tive dis­so­nance in my­self, where I can see that my best ideas have come from par­ti­ci­pat­ing on var­i­ous mailing lists and fo­rums (such as cypher­punks, ex­tropi­ans, SL4, ev­ery­thing-list, LessWrong and AI Align­ment Fo­rum), and I’ve re­ceived a cer­tain amount of recog­ni­tion as a re­sult, but when some­one asks me what I ac­tu­ally do as an “in­de­pen­dent re­searcher”, I’m em­bar­rassed to say that I mostly com­ment on other peo­ple’s posts, par­ti­ci­pate in on­line dis­cus­sions, and oc­ca­sion­ally a new idea pops into my head and I write it down as a blog/​fo­rum post of my own. I guess that’s be­cause I imag­ine it doesn’t fit most peo­ple’s image of what a re­searcher’s work con­sists of.

Once I no­ticed this, the ten­sion is easy to re­solve—in this post I’m go­ing to pro­claim/​en­dorse fo­rum par­ti­ci­pa­tion (aka com­ment­ing) as a pro­duc­tive re­search strat­egy that I’ve man­aged to stum­ble upon, and recom­mend it to oth­ers (at least to try). Note that this is differ­ent from say­ing that fo­rum/​blog posts are a good way for a re­search com­mu­nity to com­mu­ni­cate. It’s about in­di­vi­d­u­ally do­ing bet­ter as re­searchers.

# Benefits of Fo­rum Par­ti­ci­pa­tion (FP)

## FP takes lit­tle effort /​ will power

In other words it feels more like play than work, which means I rarely have is­sues with not want­ing to do some­thing that I think is im­por­tant to do (i.e., akra­sia), the only ex­cep­tion be­ing that writ­ing posts seems to take more effort so oc­ca­sion­ally I spend my time writ­ing com­ments when I per­haps should write posts in­stead. (This is the part of this post that I think may be least likely to gen­er­al­ize to other peo­ple. It could be that I’m an ex­treme out­lier in find­ing FP so low-effort. How­ever it might also be the case that it be­comes low effort for most peo­ple to write com­ments once they’ve had enough prac­tice in it.)

## FP is a good way to no­tice miss­ing back­ground knowl­edge and pro­vides in­cen­tives to learn miss­ing knowledge

If you read a post with an in­ten­tion to ques­tion or com­ment on it, it’s pretty easy to no­tice that it as­sumes some back­ground knowl­edge that you lack. The de­sire to not ask a “stupid” ques­tion or make a “stupid” com­ment pro­vides pow­er­ful in­cen­tive to learn the miss knowl­edge.

## FP is a good way to stay up to date on ev­ery­one else’s lat­est research

It’s of­ten a good idea to stay up to date on other peo­ple’s re­search, but some­times one isn’t highly mo­ti­vated to do so. FP seems to make that eas­ier. For ex­am­ple, I wasn’t fol­low­ing Stu­art’s re­search on coun­ter­fac­tual or­a­cles, un­til the re­cent con­test drew my at­ten­tion and de­sire to par­ti­ci­pate, and I ended up read­ing the lat­est posts on CO in or­der to un­der­stand the cur­rent state of the art on that topic, which turned out to be pretty in­ter­est­ing.

## Ar­gu­ments that are gen­er­ated in re­ac­tion to some spe­cific post or dis­cus­sion can be of gen­eral value

It’s not in­fre­quent that I come up with an ar­gu­ment in re­sponse to some post or dis­cus­sion thread, and later ex­pand or fol­low up that ar­gu­ment into a post be­cause it seems to ap­ply more gen­er­ally than to just that post/​dis­cus­sion. Here is one such ex­am­ple.

## FP gen­er­ates new ideas via cross-fertilization

FP in­cen­tivizes one to think deeply about many threads of re­search, and of­ten (at least for me) an idea pops into my head that seems to com­bine var­i­ous par­tial ideas float­ing in the ether into a co­her­ent or semi-co­her­ent whole (e.g., UDT), or is the re­sult of ap­ply­ing or analo­giz­ing some­one else’s lat­est idea to a differ­ent topic (e.g., “hu­man safety prob­lem”, “philos­o­phy as high com­plex­ity class”).

## FP helps pre­pare for effi­ciently com­mu­ni­cat­ing new ideas

FP is a good way to build mod­els of other peo­ple’s epistemic states, and also a good way to prac­tice com­mu­ni­cat­ing with fel­low re­searchers, both of which are good prepa­ra­tion for effi­ciently com­mu­ni­cat­ing one’s own new ideas.

# My Recommendations

## Com­ment more

To ob­tain the above benefits, one just has to write more com­ments. It may be nec­es­sary to first over­come dis­in­cen­tives to par­ti­ci­pate. If you can’t, please speak up and maybe the fo­rum ad­mins will do some­thing to help ad­dress what­ever ob­sta­cle you’re hav­ing trou­ble with.

## Prac­tice makes better

If it seems hard to write good com­ments, prac­tice might make it eas­ier even­tu­ally.

## Think of FP as some­thing to do for yourself

Some peo­ple might think of com­ment­ing as pri­mar­ily pro­vid­ing a ser­vice to other re­searchers or to the re­search com­mu­nity. I sug­gest also think­ing of it as pro­vid­ing a benefit to your­self (for the above rea­sons).

## En­courage and sup­port re­searchers who adopt FP as their pri­mary re­search strategy

I’m not aware of any or­ga­ni­za­tions that ex­plic­itly en­courage and sup­port re­searchers to spend most or much of their time com­ment­ing on fo­rum posts. But per­haps they should, if it ac­tu­ally is (or has the po­ten­tial to be) a pro­duc­tive re­search strat­egy? For ex­am­ple this could be done by pro­vid­ing fi­nan­cial sup­port and/​or sta­tus re­wards for effec­tive fo­rum par­ti­ci­pa­tion.

• I broadly agree, but there are good rea­sons for more tra­di­tional re­search as well:

• In many re­search ar­eas, ideas are com­mon, and it isn’t clear which ideas are most im­por­tant. The most use­ful con­tri­bu­tions come from some­one tak­ing an idea and demon­strat­ing that it is vi­able and im­por­tant, which of­ten re­quires a lot of soli­tary work that can’t be done in the typ­i­cal amount of time it takes to write a com­ment or post.

• FP of­ten leads to long, wind­ing dis­cus­sions that may end with two re­searchers agree­ing, but the re­sult­ing tran­script is not great for fu­ture read­ers. In con­trast, tra­di­tional re­search pro­duces more crisp dis­til­la­tions of an idea that are use­ful for com­mu­ni­cat­ing with an en­tire field. (I an­ti­ci­pate peo­ple say­ing that aca­demic pa­pers are in­com­pre­hen­si­ble. They are in­com­pre­hen­si­ble to out­siders, but of­ten are rel­a­tively easy to read to peo­ple in the field. I typ­i­cally find aca­demic pa­pers in my area to be sig­nifi­cantly bet­ter at tel­ling me what I want to know than blog posts, though the ideal com­bi­na­tion is to first read the blog post and then fol­low it up with the pa­per.)

Nev­er­the­less, I do think that FP is a bet­ter strat­egy for in­tel­lec­tual progress in AI al­ign­ment, which at least cur­rently feels more ideas-bot­tle­necked than pa­per-bot­tle­necked.

• In many re­search ar­eas, ideas are com­mon, and it isn’t clear which ideas are most im­por­tant. The most use­ful con­tri­bu­tions come from some­one tak­ing an idea and demon­strat­ing that it is vi­able and im­por­tant, which of­ten re­quires a lot of soli­tary work that can’t be done in the typ­i­cal amount of time it takes to write a com­ment or post.

Agreed. My recom­men­da­tions aren’t meant to be uni­ver­sally ap­pli­ca­ble. (ETA: Also, one could marginally in­crease one’s fo­rum par­ti­ci­pa­tion in or­der to cap­ture some of the benefits, and not nec­es­sar­ily go all the way to adopt­ing it as one’s pri­mary re­search strat­egy.)

FP of­ten leads to long, wind­ing dis­cus­sions that may end with two re­searchers agree­ing, but the re­sult­ing tran­script is not great for fu­ture read­ers.

There’s noth­ing that ex­plic­itly pre­vents peo­ple from dis­till­ing such dis­cus­sions into sub­se­quent posts or pa­pers. If peo­ple aren’t do­ing that, or are do­ing that less than they should, that could po­ten­tially be solved as a prob­lem that’s sep­a­rate from “should more peo­ple be do­ing FP or tra­di­tional re­search?”

Also, it’s not clear to me that tra­di­tional re­search pro­duces more clear dis­til­la­tions of how dis­agree­ments get re­solved. It seems like most such dis­cus­sions don’t make for pub­lish­able pa­pers and there­fore most dis­agree­ments be­tween “tra­di­tional re­searchers” just don’t get re­solved in a way that leaves a pub­lic record (or at all).

• There’s noth­ing that ex­plic­itly pre­vents peo­ple from dis­till­ing such dis­cus­sions into sub­se­quent posts or pa­pers. If peo­ple aren’t do­ing that, or are do­ing that less than they should, that could po­ten­tially be solved as a prob­lem that’s sep­a­rate from “should more peo­ple be do­ing FP or tra­di­tional re­search?”

FYI this is some­thing the LW team thinks about a bunch and I ex­pect us to have made some se­ri­ous effort to­wards in­cen­tiviz­ing and sim­plify­ing this pro­cess in the com­ing year.

• There’s noth­ing that ex­plic­itly pre­vents peo­ple from dis­till­ing such dis­cus­sions into sub­se­quent posts or pa­pers. If peo­ple aren’t do­ing that, or are do­ing that less than they should, that could po­ten­tially be solved as a prob­lem that’s sep­a­rate from “should more peo­ple be do­ing FP or tra­di­tional re­search?”

Agreed. I’m mostly say­ing that em­piri­cally peo­ple don’t do that, but yes there could be other solu­tions to the prob­lem, it need not be in­her­ent to FP.

Also, it’s not clear to me that tra­di­tional re­search pro­duces more clear dis­til­la­tions of how dis­agree­ments get re­solved.

I agree you don’t see how the dis­agree­ment gets re­solved, but you usu­ally can see the an­swer to the ques­tion that prompted the dis­agree­ment, be­cause the re­s­olu­tion it­self can be turned into a pa­per. This is as­sum­ing that the re­s­olu­tion came via new ev­i­dence. I agree that if a dis­agree­ment is re­solved via sim­ply talk­ing through the ar­gu­ments, then it doesn’t turn into a pa­per, but this seems pretty rare (at least in CS).

• In many re­search ar­eas, ideas are com­mon, and it isn’t clear which ideas are most im­por­tant. The most use­ful con­tri­bu­tions come from some­one tak­ing an idea and demon­strat­ing that it is vi­able and im­por­tant, which of­ten re­quires a lot of soli­tary work that can’t be done in the typ­i­cal amount of time it takes to write a com­ment or post.

In­ter­est­ing. Definitely not an ex­pert here, but I could imag­ine FP be­ing a good tool in this case… if the fo­rum is an effi­cient “mar­ket­place of ideas”, where per­spec­tives com­pete and poke holes in each other and adapt to crit­ics, and the strongest per­spec­tives emerge vic­to­ri­ous, then this seems like it could be a good way to figure out which ideas are the best? Some say AI al­ign­ment is like soft­ware se­cu­rity, and there’s that say­ing “with enough eyes all bugs are shal­low”. If se­cu­rity flaws tend to be a re­sult of soft­ware de­sign­ers rely­ing on faulty ab­strac­tions or oth­er­wise fal­ling prey to blind spots, then I would ex­pect that with­stand­ing a bunch of crit­ics, each critic us­ing their own set of ab­strac­tions, is a stronger in­di­ca­tor of qual­ity than any­thing one per­son is able to do in soli­tude.

(It’s pos­si­ble that you’re us­ing “im­por­tant” in a way that’s differ­ent than how I used it in the pre­ced­ing para­graph.)

• FP works well when it is easy to make progress on ideas /​ ques­tions through arm­chair rea­son­ing, which you can think of as us­ing the in­for­ma­tion or ev­i­dence you have more effi­ciently. How­ever, it is of­ten the case that you could get a lot more high-qual­ity ev­i­dence that ba­si­cally set­tles the ques­tion, if you put in many hours of work. As an illus­tra­tive ex­am­ple, con­sider try­ing to es­ti­mate the pop­u­la­tion of a city via FP, vs. go­ing out and do­ing a cen­sus. In ML, you could say “we should add such-and-such in­duc­tive bias to our mod­els so that they learn faster”, and we can de­bate how much that would help, but if you ac­tu­ally build it in and train the model and see what hap­pens, you just know the an­swer now.

• Hm, you think data soundly beats the­ory in ML? Why is HARKing a prob­lem then?

• HARKing does the right two steps in the wrong or­der—it first gets the data, and then makes the hy­poth­e­sis. This is fine for gen­er­at­ing hy­pothe­ses, but isn’t great for tel­ling whether a hy­poth­e­sis is true or not, be­cause there are likely many hy­pothe­ses that ex­plain the data and it’s not clear why the one you chose should be the right one. It’s much stronger ev­i­dence if you first have a hy­poth­e­sis, and then de­sign a test for it, be­cause then there is only one re­sult out of many pos­si­ble re­sults that con­firms your hy­poth­e­sis. (This is over­sim­plified, but cap­tures the broad point.)

I wouldn’t say “data beats the­ory”, I think the­ory (in the sense of “some way of pre­dict­ing which ideas will be good”, not nec­es­sar­ily math) is needed in or­der to figure out which ideas to bother test­ing in the first place. But if you are eval­u­at­ing on “what gives me con­fi­dence that <hy­poth­e­sis> is true”, it’s usu­ally go­ing to be data. The­o­rems could do it, but it seems pretty rare that there are the­o­rems for ac­tu­ally in­ter­est­ing hy­pothe­ses.

• I ac­tu­ally think there is an in­ter­est­ing philo­soph­i­cal puz­zle around this that has not fully been solved...

If I show you the code I’m go­ing to use to run my ex­per­i­ment, can you be con­fi­dent in guess­ing which hy­poth­e­sis I aim to test?

• If yes, then HARKing should be eas­ily de­tectable. By look­ing at my code, it should be clear that the hy­poth­e­sis I am ac­tu­ally test­ing is not the one that I pub­lished.

• If no, then the re­sult­ing data could be used to prove mul­ti­ple differ­ent hy­pothe­ses, and thus doesn’t nec­es­sar­ily con­sti­tute stronger ev­i­dence for any one of the par­tic­u­lar hy­pothe­ses it could be used to prove (e.g. the hy­poth­e­sis I pre­reg­istered).

To put it an­other way, in your first para­graph you say “there are likely many hy­pothe­ses that ex­plain the data”, but in the sec­ond para­graph, you talk as though there’s a par­tic­u­lar set of data such that if we get that data, we know there’s only one hy­poth­e­sis which it can be used to sup­port! What gives?

My solu­tion to the puz­zle: Pre-reg­is­tra­tion works be­cause it forces re­searchers to be hon­est about their prior knowl­edge. Ba­si­cally, prior knowl­edge un­en­cum­bered by hind­sight bias (“arm­chair rea­son­ing”) is un­der­rated. Any hy­poth­e­sis which has only the sup­port of arm­chair rea­son­ing or data from a sin­gle ex­per­i­ment is sus­pect. You re­ally want both.

In Bayesian terms, you have to look at both the prior and the like­li­hood. Order shouldn’t mat­ter (mul­ti­pli­ca­tion is com­mu­ta­tive), but as I said—hind­sight bias.

[There are also cases where given the data, there’s only one plau­si­ble hy­poth­e­sis which could pos­si­bly ex­plain it. A well-de­signed ex­per­i­ment will hope­fully pro­duce data like this, but I think it’s a bit or­thog­o­nal to the HARKing is­sue, be­cause we can imag­ine sce­nar­ios where post hoc data anal­y­sis sug­gests there is only one plau­si­ble hy­poth­e­sis for what’s go­ing on… al­though we should still be sus­pi­cious in that case be­cause (pre­sum­ably) we didn’t have prior be­liefs in­di­cat­ing this hy­poth­e­sis was likely to be true. Note that in both cases we are bot­tle­necked on the cre­ativity of the ex­per­i­ment de­signer/​data an­a­lyst in think­ing up al­ter­na­tive hy­pothe­ses.]

[BTW, I think “arm­chair rea­son­ing” might have the same refer­ent as phrases with a more pos­i­tive con­no­ta­tion: “de­con­fu­sion work” or “re­search dis­til­la­tion”.]

• My solu­tion to the puz­zle is a bit differ­ent (but maybe the same?) Let’s sup­pose that there’s an ex­per­i­ment we could run that would come out with some re­sult . Each po­ten­tial value of is con­sis­tent with hy­pothe­ses. There are po­ten­tial hy­pothe­ses.

Sup­pose Alice runs the ex­per­i­ment and then chooses a hy­poth­e­sis to ex­plain it. This is con­sis­tent with Alice hav­ing a uniform prior, in which case she has a chance of hav­ing set­tled on the true hy­poth­e­sis. (Why not just list all hy­pothe­ses? Be­cause Alice didn’t think of all of them—it’s hard to search the en­tire space of hy­pothe­ses.)

On the other hand, if Bob chose a hy­poth­e­sis to test via his pri­ors, ran the ex­per­i­ment, and then was con­sis­tent with that hy­poth­e­sis… you should in­fer that Bob’s pri­ors were re­ally good (i.e. not uniform) and the hy­poth­e­sis is cor­rect. After all, if Bob’s hy­poth­e­sis was cho­sen at ran­dom, he only had a chance of get­ting a that was con­sis­tent with it.

Put an­other way: When I see the first sce­nario, I ex­pect that the ev­i­dence gath­ered from the ex­per­i­ment is pri­mar­ily serv­ing to lo­cate the hy­poth­e­sis at all. When I see the sec­ond sce­nario, I ex­pect that Bob has already suc­cess­fully lo­cated the hy­poth­e­sis be­fore the ex­per­i­ment, and the ex­per­i­ment pro­vides the last lit­tle bit of ev­i­dence needed to con­firm it.

Re­lated: Priv­ileg­ing the hypothesis

If I show you the code I’m go­ing to use to run my ex­per­i­ment, can you be con­fi­dent in guess­ing which hy­poth­e­sis I aim to test?

Un­der this model, I can’t be con­fi­dent in guess­ing which hy­poth­e­sis you are try­ing to test.

My solu­tion to the puz­zle: Pre-reg­is­tra­tion works be­cause it forces re­searchers to be hon­est about their prior knowl­edge. Ba­si­cally, prior knowl­edge un­en­cum­bered by hind­sight bias (“arm­chair rea­son­ing”) is un­der­rated.

It’s pos­si­ble that Alice her­self be­lieves that the hy­poth­e­sis she set­tled on was cor­rect, rather than as­sign­ing it a prob­a­bil­ity. If that were the case, I would say it was due to hind­sight bias.

Any hy­poth­e­sis which has only the sup­port of arm­chair rea­son­ing or data from a sin­gle ex­per­i­ment is sus­pect.

Yeah, I broadly agree with this.

[BTW, I think “arm­chair rea­son­ing” might have the same refer­ent as phrases with a more pos­i­tive con­no­ta­tion: “de­con­fu­sion work” or “re­search dis­til­la­tion”.]

I definitely do not mean re­search dis­til­la­tion. De­con­fu­sion work feels like a sep­a­rate thing, which is usu­ally a par­tic­u­lar ex­am­ple of arm­chair rea­son­ing. By arm­chair rea­son­ing, I mean any sort of rea­son­ing that can be done by just think­ing with­out gath­er­ing more data. So for ex­am­ple, solv­ing a thorny al­gorithms ques­tion would in­volve arm­chair rea­son­ing.

I don’t mean to in­clude the nega­tive con­no­ta­tions of “arm­chair rea­son­ing”, but I don’t know an­other short phrase that means the same thing.

• In­ter­est­ing. I think you’re prob­a­bly right that our model should have a pa­ram­e­ter for “re­searcher qual­ity”, and if a re­searcher is able to cor­rectly pre­dict the out­come of an ex­per­i­ment, that should cause an up­date in the di­rec­tion of that re­searcher be­ing more knowl­edgable (and their prior judge­ments should there­fore carry more weight—in­clud­ing for this par­tic­u­lar ex­per­i­ment!)

But the story you’re tel­ling doesn’t seem en­tirely com­pat­i­ble with your com­ment ear­lier in this thread. Ear­lier you wrote: “How­ever, it is of­ten the case that you could get a lot more high-qual­ity ev­i­dence that ba­si­cally set­tles the ques­tion, if you put in many hours of work.” But in this re­cent com­ment you wrote: “the ex­per­i­ment pro­vides the last lit­tle bit of ev­i­dence needed to con­firm [the hy­poth­e­sis]”. In the ear­lier com­ment, it sounds like you’re talk­ing about a sce­nario where most of the ev­i­dence comes in the form of data; in the later com­ment, it sounds like you’re talk­ing about a sce­nario where most of the ev­i­dence was nec­es­sary “just to think of the cor­rect an­swer—to pro­mote it to your at­ten­tion” and the ex­per­i­ment only pro­vides “the last lit­tle bit” of ev­i­dence.

So I think the philo­soph­i­cal puz­zle is still un­solved. A few more things to pon­der if some­one wants to work on solv­ing it:

• If Bob is known to be an ex­cel­lent re­searcher, can we trust HARKing if it comes from him? Does the mechanism by which hind­sight bias works mat­ter? (Here is one pos­si­ble mechanism.)

• In your sim­plified model above, there’s no pos­si­bil­ity of a re­sult that is “just noise” and not ex­plained by any par­tic­u­lar hy­poth­e­sis. But noise ap­pears to be a pretty big prob­lem (see: repli­ca­tion crisis?) In cur­rent sci­en­tific prac­tice, the prob­a­bil­ity that a re­sult is “just noise” is a num­ber of great in­ter­est that’s al­most always calcu­lated (the p-value). How should this num­ber be fac­tored in, if at all?

• Note that p-val­ues can be used in Bayesian calcu­la­tions. For ex­am­ple, in a sim­plified uni­verse where ei­ther the null is true or the al­ter­na­tive is true, p(al­ter­na­tive|data) = p(data|al­ter­na­tive)p(al­ter­na­tive) /​ (p(data|al­ter­na­tive)p(al­ter­na­tive) + p(data|null)p(null))

• My solu­tion was fo­cused on a sce­nario where we’re con­sid­er­ing rel­a­tively ob­vi­ous hy­pothe­ses and sub­ject to lots of mea­sure­ment noise, but you con­vinced me this is in­ad­e­quate in gen­eral.

• I’m un­satis­fied with the dis­cus­sion around “Alice didn’t think of all of them”. I know noth­ing about rel­a­tivity, but I imag­ine a big part of Ein­stein’s con­tri­bu­tion was his dis­cov­ery of a rel­a­tively sim­ple hy­poth­e­sis which ex­plained all the data available to him. (By “rel­a­tively sim­ple”, I mean a hy­poth­e­sis that didn’t have hun­dreds of free pa­ram­e­ters.) Pre­sum­ably, Ein­stein had ac­cess to the same data as other con­tem­po­rary physi­cists, so it feels weird to ex­plain his con­tri­bu­tion in terms of hav­ing ac­cess to more ev­i­dence.

• In other words, it feels like the task of search­ing hy­poth­e­sis space should be fac­tored out from the task of Bayesian up­dat­ing. This seems closely re­lated to puz­zles around “re­al­iz­abil­ity”—through your search of hy­poth­e­sis space, you’re es­sen­tially “re­al­iz­ing” a par­tic­u­lar hy­poth­e­sis on the fly, which isn’t how Bayesian up­dat­ing is for­mally sup­posed to work. (But it is how deep learn­ing works, for ex­am­ple.)

• But the story you’re tel­ling doesn’t seem en­tirely com­pat­i­ble with your com­ment ear­lier in this thread.

The ear­lier com­ment was com­par­ing ex­per­i­ments to “arm­chair rea­son­ing”, while the later com­ment was com­par­ing ex­per­i­ments to “all prior knowl­edge”. I think the typ­i­cal case is:

Amount of ev­i­dence in “all prior knowl­edge” >> Amount of ev­i­dence in an ex­per­i­ment >> Amount of ev­i­dence from “arm­chair rea­son­ing”.

If Bob is known to be an ex­cel­lent re­searcher, can we trust HARKing if it comes from him?

I would pay a lit­tle more at­ten­tion, but not that much more, and would want an ex­per­i­men­tal con­fir­ma­tion any­way. It seems to be that the world is com­plex enough and hu­mans model it badly enough (for the sorts of things academia is look­ing at) that past ev­i­dence of good pri­ors on one ques­tion doesn’t im­ply good pri­ors on a differ­ent ques­tion.

(This is an em­piri­cal be­lief; I’m not con­fi­dent in it.)

In your sim­plified model above, there’s no pos­si­bil­ity of a re­sult that is “just noise” and not ex­plained by any par­tic­u­lar hy­poth­e­sis.

I ex­pect that if you made a more com­pli­cated model where each hy­poth­e­sis had a like­li­hood , and was high for hy­pothe­ses and low for the rest, you’d get a similar con­clu­sion, while ac­count­ing for re­sults that are just noise.

I know noth­ing about rel­a­tivity, but I imag­ine a big part of Ein­stein’s con­tri­bu­tion was his dis­cov­ery of a rel­a­tively sim­ple hy­poth­e­sis which ex­plained all the data available to him.

I agree that rel­a­tivity is an ex­am­ple that doesn’t fit my story, where most of the work was in com­ing up with the hy­poth­e­sis. (Though I sus­pect you could ar­gue that rel­a­tivity shouldn’t have been be­lieved be­fore ex­per­i­men­tal con­fir­ma­tion.) I claim that it is the ex­cep­tion, not the rule.

Also, I do think it is of­ten a valuable con­tri­bu­tion to even think of a plau­si­ble hy­poth­e­sis that fits the data, even if you should as­sign it a rel­a­tively low prob­a­bil­ity of be­ing true. I’m just say­ing that if you want to reach the truth, this work must be sup­ple­mented by ex­per­i­ments /​ gath­er­ing good data.

In other words, it feels like the task of search­ing hy­poth­e­sis space should be fac­tored out from the task of Bayesian up­dat­ing.

Bayesian up­dat­ing does not work well when you don’t have the full hy­poth­e­sis space. Given that you know that you don’t have the full hy­poth­e­sis space, you should not be try­ing to ap­prox­i­mate Bayesian up­dat­ing over the hy­poth­e­sis space you do have.

• Bayesian up­dat­ing does not work well when you don’t have the full hy­poth­e­sis space.

Do you have any links re­lated to this? Tech­ni­cally speak­ing, the right hy­poth­e­sis is al­most never in our hy­poth­e­sis space (“All mod­els are wrong, but some are use­ful”). But even if there’s no “use­ful” model in your hy­poth­e­sis space, it seems Bayesian up­dat­ing fails grace­fully if you have a rea­son­ably wide prior dis­tri­bu­tion for your noise pa­ram­e­ters as well (then the model fit­ting pro­cess will con­clude that the value of your noise pa­ram­e­ter must be high).

• Do you have any links re­lated to this?

No, I haven’t read much about Bayesian up­dat­ing. But I can give an ex­am­ple.

Con­sider the fol­low­ing game. I choose a coin. Then, we play N rounds. In each round, you make a bet about whether or not the coin will come up Heads or Tails at 1:2 odds which I must take (i.e. if you’re right I give you $2 and if I’m right you give me$1). Then I flip the coin and the bet re­solves.

If your hy­poth­e­sis space is “the coin has some bias b of com­ing up Heads or Tails”, then you will ea­gerly ac­cept this game for large enough N—you will quickly learn the bias b from ex­per­i­ments, and then you can keep get­ting money in ex­pec­ta­tion.

How­ever, if it turns out I am ca­pa­ble of mak­ing the coin come up Heads or Tails as I choose, then I will win ev­ery round. If you keep do­ing Bayesian up­dat­ing on your mis­speci­fied hy­poth­e­sis space, you’ll keep flip-flop­ping on whether the bias is to­wards Heads or Tails, and you will quickly con­verge to near-cer­tainty that the bias is 50% (since the pat­tern will be HTHTHTHT...), and yet I will be tak­ing a dol­lar from you ev­ery round. Even if you have the op­tion of quit­ting, you will never ex­er­cise it be­cause you keep think­ing that the EV of the next round is pos­i­tive.

Noise pa­ram­e­ters can help (though the bias b is kind of like a noise pa­ram­e­ter here, and it didn’t help). I don’t know of a gen­eral way to use noise pa­ram­e­ters to avoid is­sues like this.

• Thanks for the ex­am­ple!

• John Maxwell and I went back and forth on this ques­tion a bunch. I was ini­tially on the con side and up­dated in the di­rec­tion that in­creas­ing lurker ra­tios and com­mon knowl­edge gen­er­a­tion are big enough con­sid­er­a­tions that fo­rum par­ti­ci­pa­tion is prob­a­bly pretty good/​helpful. He was ini­tially on the pro side and I think up­dated in the di­rec­tion that with­out an an­chor of schol­ar­ship (text­books, re­search re­view, and struc­tured notes) and longer/​more in depth con­ver­sa­tions on­line ac­tivity can ac­quire a ve­neer of pro­duc­tivity that is harm­ful to both real pro­duc­tivity and men­tal health.

• Can you sum­ma­rize the ev­i­dence/​ar­gu­ments that were brought up dur­ing your dis­cus­sions?

• Maybe I re­mem­ber the con­ver­sa­tion differ­ently than Romeo, be­cause I re­mem­ber be­ing on the pro side.

Par­ti­ci­pa­tion in­equal­ity is a thing. Here is one es­ti­mate for LW. Here is a thread from 2010 where Kevin asked peo­ple to delurk which has over 600 com­ments in it. Anec­do­tally, I’m no longer sur­prised by ex­pe­riences like:

• Friend I didn’t know very well re­cently came to visit. Says my ideas about AI are in­ter­est­ing. Friend is just learn­ing to pro­gram. I didn’t even know they read LW, much less that they knew any­thing about my AI ideas.

• Write a com­ment on the EA Fo­rum. Go to an event that evening. I talk to some­one and they’re like “oh I saw your com­ment on the EA Fo­rum and it was good”.

• I vis­ited Europe and met an EA from the Czech Repub­lic. He says: “I’ve prob­a­bly read more words by you than by William MacAskill be­cause you post to the EA Fo­rum so much.”

My im­pres­sion is this is a con­trast to academia, where vir­tu­ally no one reads most aca­demic pub­li­ca­tions. I sus­pect ac­quiring an on­line fol­low­ing al­lows for greater to­tal in­fluence than as­cend­ing the aca­demic lad­der, though it’s prob­a­bly a differ­ent kind of in­fluence (peo­ple are less likely to cite your work years later? In the spirit of that, here is a LW thread from years ago on pa­pers vs fo­rums.)

How­ever:

• With great power comes great re­spon­si­bil­ity. If you are speak­ing to a big au­di­ence, spread­ing bad info is harm­ful. And con­tra XKCD, de­bunk­ing bad info is valuable. (Note that al­most ev­ery­thing started go­ing to shit on­line within 1 decade of this comic be­com­ing fa­mous.)

• I agree that the “an­chor of schol­ar­ship” is valuable. I view FP as a guilty plea­sure/​struc­tured pro­cras­ti­na­tion. I don’t nec­es­sar­ily en­dorse it as the high­est value ac­tivity, but if I’m go­ing to be goofing off any­way it’s a rel­a­tively use­ful way to goof off, and some­times it feels like my con­tri­bu­tions are pretty valuable/​I get valuable ideas from oth­ers. (I sus­pect the best fo­rum writ­ing is not only aca­dem­i­cally rigor­ous and in­no­va­tive, but also en­ter­tain­ing to read, with click­baity ti­tles and in­ter­est­ing anec­dotes and such, so you can cap­ture the cog­ni­tive sur­plus of fo­rum users as effec­tively as pos­si­ble. Ad­di­tion­ally, the best ques­tions to ask fo­rums are prob­a­bly those that benefit from crowd­sourc­ing/​mul­ti­ple per­spec­tives/​cre­ativity/​etc.)

Seems like the ideal is a bal­ance be­tween FP and schol­ar­ship. If you’re all FP, you’re likely just a clue­less per­son spread­ing clue­less­ness. If you’re all schol­ar­ship, oth­ers don’t get to share in your wis­dom much.

• Right, I’m say­ing I started off con, you started off pro, and both of us nudged to­wards the other along the di­men­sions men­tioned. I prob­a­bly up­dated more than you.

• Which is 100% how I ini­tially read your com­ment, but if you look at your com­ment it ac­tu­ally says you were both con :-)

• thanks!

• This was more than 2 years ago so un­for­tu­nately I don’t re­call.

• This is one of those “so ob­vi­ous when some­one says it” things that are not at all ob­vi­ous un­til some­one says them. Well done!

• I would note that many of these fac­tors ap­ply as benefits of office-chat par­ti­ci­pa­tion (OP) as well. The main benefit of FP ab­sent from OP, I sup­pose, is prepar­ing you for effi­cient writ­ten com­mu­ni­ca­tion, but the rest seem fea­ture in both. The fact that their benefits over­lap ex­plains why re­mote re­searchers benefit so much more than oth­ers from FP.

• What about “FP is a good way to stay up to date on ev­ery­one else’s lat­est re­search” and “FP gen­er­ates new ideas via cross-fer­til­iza­tion”? It seems like FP al­lows some­one to fol­low, par­ti­ci­pate in, and cross-fer­til­ize among many more lines of re­search than OP. (I should clar­ify that I don’t think ev­ery­one should do FP all the time. There are pros and cons to writ­ten vs ver­bal dis­cus­sions so some­one already in an office en­vi­ron­ment might be best served to do some of each, or some peo­ple within a re­search in­sti­tute can do some of each in or­der to bet­ter spread and cross-fer­til­ize ideas.)

• I agree that some peo­ple can benefit from do­ing both, al­though get­ting ev­ery­one on­line is a hard col­lec­tive ac­tion prob­lem. I just claim that many re­searchers will satisfy with OP. At MIRI/​FHI/​OpenAI there are ~30-150 re­searchers, who think about a wide range of ar­eas, which seems broadly com­pa­rable to the re­searchers among LessWrong/​AF’s ac­tive users (de­pend­ing on your defi­ni­tion of ‘re­searcher’, or ‘ac­tive’). Idea-ex­change is ex­tended by work­shops and peo­ple mov­ing jobs. Many in such a work en­vi­ron­ment will fund that FP has un­ac­cept­ably low sig­nal-noise ra­tio and will in­evitably avoid FP...

• Many in such a work en­vi­ron­ment will fund that FP has un­ac­cept­ably low sig­nal-noise ra­tio and will in­evitably avoid FP...

I think FP has a bet­ter sig­nal-cost ra­tio than work­shops I’ve been to, in part be­cause peo­ple tend to be more will­ing to talk about half-baked ideas in pri­vate, and in part be­cause if I see some con­tent on­line that I’m not in­ter­ested in, I can quickly skip over it, while di­rectly sig­nal­ing dis­in­ter­est to some­one IRL is li­able to hurt their feel­ings and ac­crue so­cial cost to my­self.

(I still try to at­tend work­shops once a while, in part to phys­i­cally meet peo­ple, in part to talk to peo­ple who rarely par­ti­ci­pate on­line, in part to get peo­ple’s pri­vate opinions that they don’t share on­line.)

I do think there are other pow­er­ful dis­in­cen­tives for FP though, and agree that it’s kind of an up­hill bat­tle to get more peo­ple on­line.

• I agree with this. Re­cently I started blog­ging about ML and (in fu­ture posts) AI safety. I in­tended this to pri­mar­ily be a learn­ing ex­pe­rience. I found self-teach­ing my­self ma­te­rial with­out the aid of a pub­lic fo­rum to be pretty bor­ing. This way I feel much more en­gaged. It also adds an ad­ver­sar­ial as­pect since I am forced to perform a men­tal check of “does what I wrote ac­tu­ally make sense” lest some­one cor­rect me. I hy­poth­e­size that this helps de­stroy a lot of be­gin­ner er­rors, and strength­ens my abil­ity to com­mu­ni­cate in the pro­cess. Also writ­ing is just a lot of fun too.

I ac­tu­ally found it pretty funny that you posted this the day I started blog­ging.

• I have the same feel­ing that com­ment­ing on posts takes very lit­tle will power but on the other hand writ­ing ideas I care about into their own posts takes a lot of will power. I don’t think that you are a unique case, and would ex­pect that most peo­ple feel that com­ment­ing is much eas­ier.

• Com­pletely agree.

This is an ex­cel­lent post, I won­der if I should com­ment more on posts in­stead of be­ing just a lurker. Since that is still some­what un­de­cided, at the very least, this post does helps me on want­ing to close the gap or to “to learn miss­ing knowl­edge”.

• This has similarly been my ap­proach. As best I can tell writ­ing pa­pers for aca­demic pub­li­ca­tion is nice but, es­pe­cially in the AI safety space, is not re­ally the best way to con­vey and dis­cuss ideas. Much more im­por­tant seems to be be­ing part of the con­ver­sa­tion about tech­ni­cal ideas, learn­ing from it, and adding to it so oth­ers can do the same. I put some small amount of effort into things out­side FP mostly be­cause I be­lieve it’s a good idea for rep­u­ta­tion effects and spread­ing ideas out­side the fo­rum bub­ble, but not be­cause I think it’s the best way to make in­tel­lec­tual progress.

It’s also nice be­cause the feed­back loops are shorter. I can com­ment on a post or write my own, have a dis­cus­sion, and then within weeks see the rip­ples of that dis­cus­sion in­fluenc­ing other dis­cus­sions. It helps me feel the im­pact I’m hav­ing, and mo­ti­vates me to keep go­ing.

Prob­a­bly the only thing su­pe­rior in my mind is do­ing prac­ti­cal work, e.g. build­ing sys­tems that test out ideas. Un­for­tu­nately many of the ideas we talk about in safety are cur­rently ahead of the tech so we don’t know how to build things yet (and for safety sake I think it’s fine to not push on that too hard since I ex­pect it will come on its own any­way), so un­til we are closer to AGI fo­rum par­ti­ci­pa­tion is likely one of the high im­pact ac­tivi­ties one can en­gage in (I’m similarly pos­i­tive about do­ing the face-to-face equiv­a­lent of talk­ing at con­fer­ences and hav­ing con­ver­sa­tions with in­ter­ested folks).

• I don’t have that much ex­pe­rience with fo­rums—when I was in re­search I learned mostly from read­ing sci­en­tific pa­pers + googling stuff to un­der­stand them. But I definitely agree that be­ing more ac­tive and en­gaged in the dis­cus­sion is helpful.

Aside from the topic of re­search, I used to be very pas­sive on my so­cial nets and ba­si­cally just con­sumed con­tent cre­ated by oth­ers. But af­ter I be­came more ac­tive I feel like I am get­ting more value out of it and at the same time spend less time there, as for­mu­lat­ing ques­tions or ideas takes effort. So it’s a nat­u­ral con­straint.