“Other people are wrong” vs “I am right”

I’ve re­cently been spend­ing some time think­ing about the ra­tio­nal­ity mis­takes I’ve made in the past. Here’s an in­ter­est­ing one: I think I have his­tor­i­cally been too hasty to go from “other peo­ple seem very wrong on this topic” to “I am right on this topic”.

Through­out my life, I’ve of­ten thought that other peo­ple had be­liefs that were re­ally re­pug­nant and stupid. Now that I am older and wiser, I still think I was cor­rect to think that these ideas were re­pug­nant and stupid. Over­all I was prob­a­bly slightly in­suffi­ciently dis­mis­sive of things like the opinions of ap­par­ent do­main ex­perts and the opinions of peo­ple who seemed smart whose ar­gu­ments I couldn’t re­ally fol­low. I also over­rated con­ven­tional wis­dom about fac­tual claims about how the world worked, though I un­der­rated con­ven­tional wis­dom about how to be­have.

Ex­am­ples of ideas where I thought the con­ven­tional wis­dom was re­ally dumb:

  • I thought that an­i­mal farm­ing was a mas­sive moral catas­tro­phe, and I thought it was a sign of ter­rible moral failure that al­most ev­ery­one around me didn’t care about this and wasn’t in­ter­ested when I brought it up.

  • I thought that AI safety was a big deal, and I thought the ar­gu­ments against it were all pretty stupid. (Nowa­days the con­ven­tional wis­dom has a much higher opinion of AI safety; I’m talk­ing about 2010-2014.)

  • I thought that peo­ple have ter­rible taste in eco­nomic policy, and that they mostly vote for good-sound­ing stuff that stops sound­ing good if you think about it prop­erly for even a minute

  • I was hor­rified by peo­ple proudly buy­ing prod­ucts that said “Made in Aus­tralia” on them; I didn’t un­der­stand how that wasn’t ob­vi­ously racist, and I thought that we should make it much eas­ier to al­low any­one who wants to to come live in Aus­tralia. (This one has be­come much less con­tro­ver­sial since Trump in­ad­ver­tently con­vinced liber­als that they should be in fa­vor of im­mi­gra­tion liber­al­iza­tion.)

  • In the spirit of loudly ac­knowl­edg­ing past mis­takes: I thought and still think that a lot of peo­ple’s ar­gu­ments about why it’s good to call the po­lice on bike thieves were dumb. See eg many of the ar­gu­ments peo­ple made in re­sponse to a post of mine about this (that in fair­ness was a re­ally dumb post, IMO)

I think I was right about other peo­ple be­ing wrong. How­ever, I think that my ac­tual opinions on these top­ics were pretty con­fused and wrong, much more than I thought at the time. Here’s how I up­dated my opinion for all the things above:

  • I have up­dated against the sim­ple view of he­do­nic util­i­tar­i­anism un­der which it’s plau­si­ble that sim­ple con­trol sys­tems can suffer. A few years ago, I was se­ri­ously wor­ried that the fu­ture would con­tain much more fac­tory farm­ing and there­fore end up net nega­tive; I now think that I over­rated this fear, be­cause (among other ar­gu­ments) al­most no-one ac­tu­ally en­dorses tor­tur­ing an­i­mals, we just do it out of ex­pe­di­ency, and in the limit of bet­ter tech­nol­ogy our weak prefer­ences will over­ride our ex­pe­di­ency.

  • My un­der­stand­ing of AI safety was “even­tu­ally some­one will build a re­cur­sively self im­prov­ing sin­gle­ton sovereign AGI, and we need to figure out how to build it such that it can have an off switch and it im­ple­ments some good value func­tion in­stead of some­thing bad.” I think this pic­ture was mas­sively over­sim­plified. On the strate­gic side, I didn’t think about the pos­si­bil­ities of slower take­offs or pow­er­ful tech­nolo­gies with­out re­cur­sive self im­prove­ment; on the tech­ni­cal safety side, I didn’t un­der­stand that it’s hard to even build a pa­per­clip max­i­mizer, and a lot of our effort might go into figur­ing out how to do that.

  • Other peo­ple have ter­rible taste in eco­nomic policy, but I think that I was at the time over­con­fi­dent in var­i­ous liber­tar­i­an­ish ideas that I’m now less en­thu­si­as­tic about. Also, I no longer think it’s a slam dunk that so­ciety is bet­ter off from be­com­ing wealthier, be­cause of con­sid­er­a­tions re­lated to the far fu­ture, an­i­mals, and whether more money makes us hap­pier.

  • I think that im­mi­gra­tion liber­al­iza­tion is more dan­ger­ous than I used to think, be­cause rich so­cieties seem to gen­er­ate mas­sive pos­i­tive ex­ter­nal­ities for the rest of the world and it seems pos­si­ble that a sud­den in­flux of less ed­u­cated peo­ple with (in my opinion) worse poli­ti­cal opinions might be kil­ling the goose that lays the golden eggs.

  • Re bike thieves: I think that even though util­i­tar­i­anism is good and stuff, it’s ex­tremely costly to have thiev­ery be tol­er­ated, be­cause then you have to do all these nega­tive-sum things like buy­ing bike locks. Also it seems like we’re gen­er­ally bet­ter off if peo­ple help with en­force­ment of laws.


In all of these cases, my ar­gu­ments against oth­ers were much higher qual­ity than my ac­tual be­liefs. Much more con­cern­ingly, I think I was much bet­ter at spot­ting the holes in other peo­ple’s ar­gu­ments than spot­ting holes in my own.

There’s also a gen­eral fac­tor here of me be­ing over­con­fi­dent in the de­tails of ideas that had some ring of truth to them. Like, the im­por­tance of AGI safety seemed re­ally ob­vi­ous to me, and I think that my sense of ob­vi­ous­ness has his­tor­i­cally been pretty good at spot­ting ar­gu­ments that later stand up to in­tense scrutiny. But I was mas­sively over­con­fi­dent in my par­tic­u­lar story for how AGI would go down. I should have been more dis­junc­tive: I should have said “It sure seems like some­thing like this ought to hap­pen, and it seems like step three could hap­pen in any of these four pos­si­ble ways, and I don’t know which of them will be true, and maybe it will ac­tu­ally be an­other one, but I feel pretty con­vinced that there’s some way it will hap­pen”.

Here are some other ideas which I con­tinue to en­dorse which had that ring of truth to them, but whose de­tails I’ve been similarly over­con­fi­dent about. (Some of these are pretty ob­scure.)

  • The simu­la­tion hypothesis

  • UDASSA

  • The ma­lig­nancy of the uni­ver­sal prior

  • The math­e­mat­i­cal uni­verse hypothesis

  • Hu­mans have weird com­plex bi­ases re­lated to cat­e­gories like race and gen­der, and we should be care­ful about this in our think­ing. (Nowa­days this idea is su­per wide­spread and so it feels weird to put it in the same list as all these crazy other ideas. But when I first en­coun­tered it se­ri­ously in my first year of col­lege, it felt like an in­ter­est­ing and new idea, in the same cat­e­gory as many of the cog­ni­tive bi­ases I heard about on LessWrong.)

And here are ideas which had this ring of truth to them that I no longer en­dorse:

  • We should fill the uni­verse with he­do­nium.

  • The fu­ture might be net nega­tive, be­cause hu­mans so far have caused great suffer­ing with their tech­nolog­i­cal progress and there’s no rea­son to imag­ine that this will change. Fu­tur­ists are bi­ased against this ar­gu­ment be­cause they per­son­ally don’t want to die and have a strong self­ish de­sire for hu­man civ­i­liza­tion to per­sist.

  • Be­cause of Lan­dauer’s limit, civ­i­liza­tions have an in­cen­tive to aes­ti­vate. (This one is wrong be­cause it in­volves a mi­s­un­der­stand­ing of ther­mo­dy­nam­ics.)


My bias to­wards think­ing my own be­liefs are more rea­son­able than they are would be dis­as­trous if it pre­vented me from chang­ing my mind in re­sponse to good new ar­gu­ments. Luck­ily, I don’t think that I am par­tic­u­larly bi­ased in that di­rec­tion, for two rea­sons. Firstly, when I’m talk­ing to some­one who thinks I’m wrong, for what­ever rea­son I usu­ally take them pretty se­ri­ously and I have a small crisis of faith that prompts me to go off and re­ex­am­ine my be­liefs a bunch. Se­condly, I think that most of the time that peo­ple pre­sent an ar­gu­ment which later changes my mind, my ini­tial re­ac­tion is con­fu­sion rather than dis­mis­sive­ness.

As an ex­am­ple of the first: Once upon a time I told some­one I re­spected that they shouldn’t eat an­i­mal prod­ucts, be­cause of the vast suffer­ing caused by an­i­mal farm­ing. He looked over scorn­fully and told me that it was pretty rich for me to say that, given that I use Ap­ple prod­ucts—hadn’t I heard about the abu­sive Ap­ple fac­tory con­di­tions and how they have nets to pre­vent peo­ple kil­ling them­selves by jump­ing off the tops of the fac­to­ries? I felt ter­rified that I’d been com­mit­ting some grave moral sin, and then went off to my room to re­search the topic for an hour or two. I even­tu­ally be­came con­vinced that the net effect of buy­ing Ap­ple prod­ucts on hu­man welfare is prob­a­bly very slightly pos­i­tive but small enough to not worry about, and also it didn’t seem to me that there’s a strong de­on­tolog­i­cal ar­gu­ment against do­ing it.

(I went back and told the guy about the re­sult of me look­ing into it. He said he didn’t feel in­ter­ested in the topic any­more and didn’t want to talk about it. I said “wow, man, I feel pretty an­noyed by that; you gave me a moral crit­i­cism and I took it real se­ri­ously; I think it’s bad form to not spend at least a cou­ple min­utes hear­ing about what I found.” Some­one else who was in the room, who was very en­thu­si­as­tic about so­cial jus­tice, came over and be­rated me for try­ing to vi­o­late some­one else’s prefer­ences about not talk­ing about some­thing. I learned some­thing that day about how use­ful it is to take moral crit­i­cism se­ri­ously when it’s from peo­ple who don’t seem to be very di­rected by their morals.)

Other ex­am­ples: When I first ran across charis­matic peo­ple who were in fa­vor of de­on­tolog­i­cal val­ues and so­cial jus­ticey be­liefs, I took those ideas re­ally se­ri­ously and mul­led them over a lot. A few weeks ago, some­one gave me some un­ex­pect­edly harsh crit­i­cism about my per­sonal man­ner and sev­eral as­pects of how I ap­proach my work; I up­dated ini­tially quite far in the di­rec­tion of their crit­i­cism, only to up­date 70% of the way back to­wards my ini­tial views af­ter I spent ten more hours think­ing and talk­ing to peo­ple about it.

Ex­am­ples of the sec­ond: When I met peo­ple whose view of AI safety didn’t match my own naive view, I felt con­fused and took them se­ri­ously (in­clud­ing when they were ex­press­ing a bunch of skep­ti­cism of MIRI). When my friend Howie told me he thought the crim­i­nal jus­tice sys­tem was re­ally racist, I was sur­prised and quickly up­dated my opinion to “I am con­fused about this”, rather than dis­miss­ing him.

I can’t think of cases where I ini­tially thought an ar­gu­ment was re­ally stupid but then it ended up con­vinc­ing ei­ther me or a ma­jor­ity of peo­ple who I think of as my epistemic peers and su­pe­ri­ors (eg peo­ple who I think have gen­er­ally good judge­ment at EA orgs).

How­ever, I can think of cases where I felt ini­tially that an ar­gu­ment is dumb, but lots of my epistemic peers think that the ar­gu­ment is at least sort of rea­son­able. I am con­cerned by this and I’m try­ing to com­bat it. For ex­am­ple, the fol­low­ing ar­gu­ments are in my cur­rent list of things that I am wor­ried I’m un­der­valu­ing be­cause they ini­tially seem im­plau­si­ble to me, and are on my to-do list to even­tu­ally look into more care­fully: Drexler’s Com­pre­hen­sive AI Sys­tems. AI safety via am­bi­tious value learn­ing. Ar­gu­ments that pow­er­ful AI won’t lead to a sin­gle­ton.

Please let me know if you have ex­am­ples along these lines where I seemed dumber than I’m pre­sent­ing here.


Here’s an­other per­spec­tive on why my ap­proach might be a prob­lem. I think that peo­ple are of­ten pretty bad at ex­press­ing why they be­lieve things, and in par­tic­u­lar they don’t usu­ally say “I don’t know why I be­lieve this, but I be­lieve it any­way.” So if I dis­miss ar­gu­ments that suck, I might be dis­miss­ing use­ful knowl­edge that other peo­ple have gained through ex­pe­rience.

I think I’ve made mis­takes along these lines in the past. For ex­am­ple, I used to have a much lower opinion of pro­fes­sion­al­ism than I now do. And there are a cou­ple of se­ri­ous per­sonal mis­takes I’ve made where I looked around for the best ar­gu­ments against do­ing some­thing weird I wanted to do, and all of those ar­gu­ments sucked, and then I de­cided to do the weird thing, and then it was a bad idea.

Katja Grace calls this mis­take “break­ing Ch­ester­ton’s fence in the pres­ence of bull”.

This would sug­gest the heuris­tic “Take re­ceived wis­dom on top­ics into ac­count, even if you ask peo­ple where the re­ceived wis­dom comes from and they tell you a source that seems ex­tremely un­re­li­able”.

I think this heuris­tic is alright but shouldn’t be an over­rid­ing con­sid­er­a­tion. The ideas that evolve through the ex­pe­rience of so­cial groups are valuable be­cause they’re some­what se­lected for truth and im­por­tance. But the se­lec­tion pro­cess for these ideas is ex­tremely sim­ple and dumb.

I’d ex­pect that in most cases where some­thing is bad, there is a leg­ible ar­gu­ment for why we shouldn’t do it (where I’m in­clud­ing ar­gu­ments from em­piri­cal ev­i­dence as leg­ible ar­gu­ments). I’d pre­fer to just learn all of the few things that so­ciety im­plic­itly knows, rather than giv­ing up ev­ery time it dis­agrees with me.

Maybe this is me be­ing ar­ro­gant again, but I feel like the mis­take I made with the bike-steal­ing thing wasn’t me re­fus­ing to bow to so­cial au­thor­ity, it was me not try­ing hard enough to think care­fully about the eco­nomics of the situ­a­tion. My in­side view is that if I now try to think about eco­nomics, I don’t need to in­cor­po­rate that much out­side-view-style dis­count­ing of my own ar­gu­ments.

I have the big ad­van­tage of be­ing around peo­ple who are re­ally good at ar­tic­u­lat­ing the ac­tual rea­sons why things are bad. Pos­si­bly the num­ber one strength of the ra­tio­nal­ist com­mu­nity is cre­at­ing and dis­sem­i­nat­ing good ex­plicit mod­els of things that are widely im­plic­itly un­der­stood (eg var­i­ants of Good­hart’s law, Moloch, Ch­ester­ton’s fence, the unilat­er­al­ist’s curse, “tox­o­plasma of rage”). If I was in any other com­mu­nity, I’m wor­ried that I’d make posts like the one about the bike, and no-one would be able to ar­tic­u­late why I was wrong in a way that was con­vinc­ing. So I don’t nec­es­sar­ily en­dorse other peo­ple tak­ing the strat­egy I take.

I am not aware of that many cases where I be­lieved some­thing re­ally stupid be­cause all the com­mon ar­gu­ments against it seemed re­ally dumb to me. If I knew of more cases like this, I’d be more wor­ried about this.


Claire Za­bel says, in re­sponse to all this:

I’d say you’re too quick to buy a whole new story if it has the ring of truth, and too quick to ask oth­ers (and prob­a­bly your­self) to ei­ther re­fute on the spot, or ac­cept, a com­plex and im­por­tant new story about some­thing about the world, and leave too lit­tle room to say “this seems sketchy but I can’t ar­tic­u­late how” or “I want to think about it for a while” or “I’d like to hear the crit­ics’ coun­ter­ar­gu­ments” or “even though none of the above has yielded fruit, I’m still not con­fi­dent about this thing”

This seems plau­si­ble. I spend a bunch of time try­ing to ex­plain why I’m wor­ried about AI risk to peo­ple who don’t know much about the topic. This re­quires cov­er­ing quite a lot of ground; per­haps I should try harder to ex­plic­itly say “by the way, I know I’m tel­ling you a lot of crazy stuff; you should take as long as it takes to eval­u­ate all of this on your own; my goal here is just to ex­plain what I be­lieve; you should use me as a dat­a­point about one place that hu­man be­liefs some­times go af­ter think­ing about the sub­ject.”


I feel like my in­tu­itive sense of whether some­one else’s ar­gu­ment is roughly le­git is pretty good, and I plan to con­tinue feel­ing pretty con­fi­dent when I in­tu­itively feel like some­one else is be­ing dumb. But I am try­ing to not make the jump from “I think that this ar­gu­ment is roughly right” to “I think that all of the steps in this fleshed out ver­sion of that ar­gu­ment are roughly right”. Please let me know if you think I’m mak­ing that par­tic­u­lar mis­take.