Trust-Building: The New Rationality Project

“I did not write half of what I saw, for I knew I would not be be­lieved”
-Marco Polo
It’s en­light­en­ingly dis­turb­ing to see speci­fi­cally how the “dis­trust those who dis­agree” heuris­tic de­scends into the mad­ness of fac­tions.
-Zack M Davis

Old LessWrong: we fail to reach the truth be­cause of our cog­ni­tive bi­ases.

New LessWrong: we fail to reach the truth be­cause of rea­son­able mis­trust.

What causes mis­trust? It’s a com­bi­na­tion of things: mis­com­mu­ni­ca­tion and mis­takes; the pres­ence of ex­ploita­tive ac­tors; lack of re­peated trust-build­ing in­ter­ac­tions with the same peo­ple; and high pay­offs for epistem­i­cally un­vir­tu­ous be­hav­ior. Enough mis­trust can de­stroy our abil­ity to con­verge on the truth.

Once mis­trust has led us into fac­tion­al­ism, can we es­cape it?

Fac­tion­al­ism is not a bad thing. It means that we are will­ing to ac­cept some ev­i­dence from other peo­ple, as long as it’s not too di­ver­gent from our pri­ors. Fac­tions are worse than unity, but bet­ter than iso­la­tion.

What we want isn’t a lack of fac­tion­al­ism, it’s unity.

This sug­gests an ac­tivism strat­egy. Let’s form three cat­e­gories of fac­tions:

  • Your Com­mu­nity: You have high trust in this net­work, and be­lieve the ev­i­dence you re­ceive from it by de­fault. Although you don’t trust ev­ery­one who calls them­selves a part of this net­work or com­mu­nity, you know who be­longs and who doesn’t, who’s re­li­able and who’s not, and how to tell the differ­ence when you meet some­one new in the net­work.

  • Strangers: This net­work is untested, and your com­mu­nity doesn’t have a strong opinion on it. It would take sub­stan­tial work to learn how to nav­i­gate this for­eign net­work. But be­cause they are rel­a­tively iso­lated from your own com­mu­nity, they have evolved a differ­ent con­stel­la­tion of ev­i­dence. The ex­is­tence of fac­tion­al­ism is it­self ev­i­dence that you’d have some­thing new to gain by trad­ing in­for­ma­tion with them.

  • Ene­mies: This net­work has been ex­am­ined, ei­ther by you or by your com­mu­nity, and la­beled a toxic breed­ing ground of mis­in­for­ma­tion. Treat­ing your en­e­mies as though they were merely strangers would only alienate you from your own com­mu­nity. They might be ex­ploita­tive or stupid, but ei­ther way, en­gag­ing with them can only make things worse. All there is to do is fight, de­pro­gram, or ig­nore this lot.

To in­crease unity and pur­sue the truth, your goal is to find for­eign com­mu­ni­ties, and de­ter­mine whether they are friendly or dan­ger­ous.

Note that the goal is not to make more friends and get ex­posed to new ideas. That’s a recipe for naivete. The real goal is to ac­cu­rately dis­t­in­guish strangers from en­e­mies, and make in­tro­duc­tions and fa­cil­i­tate shar­ing with only the stranger, but not the en­emy. We might re­spect, dis­par­age, or ig­nore our en­e­mies, but we know how to tell them apart from our al­lies and our own:

“Here peo­ple was once used to be hon­ourable: now they are all bad; they have kept one good­ness: that they are great­est booz­ers.”

One of the many difficul­ties is the work it takes to dis­cover, eval­u­ate, and bring back in­for­ma­tion about new strangers, the truly for­eign. Their names and ideas are most likely un­known to any­body in your com­mu­nity, and they speak a differ­ent lan­guage or use differ­ent con­cep­tual frame­works.

Worse, your com­mu­nity is do­ing a steady busi­ness in its own con­cep­tual frame­work. You don’t need to just ex­plain why the new peo­ple you’ve dis­cov­ered are trust­wor­thy; you need to ex­plain why their way of think­ing is valuable enough to jus­tify the work of trans­lat­ing it into your own lan­guage and con­cep­tual frame­work, or learn­ing their lan­guage.

Luck­ily, you do have one thing on your side. For­eign com­mu­ni­ties usu­ally love it when strangers ex­press a gen­uine in­ter­est in ab­sorb­ing their ideas and spread­ing them far and wide.

You might think that there is a time to ex­plore, and a time to move to­ward a definite end. But this isn’t so.

When there’s a definite end in mind, mov­ing to­ward it is the easy part.

But mean­ing and value mostly come from nov­elty.

When it feels like there’s no need to ex­plore, and all you need to do is prac­tice your rou­tine and en­joy what you have, the right as­sump­tion is that you are miss­ing an op­por­tu­nity. This is when ex­plo­ra­tion is most ur­gent. “What am I miss­ing?” is a good ques­tion.

“You will hear it for your­selves, and it will surely fill you with won­der.”

What is our com­mu­nity re­li­ably miss­ing?