Unrolling social metacognition: Three levels of meta are not enough.

Dis­claimer: This post was writ­ten time-boxed to 2 hours be­cause I think LessWrong can still un­der­stand and im­prove upon it; please don’t judge me harshly for it.

Sum­mary: I am gen­er­ally dis­mayed that many peo­ple seem to think or as­sume that only three lev­els of so­cial metacog­ni­tion mat­ter (“Alex knows that Bailey knows that Char­lie knows X”), or oth­er­wise seem gen­er­ally averse to un­rol­ling those lev­els. This post is in­tended to point out (1) how the higher lev­els sys­tem­at­i­cally get dis­til­led and chun­ked into smaller work­ing mem­ory el­e­ments through so­cial learn­ing, which leads to emo­tional track­ing of phe­nom­ena at 6 lev­els of meta and higher, and (2) what I think this means about how to ap­proach con­flict re­s­olu­tion.

Epistemic sta­tus: don’t take my word for it; con­cep­tual points in­tended to be fairly self ev­i­dent upon re­flec­tion; ac­tual tech­niques not backed up by sys­tem­atic em­piri­cal re­search and might not gen­er­al­ize to other hu­mans; all con­tent very much val­i­dated by my per­sonal ex­pe­riences with talk­ing to peo­ple about feel­ings in real life.

Re­lated Read­ing: Dun­can Sa­bien on Com­mon knowl­edge & Mi­asma; Ben Pace on The Costly Co­or­di­na­tion Mechanism of Com­mon Knowledge


I. Con­cep­tual in­tro­duc­tion, by ex­am­ple

Here’s how higher lev­els of so­cial metacog­ni­tion get dis­til­led down and rep­re­sented in emo­tions that end up track­ing them (if poorly). Each feel­ing in the ex­am­ple be­low will be fol­lowed by an un­rol­ling of the ac­tual event or events it is im­plic­itly track­ing or refer­ring to.

Warn­ing: read­ing this first sec­tion (I) will re­quire a fair bit of sym­bolic rea­son­ing/​think­ing, so you might find it tiring and pre­fer to skip to later sec­tions. A bet­ter writ­ing of this sec­tion would do more work in be­tween these sym­bolic rea­son­ing bits to dis­till things out and make them eas­ier to di­gest.

Scale 1: One event, four lev­els of meta (yes, we’re start­ing with four)

1.1) Alex leaves out the milk for 5 minutes

1.2) Bailey ob­serves (1.1), and feels it was bad.

Un­rol­ling of refer­ents: Bailey felt that Alex leav­ing out the milk was bad.

1.3) Alex ob­serves (1.2), and feels judged.

Un­rol­ling of refer­ents: Alex felt that Bailey felt that Alex leav­ing out the milk was bad.

1.4) Alex re­flects on feel­ing judged, doesn’t like it, and con­cludes that Bailey is “a downer”.

Un­rol­ling of refer­ents: Alex felt it was bad that Alex felt that Bailey felt that Alex leav­ing out the milk was bad.

No­tice that the un­rol­lings look and sound very differ­ent from the dis­til­la­tions. That’s in large part be­cause the un­rol­ling is not our na­tive for­mat for stor­ing so­cial metacog­ni­tion; it’s stored via con­cepts like “feel­ing judged” or “be­ing a downer”. How­ever, to the ex­tent that the feel­ing “Bailey is a downer” is track­ing some­thing in re­al­ity, it’s track­ing things that track things that track things that track re­al­ity: in this case, milk spoilage.

(An aside: no­tice also that 1.4 in­volves Alex’s feel­ings about Alex’s feel­ings. Some peo­ple wouldn’t call that an ex­tra level of so­cial metacog­ni­tion, and would just com­bine it all to­gether into “Alex’s feel­ings”. How­ever, I’m sep­a­rat­ing those lay­ers for two rea­sons: (1) the sep­a­ra­tion in count­ing won’t af­fect my con­clu­sion that the to­tal num­ber of lev­els be­ing im­plic­itly tracked greatly ex­ceeds three, and (2) I think it’s es­pe­cially im­por­tant to note when peo­ple have feel­ings about their own feel­ings, as that can lead to cir­cu­lar defi­ni­tions in what their feel­ings are track­ing; but that’s a topic for an­other day.)

Scale 2: mul­ti­ple events, six lev­els of meta

I’ll start the num­ber­ing at 4 here:

2.4) Mul­ti­ple similar Scale 1 events hap­pen Alex does some­thing X, and ends up feel­ing that Bailey was “a downer” about it.

Par­tial un­rol­ling of refer­ents: Alex feels that Bailey is of­ten a downer

Com­plete un­rol­ling of refer­ents: Alex felt it was bad that Alex felt that Bailey felt that Alex do­ing X was bad, for mul­ti­ple val­ues of X.

2.5) Char­lie ob­serves Alex treat­ing Bailey like “a downer”, thinks this is base­less, and feels Alex is “a snob”.

Par­tial un­rol­ling of refer­ents: Char­lie felt it was bad that Alex felt that Bailey is a downer.

2.6) Bailey ob­serves Char­lie’s op­po­si­tion to Alex’s snob­bi­ness, feels so­cially in­cluded by Char­lie, and con­cludes that Char­lie is “pro­tec­tive & wel­com­ing.”

Par­tial un­rol­ling of refer­ents: Bailey felt it was good that Char­lie felt it was bad that Alex was feel­ing that Bailey was of­ten a downer.

Com­plete un­rol­ling of refer­ents: Bailey felt it was good that Char­lie felt it was bad that Alex felt it was bad that Alex felt that Bailey felt it was bad that Alex did X, for mul­ti­ple val­ues of X.

If you count the (+/​-) signs im­plicit in the “good” and “bad” judge­ments here, they sug­gest that Char­lie is im­plic­itly con­demn­ing Alex’s mul­ti­ple “X” be­hav­iors (per­haps among other things, such as Alex’s style so­cial of de­liv­ery). Char­lie might not in­tend or even be aware of this effect, and it might take ex­plicit work and dis­course for the group to un­tan­gle and no­tice. What could re­sult from the group re­flect­ing upon this to­gether? Well, it’s not hard at all to imag­ine that Char­lie, upon har­ing about the milk and other situ­a­tions (val­ues of “X”), might con­clude that Alex’s be­hav­ior with the milk was rea­son­able, and that “Alex was right to think that Bailey was be­ing a downer”. In turn, this could burst Bailey’s bub­ble of so­cial sup­port, and re­sult in Bailey hav­ing a change of heart about how crit­i­cal to be of oth­ers.

The par­tic­u­lar con­se­quences here of course a hy­po­thet­i­cal, and things could go very differ­ently de­pend­ing on the de­tails; it’s just meant to illus­trate how “changes of heart” can prop­a­gate through un­rol­lings of so­cial metacog­ni­tion.

ETA (8/​27): Note there is an im­por­tant dis­tinc­tion here be­tween im­plicit and ex­plicit metacog­ni­tion. Bailey alone is not (nec­es­sar­ily) load­ing up a 5-layer-deep cog­ni­tive model of what’s go­ing on, all at once. Rather, the lay­ers are dis­tributed across peo­ple, whence the term “so­cial metacog­ni­tion”. How­ever, six lev­els of metacog­ni­tion re­ally are needed for some­one to aiming to ground out out all these feel­ings in “ob­ject-level” re­al­ity (i.e., non-men­tal phe­nom­ena like milk spoilage).

(This is the end of the tiring sym­bolic rea­son­ing sec­tion.)


II. Higher lev­els of meta

Without go­ing into fur­ther ex­plicit de­tail, I hope you an see the pat­tern. Levels of so­cial metacog­ni­tion get dis­til­led into sim­ple, re­peated con­cepts like “feel­ing judged”, “be­ing a downer”, “be­ing a snob”, “be­ing wel­com­ing”, and so on. To the ex­tent that these dis­til­led con­cepts be­have in a some­what sys­tem­atic man­ner in re­la­tion to re­al­ity, they have some ten­dency to be ac­tu­ally track­ing things that are ac­tu­ally hap­pen­ing. It’s not un­com­mon for me to ob­serve six lev­els of so­cial metacog­ni­tion in a given dis­agree­ment or con­flict, which is why I chose six for this post.

III. Peo­ple don’t usu­ally un­roll things this way. Why?

Un­for­tu­nately, I think a lot of peo­ple aren’t aware that it it even makes sense to try to ground out these sorts of so­cial meta cog­ni­tion in more ex­plicit terms to be rea­soned and dis­agreed about. I think this is be­cause it takes a lot of work­ing mem­ory slots to do, such that you ba­si­cally need a shared piece of pa­per, white­board, or a shared Google doc to do it rea­son­ably and col­lab­o­ra­tively (rather than just sling­ing hard-to-un­pack nega­tive judge­ments at each other, ad­ver­sar­i­ally, ei­ther in per­son or on the in­ter­net). How­ever, I’ve re­solved con­flicts through co-writ­ing and co-di­a­gram­ming rele­vant lev­els of so­cial metacog­ni­tion many times now, and found it to be very en­light­en­ing al­most ev­ery time it a way that di­rectly benefit­ted the “so­cial situ­a­tion”. I’ve found it’s best if the shared writ­ing medium is used for dis­til­la­tion mechanism, and is aug­mented by ac­tual real-time con­ver­sa­tion over the cre­ation of the doc­u­ment.

IV. A fruit­ful application

For in­stance, in the past week, Alex (anonymized) felt judged by me for a thing I no­ticed Alex do­ing. I said, (a) “Don’t worry, I don’t think you did any­thing bad”, but Alex didn’t find this re­as­sur­ing. To check, I asked “Do you feel like I feel like you did some­thing bad?” and Alex said “No”. This ran up against my ex­plicit mod­els of peo­ple feel­ing judged that had fit well with past un­rol­ling of the con­cept. So, I broke out a Google doc (in per­son) and started un­rol­ling stuff. The situ­a­tion was more com­pli­cated than de­scribed above, so the doc gave us men­tal space to ex­plore other ideas for re­s­olu­tion. We even­tu­ally looped back to my ques­tion (a) above, and Alex said “Huh, yeah, I think I do feel that you feel that I did some­thing bad.” Once that aware­ness ex­isted, I re­sponded “Cool! Well guess what? I don’t think you did any­thing bad.”, and this time, it res­onated with Alex and Alex no longer felt judged. I then apol­o­gized with “Also, I’m sorry you felt judged. Given that I didn’t ac­tu­ally feel you were do­ing some­thing bad, this was a mis­take on my part, and I’m sorry.”
This fur­ther cleared things up.

This whole pro­cess took about 15 min­utes. In ret­ro­spect it might seem like we could have jumped straight to this solu­tion by me say­ing “I’m sorry I made you feel judged”, but that wasn’t an available strat­egy ex ante, for two rea­sons:

(1) Some­times I re­ally am judg­ing some­one, and I’m okay with them feel­ing judged, be­cause I do in fact think they did some­thing wrong. As a re­sult of this will­ing­ness in my­self and oth­ers, it’s not always be­liev­able to say “Sorry, I wish I hadn’t made you feel judged”. In­deed, to many this feels like a plat­i­tude. But, by ac­tu­ally go­ing through the work of ac­tu­ally un­rol­ling whether or not I thought Alex did a bad thing, and the other de­tails of what was go­ing on be­tween us, we es­tab­lished enough shared clar­ity about the situ­a­tion that we man­aged to “get on the same page” what whether a bad thing was done, who thought or didn’t think that, and who mis­com­mu­ni­cated or didn’t mis­com­mu­ni­cate about it.

(2) There were many other things go­ing on that the Google doc helped to or­ga­nize and sift through with­out get­ting us lost. Without that func­tion­al­ity, I don’t think we would have been able to hone in on the par­tic­u­lar nar­ra­tive re­s­olu­tion above.

V. How gen­er­al­iz­able is this ‘un­rol­ling’ tech­nique?

The ap­pli­ca­tion (IV) above not an iso­lated in­ci­dent. I’ve found­ing co-writ­ing and co-draw­ing to be ex­tremely valuable in set­tling so­cial dis­agree­ments and con­flicts on at least 30 oc­ca­sions now, with at least 7 differ­ent peo­ple, of vary­ing de­grees of in­cli­na­tion to­ward ex­plicit sym­bolic rea­son­ing. I imag­ine some in­cli­na­tion is nec­es­sary, but much less than I would I have ex­pected per­vi­ously. For in­stance, I’ve used this sort of un­rol­ling heuris­tic fruit­fully in nu­mer­ous con­ver­sa­tions with folks close to me who

(1) didn’t go to col­lege or oth­er­wise study a sym­bolic dis­ci­pline like math or lin­guis­tics, but who

(2) were gen­er­ally open minded enough to be will­ing to try out a “weird con­flict re­s­olu­tion tech­nique I’m ex­per­i­ment­ing with” where we sat down to­gether and tried un­packed our feel­ings in ex­plicit terms in a com­mon medium (usu­ally a Google doc).

I’ll defer to the find­ing of the broader com­mu­nity here to see if oth­ers can make this sort of thing work use­fully.

VI. Re­la­tion to “mi­asma” and “hype”

The con­cept of “mi­asma” that Dun­can is ges­tur­ing at in Com­mon Knowl­edge and Mi­asma feels like a real so­cial phe­nomenon to me, suc­cinctly defin­able as “nega­tive un­grounded so­cial metacog­ni­tion”. There is such a thing as pos­i­tive un­grounded so­cial metacog­ni­tion, as well, which I think is nor­mally called “hype”, at least in Sili­con Valley. I think both hype and mi­asma are failures of group co­or­di­na­tion, and both are costly to re­solve, along the lines pointed out by Ben in The Costly Co­or­di­na­tion Mechanism of Com­mon Knowl­edge. How­ever, the com­mu­nica­tive costs of re­solv­ing these prob­lems can be sig­nifi­cantly de­creased if peo­ple are aware of what they are. Both re­quire cre­at­ing and shar­ing of ideas in places of com­mon view, like writ­ing blog posts that a lot of peo­ple see each other com­ment­ing on, or hold­ing meet­ings that a lot of peo­ple can see each other at­tend­ing, or for com­plex top­ics, sit­ting to­gether and co-au­thor­ing a doc­u­ment.

VII. Apology

I’m sorry I put very lit­tle effort into the ped­a­gogy of this post, due to hav­ing too lit­tle time to write it. Hope­fully it will be of some value any­way, due to much bet­ter posts hav­ing been writ­ten and cir­cu­lated on com­mon knowl­edge re­cently, and due to the gen­eral in­tel­lec­tual health of LessWrong ap­pear­ing, to me, to be able to ab­sorb mediocrely-ex­plained ideas and flesh them out into bet­ter ones. My sense is that the cul­ture here has been try­ing to move to­wards peo­ple not wait­ing un­til an idea is perfectly elab­o­rated be­fore start­ing to talk about it, so to the ex­tent these ideas might be valuable, I’m punt­ing to the com­mu­nity to do more elab­o­ra­tion and/​or dis­til­la­tion of them. In­deed, wish­ing not to be a part of a “com­mon knowl­edge break­down” prob­lem is one rea­son I time-boxed two hours to write this post in­stead of wait­ing to im­prove it.