Un­rolling so­cial meta­cog­ni­tion: Three levels of meta are not enough.

Dis­claimer: This post was writ­ten time-boxed to 2 hours be­cause I think LessWrong can still un­der­stand and im­prove upon it; please don’t judge me harshly for it.

Sum­mary: I am gen­er­ally dis­mayed that many people seem to think or as­sume that only three levels of so­cial meta­cog­ni­tion mat­ter (“Alex knows that Bailey knows that Charlie knows X”), or oth­er­wise seem gen­er­ally averse to un­rolling those levels. This post is in­ten­ded to point out (1) how the higher levels sys­tem­at­ic­ally get dis­tilled and chunked into smal­ler work­ing memory ele­ments through so­cial learn­ing, which leads to emo­tional track­ing of phe­nom­ena at 6 levels of meta and higher, and (2) what I think this means about how to ap­proach con­flict res­ol­u­tion.

Epistemic status: don’t take my word for it; con­cep­tual points in­ten­ded to be fairly self evid­ent upon re­flec­tion; ac­tual tech­niques not backed up by sys­tem­atic em­pir­ical re­search and might not gen­er­al­ize to other hu­mans; all con­tent very much val­id­ated by my per­sonal ex­per­i­ences with talk­ing to people about feel­ings in real life.

Related Read­ing: Dun­can Sa­bien on Com­mon know­ledge & Mi­asma; Ben Pace on The Costly Coordin­a­tion Mech­an­ism of Com­mon Knowledge


I. Con­cep­tual in­tro­duc­tion, by ex­ample

Here’s how higher levels of so­cial meta­cog­ni­tion get dis­tilled down and rep­res­en­ted in emo­tions that end up track­ing them (if poorly). Each feel­ing in the ex­ample be­low will be fol­lowed by an un­rolling of the ac­tual event or events it is im­pli­citly track­ing or re­fer­ring to.

Warn­ing: read­ing this first sec­tion (I) will re­quire a fair bit of sym­bolic reas­on­ing/​think­ing, so you might find it tir­ing and prefer to skip to later sec­tions. A bet­ter writ­ing of this sec­tion would do more work in between these sym­bolic reas­on­ing bits to dis­till things out and make them easier to di­gest.

Scale 1: One event, four levels of meta (yes, we’re start­ing with four)

1.1) Alex leaves out the milk for 5 minutes

1.2) Bailey ob­serves (1.1), and feels it was bad.

Un­rolling of ref­er­ents: Bailey felt that Alex leav­ing out the milk was bad.

1.3) Alex ob­serves (1.2), and feels judged.

Un­rolling of ref­er­ents: Alex felt that Bailey felt that Alex leav­ing out the milk was bad.

1.4) Alex re­flects on feel­ing judged, doesn’t like it, and con­cludes that Bailey is “a downer”.

Un­rolling of ref­er­ents: Alex felt it was bad that Alex felt that Bailey felt that Alex leav­ing out the milk was bad.

Notice that the un­rollings look and sound very dif­fer­ent from the dis­til­la­tions. That’s in large part be­cause the un­rolling is not our nat­ive format for stor­ing so­cial meta­cog­ni­tion; it’s stored via con­cepts like “feel­ing judged” or “be­ing a downer”. However, to the ex­tent that the feel­ing “Bailey is a downer” is track­ing some­thing in real­ity, it’s track­ing things that track things that track things that track real­ity: in this case, milk spoil­age.

(An aside: no­tice also that 1.4 in­volves Alex’s feel­ings about Alex’s feel­ings. Some people wouldn’t call that an ex­tra level of so­cial meta­cog­ni­tion, and would just com­bine it all to­gether into “Alex’s feel­ings”. However, I’m sep­ar­at­ing those lay­ers for two reas­ons: (1) the sep­ar­a­tion in count­ing won’t af­fect my con­clu­sion that the total num­ber of levels be­ing im­pli­citly tracked greatly ex­ceeds three, and (2) I think it’s es­pe­cially im­port­ant to note when people have feel­ings about their own feel­ings, as that can lead to cir­cu­lar defin­i­tions in what their feel­ings are track­ing; but that’s a topic for an­other day.)

Scale 2: mul­tiple events, six levels of meta

I’ll start the num­ber­ing at 4 here:

2.4) Mul­tiple sim­ilar Scale 1 events hap­pen Alex does some­thing X, and ends up feel­ing that Bailey was “a downer” about it.

Par­tial un­rolling of ref­er­ents: Alex feels that Bailey is of­ten a downer

Com­plete un­rolling of ref­er­ents: Alex felt it was bad that Alex felt that Bailey felt that Alex do­ing X was bad, for mul­tiple val­ues of X.

2.5) Charlie ob­serves Alex treat­ing Bailey like “a downer”, thinks this is base­less, and feels Alex is “a snob”.

Par­tial un­rolling of ref­er­ents: Charlie felt it was bad that Alex felt that Bailey is a downer.

2.6) Bailey ob­serves Charlie’s op­pos­i­tion to Alex’s snob­bi­ness, feels so­cially in­cluded by Charlie, and con­cludes that Charlie is “pro­tect­ive & wel­com­ing.”

Par­tial un­rolling of ref­er­ents: Bailey felt it was good that Charlie felt it was bad that Alex was feel­ing that Bailey was of­ten a downer.

Com­plete un­rolling of ref­er­ents: Bailey felt it was good that Charlie felt it was bad that Alex felt it was bad that Alex felt that Bailey felt it was bad that Alex did X, for mul­tiple val­ues of X.

If you count the (+/​-) signs im­pli­cit in the “good” and “bad” judge­ments here, they sug­gest that Charlie is im­pli­citly con­demning Alex’s mul­tiple “X” be­ha­vi­ors (per­haps among other things, such as Alex’s style so­cial of de­liv­ery). Charlie might not in­tend or even be aware of this ef­fect, and it might take ex­pli­cit work and dis­course for the group to un­tangle and no­tice. What could res­ult from the group re­flect­ing upon this to­gether? Well, it’s not hard at all to ima­gine that Charlie, upon har­ing about the milk and other situ­ations (val­ues of “X”), might con­clude that Alex’s be­ha­vior with the milk was reas­on­able, and that “Alex was right to think that Bailey was be­ing a downer”. In turn, this could burst Bailey’s bubble of so­cial sup­port, and res­ult in Bailey hav­ing a change of heart about how crit­ical to be of oth­ers.

The par­tic­u­lar con­sequences here of course a hy­po­thet­ical, and things could go very dif­fer­ently de­pend­ing on the de­tails; it’s just meant to il­lus­trate how “changes of heart” can propag­ate through un­rollings of so­cial meta­cog­ni­tion.

ETA (8/​27): Note there is an im­port­ant dis­tinc­tion here between im­pli­cit and ex­pli­cit meta­cog­ni­tion. Bailey alone is not (ne­ces­sar­ily) load­ing up a 5-layer-deep cog­nit­ive model of what’s go­ing on, all at once. Rather, the lay­ers are dis­trib­uted across people, whence the term “so­cial meta­cog­ni­tion”. However, six levels of meta­cog­ni­tion really are needed for someone to aim­ing to ground out out all these feel­ings in “ob­ject-level” real­ity (i.e., non-men­tal phe­nom­ena like milk spoil­age).

(This is the end of the tir­ing sym­bolic reas­on­ing sec­tion.)


II. Higher levels of meta

Without go­ing into fur­ther ex­pli­cit de­tail, I hope you an see the pat­tern. Levels of so­cial meta­cog­ni­tion get dis­tilled into simple, re­peated con­cepts like “feel­ing judged”, “be­ing a downer”, “be­ing a snob”, “be­ing wel­com­ing”, and so on. To the ex­tent that these dis­tilled con­cepts be­have in a some­what sys­tem­atic man­ner in re­la­tion to real­ity, they have some tend­ency to be ac­tu­ally track­ing things that are ac­tu­ally hap­pen­ing. It’s not un­com­mon for me to ob­serve six levels of so­cial meta­cog­ni­tion in a given dis­agree­ment or con­flict, which is why I chose six for this post.

III. People don’t usu­ally un­roll things this way. Why?

Un­for­tu­nately, I think a lot of people aren’t aware that it it even makes sense to try to ground out these sorts of so­cial meta cog­ni­tion in more ex­pli­cit terms to be reasoned and dis­agreed about. I think this is be­cause it takes a lot of work­ing memory slots to do, such that you ba­sic­ally need a shared piece of pa­per, white­board, or a shared Google doc to do it reas­on­ably and col­lab­or­at­ively (rather than just sling­ing hard-to-un­pack neg­at­ive judge­ments at each other, ad­versari­ally, either in per­son or on the in­ter­net). However, I’ve re­solved con­flicts through co-writ­ing and co-dia­gram­ming rel­ev­ant levels of so­cial meta­cog­ni­tion many times now, and found it to be very en­light­en­ing al­most every time it a way that dir­ectly be­ne­fit­ted the “so­cial situ­ation”. I’ve found it’s best if the shared writ­ing me­dium is used for dis­til­la­tion mech­an­ism, and is aug­men­ted by ac­tual real-time con­ver­sa­tion over the cre­ation of the doc­u­ment.

IV. A fruit­ful application

For in­stance, in the past week, Alex (an­onym­ized) felt judged by me for a thing I no­ticed Alex do­ing. I said, (a) “Don’t worry, I don’t think you did any­thing bad”, but Alex didn’t find this re­as­sur­ing. To check, I asked “Do you feel like I feel like you did some­thing bad?” and Alex said “No”. This ran up against my ex­pli­cit mod­els of people feel­ing judged that had fit well with past un­rolling of the concept. So, I broke out a Google doc (in per­son) and star­ted un­rolling stuff. The situ­ation was more com­plic­ated than de­scribed above, so the doc gave us men­tal space to ex­plore other ideas for res­ol­u­tion. We even­tu­ally looped back to my ques­tion (a) above, and Alex said “Huh, yeah, I think I do feel that you feel that I did some­thing bad.” Once that aware­ness ex­is­ted, I re­spon­ded “Cool! Well guess what? I don’t think you did any­thing bad.“, and this time, it res­on­ated with Alex and Alex no longer felt judged. I then apo­lo­gized with “Also, I’m sorry you felt judged. Given that I didn’t ac­tu­ally feel you were do­ing some­thing bad, this was a mis­take on my part, and I’m sorry.”
This fur­ther cleared things up.

This whole pro­cess took about 15 minutes. In ret­ro­spect it might seem like we could have jumped straight to this solu­tion by me say­ing “I’m sorry I made you feel judged”, but that wasn’t an avail­able strategy ex ante, for two reas­ons:

(1) So­me­times I really am judging someone, and I’m okay with them feel­ing judged, be­cause I do in fact think they did some­thing wrong. As a res­ult of this will­ing­ness in my­self and oth­ers, it’s not al­ways be­liev­able to say “Sorry, I wish I hadn’t made you feel judged”. Indeed, to many this feels like a plat­it­ude. But, by ac­tu­ally go­ing through the work of ac­tu­ally un­rolling whether or not I thought Alex did a bad thing, and the other de­tails of what was go­ing on between us, we es­tab­lished enough shared clar­ity about the situ­ation that we man­aged to “get on the same page” what whether a bad thing was done, who thought or didn’t think that, and who mis­com­mu­nic­ated or didn’t mis­com­mu­nic­ate about it.

(2) There were many other things go­ing on that the Google doc helped to or­gan­ize and sift through without get­ting us lost. Without that func­tion­al­ity, I don’t think we would have been able to hone in on the par­tic­u­lar nar­rat­ive res­ol­u­tion above.

V. How gen­er­al­iz­able is this ‘un­rolling’ tech­nique?

The ap­plic­a­tion (IV) above not an isol­ated in­cid­ent. I’ve found­ing co-writ­ing and co-draw­ing to be ex­tremely valu­able in set­tling so­cial dis­agree­ments and con­flicts on at least 30 oc­ca­sions now, with at least 7 dif­fer­ent people, of vary­ing de­grees of in­clin­a­tion to­ward ex­pli­cit sym­bolic reas­on­ing. I ima­gine some in­clin­a­tion is ne­ces­sary, but much less than I would I have ex­pec­ted per­vi­ously. For in­stance, I’ve used this sort of un­rolling heur­istic fruit­fully in nu­mer­ous con­ver­sa­tions with folks close to me who

(1) didn’t go to col­lege or oth­er­wise study a sym­bolic dis­cip­line like math or lin­guist­ics, but who

(2) were gen­er­ally open minded enough to be will­ing to try out a “weird con­flict res­ol­u­tion tech­nique I’m ex­per­i­ment­ing with” where we sat down to­gether and tried un­packed our feel­ings in ex­pli­cit terms in a com­mon me­dium (usu­ally a Google doc).

I’ll de­fer to the find­ing of the broader com­munity here to see if oth­ers can make this sort of thing work use­fully.

VI. Rela­tion to “mi­asma” and “hype”

The concept of “mi­asma” that Dun­can is ges­tur­ing at in Com­mon Know­ledge and Mi­asma feels like a real so­cial phe­nomenon to me, suc­cinctly defin­able as “neg­at­ive un­groun­ded so­cial meta­cog­ni­tion”. There is such a thing as pos­it­ive un­groun­ded so­cial meta­cog­ni­tion, as well, which I think is nor­mally called “hype”, at least in Sil­icon Val­ley. I think both hype and mi­asma are fail­ures of group co­ordin­a­tion, and both are costly to re­solve, along the lines poin­ted out by Ben in The Costly Coordin­a­tion Mech­an­ism of Com­mon Know­ledge. However, the com­mu­nic­at­ive costs of resolv­ing these prob­lems can be sig­ni­fic­antly de­creased if people are aware of what they are. Both re­quire cre­at­ing and shar­ing of ideas in places of com­mon view, like writ­ing blog posts that a lot of people see each other com­ment­ing on, or hold­ing meet­ings that a lot of people can see each other at­tend­ing, or for com­plex top­ics, sit­ting to­gether and co-au­thor­ing a doc­u­ment.

VII. Apology

I’m sorry I put very little ef­fort into the ped­agogy of this post, due to hav­ing too little time to write it. Hope­fully it will be of some value any­way, due to much bet­ter posts hav­ing been writ­ten and cir­cu­lated on com­mon know­ledge re­cently, and due to the gen­eral in­tel­lec­tual health of LessWrong ap­pear­ing, to me, to be able to ab­sorb me­diocrely-ex­plained ideas and flesh them out into bet­ter ones. My sense is that the cul­ture here has been try­ing to move to­wards people not wait­ing un­til an idea is per­fectly elab­or­ated be­fore start­ing to talk about it, so to the ex­tent these ideas might be valu­able, I’m punt­ing to the com­munity to do more elab­or­a­tion and/​or dis­til­la­tion of them. Indeed, wish­ing not to be a part of a “com­mon know­ledge break­down” prob­lem is one reason I time-boxed two hours to write this post in­stead of wait­ing to im­prove it.