Schelling Categories, and Simple Membership Tests

Fol­lowup to: Where to Draw the Boundaries?

Or there might be so­cial or psy­cholog­i­cal forces an­chor­ing word us­ages on iden­ti­fi­able Schel­ling points that are easy for differ­ent peo­ple to agree upon, even at the cost of some statis­ti­cal “fit”

The one comes to you and says, “That para­graph about Schel­ling points sounded in­ter­est­ing. What did you mean by that? Can you give an ex­am­ple?”

Sure. Pre­vi­ously on Less Wrong, in “The Uni­vari­ate Fal­lacy”, we stud­ied points sam­pled from two mul­ti­vari­ate prob­a­bil­ity dis­tri­bu­tions and , and showed that it was pos­si­ble to in­fer with very high prob­a­bil­ity which dis­tri­bu­tion a given point was sam­pled from, de­spite sig­nifi­cant over­lap in the marginal dis­tri­bu­tions for any one vari­able con­sid­ered in­di­vi­d­u­ally.

From the stand­point of “the way to carve re­al­ity at its joints, is to draw your bound­aries around con­cen­tra­tions of un­usu­ally high prob­a­bil­ity den­sity in Thingspace”, the cor­rect cat­e­go­riza­tion of the points in that ex­am­ple is clear. We have two clearly dis­t­in­guish­able clusters. The con­di­tional in­de­pen­dence prop­erty is satis­fied: given a point’s cluster-mem­ber­ship, know­ing one of the doesn’t tell you any­thing about for ji. So we should draw a cat­e­gory bound­ary around each cluster. Ob­vi­ously. We might ask hy­pophor­i­cally: what could pos­si­bly change this moral?

More con­straints on the prob­lem, that’s what!

Sup­pose you needed to co­or­di­nate with some­one else to make de­ci­sions about these points—that is, it’s im­por­tant not just that you and your part­ner make good de­ci­sions, but also that you make the same de­ci­sion—but that each of you only got to ob­serve one co­or­di­nate from each point. As we saw, the pre­dic­tive work we get from cat­e­gory-mem­ber­ship in this sce­nario is spread across many vari­ables: if you only get to ob­serve a few di­men­sions, you have a lot of un­cer­tainty about cluster-mem­ber­ship (which car­ries over into ad­di­tional un­cer­tainty about the other di­men­sions that you haven’t ob­served, but which af­fect the ex post qual­ity of your de­ci­sion).

If you and your part­ner were both ideal Bayesian calcu­la­tors who could com­mu­ni­cate costlessly, you would share your ob­ser­va­tions, work out the cor­rect prob­a­bil­ity, and use that to make op­ti­mal de­ci­sions. But sup­pose you couldn’t do that—ei­ther be­cause com­mu­ni­ca­tion is ex­pen­sive, or your part­ner was bad at math, or any other rea­son. Then it would be sad if you hap­pened to see = 2 and said “It’s an A (prob­a­bly)!”, and your part­ner hap­pened to see = 3 and said “It’s a B (I think)!”, and the two of you made in­con­sis­tent de­ci­sions.

Okay, now sup­pose that there’s ac­tu­ally a forty-first, bi­nary, vari­able that I didn’t tell you about ear­lier, dis­tributed like so:

Ob­serv­ing gives you ≈ 1.585 bits of ev­i­dence about cluster-mem­ber­ship, which is more than the

≈ 1.18 bits you can get from any one ob­ser­va­tion of one of the for i ∈ {1...40}.

If you and your part­ner can both ob­serve , you might end up want­ing to base your shared cat­e­gories and lan­guage on that—call­ing a point an “A” if it has = 0, even though such points ac­tu­ally came from a full quar­ter of the time—even if it­self has no effect on the qual­ity of your de­ci­sions, and what you ac­tu­ally care about is wholely de­ter­mined by the val­ues of through ! It’s not the in­ten­sion you would pick if you could make (and share) more ob­ser­va­tions—but ex hy­poth­esi, you can’t.

If you and your part­ner only get to ob­serve one vari­able, is your best choice—the sin­gle vari­able that gives you the most in­for­ma­tion about the “nat­u­ral” cluster-mem­ber­ship. That also makes it a Schel­ling point—if you and your part­ner didn’t get to com­m­mu­ni­cate in ad­vance about how you want to draw your shared cat­e­gory bound­aries, you could pick as your defin­ing ob­ser­va­tion and be pretty con­fi­dent your part­ner would make the same choice. We could imag­ine an even more pes­simistic sce­nario in which the Schel­ling point cat­e­gory defi­ni­tion (a set of vari­ables that “stuck out” from all the oth­ers) was less pre­dic­tive than some other can­di­dates—but if you couldn’t co­or­di­nate to pick one of the more pre­dic­tive cat­e­gory sys­tems, you might be stuck with the Schel­ling point.

In con­clu­sion, the right cat­e­gories to use given con­straints on com­mu­ni­ca­tion and ob­ser­va­tion, might be differ­ent from the cat­e­gory bound­aries you would draw from a “God’s eye view”, in part be­cause con­sid­er­a­tion of which cat­e­gories are easy for differ­ent agents to co­or­di­nate on is rele­vant, not just raw in­for­ma­tion-the­o­retic ex­pres­sive power. Thus, “Schel­ling cat­e­gories.”

Thanks for read­ing!


The one says, “No, I meant, like, a real world ex­am­ple, not some dumb math thing for nerds. What is this post re­ally about?”

It’s about … math? Or like, the re­la­tion­ship be­tween math and hu­man nat­u­ral lan­guage? Like, I was won­der­ing what “sec­ond-or­der” caveats or com­pli­ca­tions there might be to the ba­sic “carve re­al­ity at the joints” moral of our stan­dard Bayesian philos­o­phy of lan­guage, and some of the peo­ple I’ve been col­lab­o­rat­ing with lately had been talk­ing a lot about the im­por­tance of in­ter­sub­jec­tive episte­mol­ogy—that is, shared map­mak­ing, so—

“But where’s the ac­tion­able take­away? What’s your real agenda here, huh?”

Oh. One of those read­ers, I see. Fine, I can prob­a­bly think of some—how do you say?—”ap­pli­ca­tions.”

Um­mmm …

Let’s see …

Okay, here’s some­thing, maybe. What’s the deal with the age of ma­jor­ity?

So­ciety needs to de­cide who it wants to be al­lowed to vote, stand trial, sign con­tracts, serve in the mil­i­tary, &c. Whether it’s a good idea for a par­tic­u­lar per­son to have these priv­ileges pre­sum­ably de­pends on var­i­ous rele­vant fea­tures of that per­son: things like cog­ni­tive abil­ity, fore­sight, wis­dom, rele­vant life ex­pe­riences, &c. In par­tic­u­lar, it would be pretty weird for some­one’s fit­ness to vote to di­rectly de­pend on how many times the Earth has gone around the sun since they were born. What does that num­ber have to do with any­thing?

It doesn’t! But if So­ciety isn’t well-co­or­di­nated enough to agree on the ex­act pre­req­ui­sites for vot­ing and how to mea­sure them, but can agree that most twenty-five-year-olds have them and most eleven-year-olds don’t, then we end up choos­ing some ar­bi­trary age cut­off as the crite­rion for our “le­gal adult­hood” so­cial con­struct. It works, but it’s just a le­gal fic­tion—and not nec­es­sar­ily a par­tic­u­larly good fic­tion, as any bright teenagers read­ing this will doubtlessly at­test.

If I told you that a par­tic­u­lar four­teen-year-old was very “ma­ture”, that’s a con­tent­ful state­ment: we have shared mean­ing at­tached to the word ma­ture, such that my de­scribing some­one that way con­strains your an­ti­ci­pa­tions. But it’s a re­ally com­pli­cated mean­ing, a statis­ti­cal sig­nal in be­hav­ior that your brain can pick up on, but which isn’t par­tic­u­larly ver­ifi­able to oth­ers who might have rea­sons to doubt my char­ac­ter as­sess­ment. In con­trast, age is easy for ev­ery­one to agree on. We could imag­ine some hy­po­thet­i­cal sci­ence-fic­tional So­ciety that used brain scans and some so­phis­ti­cated ma­chine-learn­ing clas­sifer to de­ter­mine which cit­i­zens get which priv­ileges—but in our dumber, poorer world, cal­en­dars and sub­trac­tion will have to do.

In terms of Scott Garrabrant’s tax­on­omy of ap­pli­ca­tions of Good­hart’s law, this is re­gres­sional Good­hart: So­ciety wants to se­lect for ma­tu­rity, chooses age as a proxy, and in the pro­cess, ends up grant­ing or with­hold­ing priv­ileges that a more dis­crim­i­nat­ing So­ciety maybe wouldn’t.

The age of ma­jor­ity is a case of re­plac­ing a com­pli­cated, illeg­ible cat­e­gory (“ma­tu­rity”, the kind of ab­stract thing you might want to model as a cluster in a forty- or forty-one-di­men­sional space) with a sim­ple mem­ber­ship test (an age cut­off that ev­ery­one knows how to com­pute). Differ­ent peo­ple might make make differ­ent sub­jec­tive (but not ar­bi­trary) judge­ments of the com­pli­cated, illeg­ible cat­e­gory, so in or­der to get a more in­ter­sub­jec­tively ro­bust ver­dict on cat­e­gory-mem­ber­ship, we rely on an ob­jec­tive mea­sure­ment that ev­ery­one can agree on.

If no con­ve­nient ob­jec­tive mea­sure­ment is available, an­other strat­egy is pos­si­ble: we can del­e­gate to some canon­i­cal trusted au­thor­ity, whose opinion of the com­pli­cated cat­e­gory will take prec­dence over ev­ery­one else’s. An ex­am­ple of this is com­mod­ity grad­ing stan­dards. What is a “Grade AA” egg? Well, there’s a com­pli­cated defi­ni­tion writ­ten down in a man­ual some­where that you could try ap­ply­ing your­self—but for most peo­ple, Grade AA eggs are sim­ply “those which have been cer­tified as Grade AA by the USDA.”[1]

It’s even pos­si­ble for the “sim­ple ob­jec­tive mea­sure­ment” and “del­e­gate to an au­thor­ity’s sub­jec­tive judge­ment” strate­gies to be com­bined. In “The Ide­ol­ogy Is Not the Move­ment”, the im­mor­tal Scott Alexan­der writes about his model of the gen­e­sis of so­cial groups—

Pre-ex­ist­ing differ­ences are the raw ma­te­ri­als out of which tribes are made. A good tribe com­bines peo­ple who have similar in­ter­ests and styles of in­ter­ac­tion even be­fore the ethno­gen­e­sis event. Any de­scrip­tion of these differ­ences will nec­es­sar­ily in­volve stereo­types, but a lot of them should be hard to ar­gue. [...] There are sub­tle habits of thought, not yet de­scribed by any word or sen­tence, which athe­ists are more likely to have than other peo­ple. [...]

The ral­ly­ing flag is the ex­plicit pur­pose of the tribe. It’s usu­ally a be­lief, event, or ac­tivity that get peo­ple with that spe­cific pre-ex­ist­ing differ­ence to­gether and ex­cited. Often it brings pre­vi­ously la­tent differ­ences into sharp re­lief. Peo­ple meet around the ral­ly­ing flag, en­counter each other, and say “You seem like a kin­dred soul!” or “I thought I was the only one!” Usu­ally it sug­gests some course of ac­tion, which pro­vides the tribe with a pur­pose.

Eliezer Yud­kowsky’s “A Fable of Science and Poli­tics” de­picts a fic­tional un­der­ground so­ciety split be­tween two such tribes: an pre­dom­i­nantly ur­ban tribe that be­lieves that the un­seen sky is blue (and fa­vors an in­come tax, strong mar­riage laws, and an Earth-cen­tric cos­mol­ogy), and pre­dom­i­nanty ru­ral one that be­lieves that the sky is green (and fa­vors mer­chant taxes, no-fault di­vorce, and a he­lio­cen­tric cos­mol­ogy). In this story, be­liefs about the color of the sky are func­tion­ing as the “ral­ly­ing flag” for tribe-for­ma­tion in Alexan­der’s model—and as a Schel­ling point for cat­e­gory defi­ni­tion.

We don’t know how to talk about the preëx­ist­ing un­defin­able habits of thought that make so­cial groups work—it’s hard to ex­plic­itly ar­tic­u­late what ex­act statis­ti­cal reg­u­lar­ity our brains have de­tected in five-and-more-di­men­sional lo­cale/​sky-be­lief/​tax-be­lief/​di­vorce-be­lief/​cos­mol­ogy/​&c.-space. (Although we could imag­ine some hy­po­thet­i­cal sci­ence-fic­tional So­ciety that did know how to ar­tic­u­late it, and con­se­quently had richer forms of so­cial and poli­ti­cal or­ga­ni­za­tion than our own.) It’s a lot sim­pler to talk about whether some­one has pledged alle­giance to the ral­ly­ing flag: just ask some­one, “What color do you be­lieve the sky is?” (us­ing sky-be­liefs as as an “ob­jec­tive” sim­ple mem­ber­ship test), or sim­ply, “Are you a Blue or a Green?” (del­e­gat­ing the clas­sifi­ca­tion prob­lem to the per­son them­selves as the au­thor­ity whose dis­cern­ment is to be trusted)—and what­ever they say, that’s what they are.

Well, prob­a­bly. We’ve seen that ob­jec­tive mea­sure­ments like age are sub­ject to re­gres­sional Good­hart, but the del­e­ga­tion-to-au­thor­ity strat­egy is fur­ther­more sub­ject to ad­ver­sar­ial Good­hart: once a cat­e­gory-mem­ber­ship test has been es­tab­lished, some agents might have an in­cen­tive to cre­ate ex­am­ples that pass the test, but don’t have the com­pli­cated, illeg­ible prop­er­ties than made the test a use­ful proxy in the first place.

We’ve seen this, for ex­am­ple, with ti­tle in­fla­tion: we ex­pect the “job ti­tle” (the words that get printed on busi­ness cards or im­mi­gra­tion spon­sor­ship forms) to be the canon­i­cal de­scrip­tion of what some­one “does”, even if the va­garies of the work­day en­com­pass many tasks,[2] and an alien an­thro­pol­o­gist tasked with ob­serv­ing the work­site and sum­ma­riz­ing what each of the hu­mans did might slice up her ob­ser­va­tions into cat­e­gories with lit­tle re­sem­blance to the com­pany’s for­mal org chart. But since we don’t know how to do the ob­vi­ous thing and av­er­age over all pos­si­ble alien an­thro­pol­o­gists weighted by sim­plic­ity, we can only rely on the org chart—which peo­ple have poli­ti­cal in­cen­tives to ma­nipu­late, with the re­sult that ev­ery­one in the fi­nance in­dus­try is a “vice pres­i­dent” of some sort or an­other.

But “Vice Pres­i­dent” has a literal mean­ing. Or it used to. Vice, “in place of; sub­or­di­nate to.” Pres­i­dent, one who pre­sides over some de­liber­a­tive body. The ad­ver­sar­ial-Good­hart pres­sures on lan­guage “ex­ploit[ ] the trust we have in a func­tion­ing piece of lan­guage un­til it’s lost all mean­ing”.

So for read­ers who de­mand a take­away be­yond just an edge case in the math, per­haps take away this: co­or­di­na­tion is costly. From the stand­point of lan­guage as an AI ca­pa­bil­ity, the so­cial con­struc­tions that fee­ble hu­mans need in or­der to work to­gether may be un­avoid­ably dumbed-down for mass con­sump­tion, but that’s no rea­son to not as­pire to the true pre­ci­sion of the Bayes-struc­ture to what­ever ex­tent pos­si­ble.

(Thanks to Ben Hoff­man for the et­y­mol­ogy of “Vice Pres­i­dent.”)


  1. Or the analo­gous agency in your coun­try. ↩︎

  2. When I worked in a su­per­mar­ket, two days a week I did Tracy’s book­keep­ing/​cus­tomer-ser­vice job while Tracy had her week­end, which en­tailed count­ing the money from last night’s tills and swap­ping in new coin­mags and com­plet­ing the FSM re­port and an­swer­ing the phone and sel­l­ing money or­ders and cov­er­ing the flo­ral stand when the flo­ral lady was on lunch, &c. I’m ac­tu­ally not sure what offi­cial name this role had in Safe­way’s offi­cial org chart. We just called it “the booth.” ↩︎