Order and Chaos

Fol­lows from:

Map and Territory

Bab­ble and Prune

Warn­ing: I strongly recom­mend not us­ing the con­cepts in this se­quence to try and build a gen­er­al­ized ar­tifi­cial in­tel­li­gence. Th­ese con­cepts de­scribe how hu­mans func­tion as con­scious en­tities, and hu­mans are not known for be­ing safe or friendly to hu­man val­ues. Hu­man limi­ta­tions cur­rently provide the only check on hu­man abuse of power, and build­ing some­thing with hu­man cog­ni­tive abil­ities but with­out those limi­ta­tions would be ill-ad­vised. Hu­man.


This se­ries of ar­ti­cles is about ap­plied metacog­ni­tion, lay­ing the ground­work for de­vel­op­ing the skills to ap­proach and effec­tively han­dle any type of prob­lem or situ­a­tion.

The con­cepts in this par­tic­u­lar ar­ti­cle may already be fa­mil­iar. I’m pre­sent­ing them here be­cause we will use them in later ar­ti­cles to de­rive the differ­ent types of skills and ex­plain how those skills work and how they fit into our toolbox.

The Map and the Territory

A pre­vi­ous ar­ti­cle, No Really, Why Aren’t Ra­tion­al­ists Win­ning?, es­tab­lished that skills are highly com­pressed pro­ce­du­ral in­for­ma­tion. In our se­quence pre­mier, The Foun­da­tional Toolbox for Life: In­tro­duc­tion, we looked at them from a differ­ent an­gle: skills start as paradigms which filter out in­for­ma­tion. We de­velop our paradigms into skills by cal­ibrat­ing them with ex­pe­rience to pro­duce use­ful an­swers to prob­lems in a prac­ti­cal time frame.

Now we will look at the map/​ter­ri­tory dis­tinc­tion and how we use it to define the ba­sic build­ing blocks of cog­ni­tion it­self. Once we’ve done that, we can fi­nally move on to what we can build with those blocks.

Every skil­lset, from sci­ence to art to ath­let­ics to man­age­ment, re­quires an ex­plicit or im­plicit men­tal model of cer­tain as­pects of the world: a map. Every per­son has at least one map in their brain, which rep­re­sents the ter­ri­tory that is the real world, or at least the part they deal with. The map lets a per­son pre­dict the out­comes of their ac­tions, and thereby al­lows them to effec­tively nav­i­gate the ter­ri­tory and change it in pur­suit of their de­sires. Without the map, there would be no way for a per­son to pre­dict which op­tions lead to de­sir­able out­comes.

Even prim­i­tive life forms have evolved rudi­men­tary maps. Their in­stincts rep­re­sent the effects of their po­ten­tial re­sponses to spe­cific stim­uli on the prob­a­bil­ity that they will sur­vive and re­pro­duce. The cor­re­la­tions en­coded in these in­stincts are a nar­row, low-re­s­olu­tion map of the or­ganisms’ na­tive en­vi­ron­ments.

How­ever, in­stinct maps are up­dated by the pro­cesses of mu­ta­tion and nat­u­ral se­lec­tion—in other words, chance and death. Each in­di­vi­d­ual is stuck with an in­stinct map that ei­ther suc­ceeds or fails fatally. Hu­mans gen­er­ally want to im­prove their mod­els of re­al­ity in less lethal ways, so they use their more so­phis­ti­cated neu­ral hard­ware to learn about the world and up­date their maps on the in­di­vi­d­ual and cul­tural lev­els rather than on the species genome level.

Order and Chaos

The map’s re­la­tion­ship with the ter­ri­tory cre­ates a fun­da­men­tal di­chotomy that helps define ev­ery tool in our toolbox: the du­al­ity of or­der and chaos.

“Order” rep­re­sents the de­gree to which the map ac­cu­rately re­flects the ter­ri­tory. This ac­cu­racy is mea­sured by how well the map makes pre­dic­tions. In short, or­der is what we say we “know”. When we speak of re­quire­ments and limits, what must hap­pen or what can­not hap­pen, we are speak­ing of or­der.

Ad­di­tion­ally, “or­der” can re­fer to how eas­ily knowl­edge and in­for­ma­tion can be com­pressed, or how much in­for­ma­tion we can de­rive from a small sam­ple size. Pat­terns across time or space are called “or­derly” be­cause know­ing only a frac­tion of the pat­tern can en­able us to pre­dict the rest. For all ter­ri­to­ries of a given size, the more or­derly ones re­quire fewer bits of in­for­ma­tion to de­scribe them with a map. For ex­am­ple, a bilat­er­ally sym­met­ri­cal ob­ject al­lows you to pre­dict what is on one side if you have already seen the other side, so you can de­scribe it in full by show­ing only one side and defin­ing the plane of sym­me­try. The map of a par­tic­u­larly or­derly ter­ri­tory might trans­late to a few sam­ple data points and a rel­a­tively sim­ple rule.

By con­trast, “chaos” rep­re­sents the omis­sions and er­rors in the map, the de­gree to which the map fails to ac­cu­rately rep­re­sent the ter­ri­tory. Chaos is the “un­known”. When we speak of pos­si­bil­ities and un­cer­tain­ties, of what may or may not hap­pen, we are speak­ing of chaos.

The un­knowns of chaos in­cludes both un­known un­knowns (pure chaos) and known un­knowns (chaos bounded by or­der). Pure chaos man­i­fests as out­side con­text prob­lems or black swan events, like be­ing in­vaded by a con­ti­nent you never sus­pected ex­isted. How­ever, much of the chaos that adult hu­mans ex­pe­rience is bounded by or­der. Although they don’t know ex­actly what will hap­pen, they feel fairly cer­tain it will fall within a range of “nor­mal” events. The roll of a die pro­vides a more spe­cific ex­am­ple of bounded chaos, since we know ev­ery pos­si­ble face value even if we don’t know which one it will be. A trusted prob­a­bil­ity dis­tri­bu­tion also im­poses some cer­tainty on un­pre­dictable out­comes, at least with large sam­ple sizes. Even if in­di­vi­d­ual mea­sure­ments may vary, we know roughly what the data on the group as a whole will look like.

When­ever some­thing you thought you knew turns out to be false or in­com­plete, that is chaos as well. The truest knowl­edge of the ter­ri­tory is limited to our scat­tered data points of di­rect ex­pe­rience, and we cre­ate our maps to in­ter­po­late and ex­plain those data points as best we know how. When­ever we get a new data point that falsifies the map we were us­ing, when we try to pre­dict the ter­ri­tory and fail, it is an­other man­i­fes­ta­tion of chaos.

More­over, chaos can re­fer to how difficult it is to com­press in­for­ma­tion, or to figure out the de­tails of a situ­a­tion from limited data. A situ­a­tion de­scribed as “chaotic” is difficult to pre­dict be­cause the in­for­ma­tion you have about it can­not be used to de­rive the in­for­ma­tion you want. For ex­am­ple, in a messy room, the knowl­edge of one sock’s lo­ca­tion does not al­low you to lo­cate its pair.

Although (or be­cause) they are op­po­site con­cepts, or­der and chaos are more or less in­ex­tri­ca­ble. The known and un­known are pre­sent to vary­ing de­grees in al­most ev­ery situ­a­tion you en­counter, be­cause they are es­sen­tial to con­scious ex­is­tence as we know it. We of­ten say that perfectly cer­tain knowl­edge of the ter­ri­tory is im­pos­si­ble, but we don’t clas­sify ev­ery­thing as com­pletely un­known, ei­ther. In­stead of a bi­nary la­bel of “known” or “un­known,” we have gra­di­ents of cer­tainty that in­form how much of our re­sources and safety we are will­ing to bet on var­i­ous un­knowns.

As a tech­ni­cal ex­pla­na­tion of these con­cepts, “chaos” sim­ply de­scribes a rel­a­tively smooth and some­what even dis­tri­bu­tion of prob­a­bil­ity mass across a range of hy­pothe­ses, where no hy­poth­e­sis in the range is con­sid­ered over­whelm­ingly more likely than an­other. “Order” de­scribes a sharper, un­even dis­tri­bu­tion where prob­a­bil­ity mass is con­cen­trated into a rel­a­tively small num­ber of hy­pothe­ses. If (as usual) you have a sub­set of hy­pothe­ses that are over­whelm­ingly more likely than all oth­ers but roughly equal in prob­a­bil­ity with each other, that’s bounded un­cer­tainty: chaos bounded by or­der (or chaos in­her­ent in or­der, de­pend­ing on which one you want to im­ply is dom­i­nant).

You may have gath­ered that or­der and chaos are also sub­jec­tive in their ap­pli­ca­tion. A ter­ri­tory can­not be in­trin­si­cally “or­derly” or “chaotic” with­out refer­ence to a given map (or com­pres­sion al­gorithm). A situ­a­tion will ap­pear more or less chaotic to you in pro­por­tion to your ig­no­rance of it. After all, con­fu­sion (or lack thereof) is in the map.

Order and chaos are also im­plic­itly based on what in­for­ma­tion peo­ple con­sider im­por­tant. The die roll men­tioned above is “bounded by or­der” be­cause it has a finite num­ber of defined re­sults. How­ever, the rea­son the num­ber of defined re­sults is so low is be­cause we don’t pay any at­ten­tion to the lo­ca­tion at which the die comes to rest, or the di­rec­tion it faces, or the amount of time it takes to stop rol­ling…

The fact that these defi­ni­tions of or­der and chaos are rel­a­tive rather than ob­jec­tive is to be ex­pected, be­cause all the con­cepts in our toolbox are based on solv­ing prob­lems. Prob­lems can only be defined in terms of a per­son’s de­sires, what sorts of ob­sta­cles stand in the way of those de­sires, and what the per­son can do to over­come those ob­sta­cles.

The next sec­tion deals with how our minds pro­cess or­der and chaos. Un­der­stand­ing how we deal with the shape of what we know and what we don’t know (and what we don’t know we don’t know) is vi­tal for de­scribing how our skills work.

Guess­ing and Checking

At the most fun­da­men­tal level of men­tal ac­tivity that is still com­plex enough to be rec­og­nized as men­tal, we find two pro­cesses. Th­ese pro­cesses ex­plore chaos and or­der, re­spec­tively, so that the mind can de­velop and re­fine its map. I call these pro­cesses “guess­ing and check­ing”. Else­where on this site, they are known as “bab­ble and prune”. As far as I can tell, these phrases re­fer to the same pair of con­cepts.

Guess­ing is more or less free as­so­ci­a­tion: it links our cur­rent ex­pe­riences and thoughts with any con­cepts that are re­motely similar, and calls our at­ten­tion to those con­cepts. To guess is to throw one’s map up against the ter­ri­tory in var­i­ous ways (with­out judg­ing the re­sults—that’s where check comes in). Guess­ing is the mind wran­gling chaos. It fol­lows pos­si­bil­ities based on an ini­tial idea and makes them con­crete in the mind. It al­lows the mind to model (and there­fore ad­dress) the un­known ter­ri­tory by giv­ing shapes to the po­ten­tial that lurks within.

Check­ing is the pro­cess by which we judge whether a con­cept is rele­vant to the cur­rent situ­a­tion. It eval­u­ates how we are ap­ply­ing the map to the ter­ri­tory, and the pre­dic­tions we make from it, by com­par­ing them to other ob­ser­va­tions of the ter­ri­tory or to mem­o­ries that our guess­ing has sum­moned up. Based on this eval­u­a­tion, the check ac­cepts or re­jects the ac­cu­racy (pre­dic­tive util­ity) of the con­cept in the given situ­a­tion. Check­ing is the mind wran­gling or­der, be­cause it de­cides what in­for­ma­tion gets to be­come and re­main part of the map. It al­lows the mind to pro­duce and cu­rate knowl­edge by judg­ing how well the map of the known matches the ter­ri­tory (or other parts of the map) in the way that guess­ing has ap­plied it.

To illus­trate how these cog­ni­tive pro­cesses work, we can look at what hap­pens when each of them is shut off.

All guess­ing and no check­ing would be like an in­co­her­ent dream, or (peo­ple tell me) the effects of some recre­ational drugs: a pa­rade of ran­dom im­pres­sions. It would con­sist of com­plete free as­so­ci­a­tion, but noth­ing for as­sess­ing cor­re­spon­dence with re­al­ity and fil­ter­ing out what doesn’t fit.

In­versely, all check­ing and no guess­ing would al­low one to ap­ply a sin­gle con­cept, but one would have no abil­ity to up­date the paradigm of how to ap­ply it. For in­stance, an en­tity with check­ing but no guess­ing might be able to clas­sify or­ganisms as cats or dogs, but it wouldn’t be able to re­al­ize that some or­ganisms are nei­ther (un­less it already had a la­bel for that).

From these ex­am­ples, it is clear that both guess­ing and check­ing are nec­es­sary as­pects of any skill, be­cause both are nec­es­sary to gen­er­ate and cal­ibrate our maps.

As a more tech­ni­cal ex­pla­na­tion of guess­ing and check­ing, guess­ing iter­ates through lo­ca­tions in hy­poth­e­sis space. How­ever, hy­poth­e­sis space—the space of pos­si­ble maps—is the­o­ret­i­cally in­finite, with in­finite di­men­sions, and is there­fore non-or­dered (that is, it doesn’t have a lin­ear se­quence). To make iter­a­tion through un­limited pos­si­bil­ities com­pu­ta­tion­ally tractable, our brains use free as­so­ci­a­tion. The brain keys off of im­me­di­ate sen­sory in­puts or thoughts to find pos­si­bly re­lated con­cepts, then keys off of those con­cepts to find more dis­tantly re­lated con­cepts, and so on. Through this pro­cess, guess­ing takes us to the most salient-seem­ing lo­ca­tions in hy­poth­e­sis space. Each lo­ca­tion vis­ited, cor­rect or not, is added to a map of pos­si­bil­ities as a can­di­date for rep­re­sent­ing the ter­ri­tory.

As we guess each hy­poth­e­sis in turn, check­ing will ac­cept or re­ject the hy­poth­e­sis with vary­ing de­grees of con­fi­dence by pump­ing prob­a­bil­ity mass into or out of it, re­dis­tribut­ing the prob­a­bil­ity mass across the range of hy­poth­e­sis space we are ex­plor­ing. It performs this re­dis­tri­bu­tion by up­dat­ing our map of pos­si­bil­ities, re­vis­ing the de­grees of cer­tainty across the board based on its eval­u­a­tion of each op­tion.

Nat­u­rally, if we ex­am­ine all the ma­jor hy­pothe­ses and de­cide they’re still equally likely, then our pump­ing has can­celed it­self out. If our check­ing, based on our prior prob­a­bil­ities, has de­cided they all match quite well, then we’re still un­de­cided. If check­ing de­cides the hy­pothe­ses all match equally poorly, it may be that some­thing im­prob­a­ble hap­pened, or that our guess­ing didn’t go far enough to come up with a more likely hy­poth­e­sis, or that our check re­jected some­thing in er­ror be­cause we were ig­no­rant of a fac­tor mak­ing it more prob­a­ble.

Distinct and Subliminal

There’s one other di­chotomy that we need to finish lay­ing the ground­work for the ba­sic skills: dis­tinct ver­sus sub­limi­nal.

When a guess­ing or check­ing ac­tivity takes place in our brains, it can do so in two modes.

First, it can run dis­tinctly, where there is an ex­plicit record (a mem­ory) of the iter­a­tion pro­cess and we are fully aware of what pos­si­bil­ities or im­pli­ca­tions we are con­sid­er­ing. We re­fer to dis­tinct pro­cesses as tak­ing place in our Sys­tem 2, which might also be called the “man­ual” sys­tem.

Se­cond, guess­ing or check­ing can run sub­limi­nally (“be­low the thresh­old”) where there is no ex­plicit record of the pro­cess, and we are only left aware of the end re­sult, if even that. We fre­quently form be­liefs and de­ci­sions based on sub­limi­nal pro­cesses with­out re­al­iz­ing we’ve done so, to our detri­ment or benefit. We say these pro­cesses are part of Sys­tem 1, of­ten called the “au­to­matic” sys­tem.

The modes in which our guess­ing and check­ing pro­cesses run de­ter­mine what sort of maps we use and how we up­date them. The maps we use define the types of skills that we em­ploy, what as­pects of situ­a­tions they deal with, and their ad­van­tages and dis­ad­van­tages. We’ll go into more de­tail on these maps and what hap­pens when cog­ni­tive pro­cesses run in var­i­ous modes in the next part of the se­quence.


All skills in­volve both guess­ing and check­ing in or­der to up­date our maps. What differ­en­ti­ates one skill from an­other is how each pro­cess runs: dis­tinctly or sub­limi­nally. Th­ese modes of guess­ing and check­ing in­form what sorts of fea­tures a skill’s map con­tains and there­fore what as­pects of the ter­ri­tory its map rep­re­sents. The di­chotomy of or­der and chaos, de­scribing the de­gree to which a map cor­re­sponds with a ter­ri­tory, is a core con­cept for dis­t­in­guish­ing the differ­ent tools in our toolbox.

In the next ar­ti­cle I’ll in­tro­duce the ba­sic mind­sets and ex­plain what sorts of maps they use and how they are defined by how they guess and check.