Link post

This post is an ex­er­cise in “iden­ti­fy­ing with the al­gorithm.” I’m a big fan of the prob­a­bil­is­tic method and ran­dom­ized al­gorithms, so my bi­ases will show.

How do hu­man be­ings pro­duce knowl­edge? When we de­scribe ra­tio­nal thought pro­cesses, we tend to think of them as es­sen­tially de­ter­minis­tic, de­liber­ate, and al­gorith­mic. After some self-ex­am­i­na­tion, how­ever, I’ve come to think that my pro­cess is closer to bab­bling many ran­dom strings and later fil­ter­ing by a heuris­tic. I think ver­bally, and my pro­cess for gen­er­at­ing knowl­edge is vir­tu­ally in­dis­t­in­guish­able from my pro­cess for gen­er­at­ing speech, and also quite similar to my pro­cess for gen­er­at­ing writ­ing.

Here’s a sim­plis­tic model of how this works. I try to build a co­her­ent sen­tence. At each step, to pick the next word, I ran­domly gen­er­ate words in the cat­e­gory (cor­rect part of speech, rele­vance) and sound them out one by one to see which con­tinues the sen­tence most co­her­ently. So, in­stead of de­liber­ately and care­fully gen­er­at­ing sen­tences in one go, the al­gorithm is some­thing like:

  1. Bab­ble. Use a weak and lo­cal filter to ran­domly gen­er­ate a lot of pos­si­bil­ities. Is the word the right part of speech? Does it lie in the same re­gion of thingspace? Does it fit the con­text?

  2. Prune. Use a strong and global filter to test for the best, or at least a satis­fac­tory, choice. With this word in the blank, do I ac­tu­ally be­lieve this sen­tence? Does the word have the right con­no­ta­tions? Does the whole thought read smoothly?

This is a bab­ble about em­brac­ing ran­dom­ness.

Baby Babble

Re­search on lan­guage de­vel­op­ment sug­gests that baby bab­ble is an di­rect fore­run­ner to lan­guage. You might imag­ine that in­fants learn by imi­ta­tion, and that baby bab­ble is just an im­perfect imi­ta­tion of words the baby hears, and progress oc­curs as they phys­iolog­i­cally adapt to bet­ter pro­duce those sounds. You would be wrong.

In­stead, in­fants are ini­tially ca­pa­ble of pro­duc­ing all the phonemes that ex­ist in all hu­man lan­guages, and they slowly prune out which ones they need via re­in­force­ment learn­ing. Based on the sounds that their par­ents pro­duce and re­spond to, ba­bies slowly filter out un­nec­es­sary phonemes. Their bab­bles be­gin to drift as they prune out more and more phonemes, and they start to com­bine syl­la­bles into proto-words. Bab­ble is the pro­cess of gen­er­at­ing ran­dom sounds, and look­ing for clues about which ones are use­ful. Some­thing some­thing re­in­force­ment learn­ing par­tially ob­serv­able Markov de­ci­sion pro­cess I’m in over my head.

So, we’ve learned that ba­bies use the Bab­ble and Prune al­gorithm to learn lan­guage. But this is quite a gen­eral al­gorithm, and evolu­tion is a con­ser­va­tive force. It stands to rea­son that hu­man be­ings might learn other things by a similar al­gorithm. I don’t think it’s a par­tic­u­larly con­tro­ver­sial sug­ges­tion that hu­man thought pro­ceeds roughly by cheaply con­struct­ing a lot of low-re­s­olu­tion hy­pothe­ses and then siev­ing from them by al­low­ing them to play out to their log­i­cal con­clu­sions.

The point I want to em­pha­size is that the al­gorithm has two dis­tinct phases, both of which can be in­de­pen­dently op­ti­mized. The stric­ter and stronger your Prune filter, the higher qual­ity con­tent you stand to pro­duce. But one com­mon bug is re­lated to this: if the qual­ity of your Bab­ble is much lower than that of your Prune, you may end up with noth­ing to say. Every­thing you can imag­ine say­ing or writ­ing sounds cringey or con­tent-free. Ten min­utes af­ter the con­ver­sa­tion moves on from that topic, your Bab­ble gen­er­a­tor fi­nally re­turns that witty come­back you were look­ing for. You’ll prob­a­bly spend your en­tire evening wait­ing for an op­por­tu­nity to force it back in.

Your pseu­do­ran­dom Bab­ble gen­er­a­tor can also be op­ti­mized, and in two differ­ent ways. On the one hand, you can im­prove the weak filter you’re us­ing, to in­crease the prob­a­bil­ity of gen­er­at­ing higher-qual­ity thoughts. The other way is one of the things named “cre­ativity”: you can try to elimi­nate sys­tem­atic bi­ases in the Bab­ble gen­er­a­tor, with the effect of hit­ting a more uniform sub­set of rele­vant con­cept-space. Ex­er­cises that might help in­clude ex­pand­ing your vo­cab­u­lary, read­ing out­side your com­fort zone, and en­gag­ing in the sub­tle art of non­stan­dard sen­tence con­struc­tion.

Poetry is Bab­ble Study

Poetry is at its heart an iso­la­tion ex­er­cise for your Bab­ble gen­er­a­tor. When cre­at­ing po­etry, you re­place your com­plex, inar­tic­u­late, and highly op­ti­mized Prune filter with a sim­ple, ex­plicit, and weird one that you’re not at­tached to. In­stead of pick­ing words that max­i­mize mean­ing, rele­vance, or so­cial sig­nals, you pick words with the right num­ber of syl­la­bles that rhyme cor­rectly and fol­low the right me­ter.

Now, with the Prune filter sim­plified and fixed, all the at­ten­tion is placed on the Bab­ble. What does it feel like to write a poem (not one of those free-form mod­ern ones)? Prob­a­bly most of your effort is spent Bab­bling al­most-words that fit the me­ter and rhyme scheme. If you’re any­thing like me, it feels al­most ex­actly like play­ing a game of Scrab­ble, fit­ting let­ters and syl­la­bles onto a board by trial and er­ror. Scrab­ble is just like po­etry: it’s all about be­ing good at Bab­ble. And no, I gra­ciously de­cline to write po­etry in pub­lic, even though Scrab­ble does con­ve­niently rhyme with Bab­ble.

Puns and word games are Bab­ble. You’ll no­tice that when you Bab­ble, each new word isn’t at all in­de­pen­dent from its pre­de­ces­sors. In­stead, Bab­ble is more like ini­ti­at­ing a ran­dom walk in your dic­tio­nary, one let­ter or syl­la­ble or in­fer­en­tial step at a time. That’s why word lad­ders are so ap­peal­ing—be­cause they stem from a nat­u­ral cog­ni­tive al­gorithm. I think Scott Alexan­der’s writ­ing qual­ity is great partly be­cause of his love of puns, a sure sign he has a great Bab­ble gen­er­a­tor.

If po­etry and puns are pho­netic Bab­ble, then “Deep Wis­dom” is se­man­tic Bab­ble. In­stead of ran­domly ar­rang­ing words by sound, we’re ar­rang­ing a rather small set of words to sound wise. More of­ten than not, “deep wis­dom” boils down to word games any­way, e.g. wise old say­ings:

“A blind per­son who sees is bet­ter than a see­ing per­son who is blind.”

“A proverb is a short sen­tence based on long ex­pe­rience.”

“Econ­omy is the wealth of the poor and the wis­dom of the rich.”

Read­ing is Out­sourc­ing Babble

Read­ing and con­ver­sa­tion out­source Bab­ble to oth­ers. In­stead of us­ing your own Bab­ble gen­er­a­tor, you flood your brain with other peo­ple’s words, and then ap­ply your Prune filter. Be­cause oth­ers have already Pruned once, the in­put is par­tic­u­larly high-qual­ity Bab­ble, and you reap par­tic­u­larly beau­tiful fruit. How many times have you read a thou­sand-page book, only to fix­ate on a hand­ful of strik­ing lines or pas­sages?

Prune goes into over­drive when you out­source Bab­ble. A bug I men­tioned ear­lier is hav­ing way too strict of a Prune filter, com­pared to the qual­ity of your Bab­ble. This oc­curs par­tic­u­larly to peo­ple who read and listen much more than they write or speak. When they fi­nally trudge into the at­tic and turn on that dusty old Bab­ble gen­er­a­tor, it doesn’t pro­duce thoughts nearly as co­her­ent, witty, or wise as their hy­per-de­vel­oped Prune filter is used to pro­cess­ing.

Im­pose Bab­ble tar­iffs. Your con­ver­sa­tion will never be as dry and smart as some­thing from a sit­com. If you can’t think of any­thing to say, re­lax your Prune filter at least tem­porar­ily, so that your Bab­ble gen­er­a­tor can catch up. Every­one starts some­where—Bab­bling plat­i­tudes is bet­ter than be­ing silent al­to­gether.

Con­versely, some peo­ple have no filter, and these are ex­actly the kind of peo­ple who don’t read or listen enough. If all your Bab­ble goes di­rectly to your mouth, you need to in­stall a bet­ter Prune filter. Im­pose ex­port tar­iffs.

The rea­son the Post­mod­ernism Gen­er­a­tor is so fun to read is be­cause com­put­ers are now ca­pa­ble of pro­duc­ing great Bab­ble. Read­ing po­etry and ran­domly gen­er­ated post­mod­ernism, talk­ing to chat­bots, these ac­tivi­ties all amount to frolick­ing in the un­canny valley be­tween Bab­ble and the Pruned.

Tower of Babble

A wise man once said, “Do not build Tow­ers out of Bab­ble. You wouldn’t build one out of Pizza, would you?”


NP is the God of Bab­ble. His law is: hu­mans will always be much bet­ter at ver­ify­ing wis­dom than pro­duc­ing it. There­fore, go forth and Bab­ble! After all, how did Shake­speare write his fa­mous plays, ex­cept by ran­domly press­ing keys on a key­board?

NP has a lit­tle brother called P. The law of P is: never try things you don’t un­der­stand com­pletely. Ran­domly thrash­ing around will get you nowhere.

P be­lieves him­self to be a God, an equal to his brother. He is not.