“How­ever many ways there may be of be­ing al­ive, it is cer­tain that there are vastly more ways of be­ing dead.”
-- Richard Dawkins

In the com­ing days, I ex­pect to be asked: “Ah, but what do you mean by ‘in­tel­li­gence’?” By way of un­tan­gling some of my de­pen­dency net­work for fu­ture posts, I here sum­ma­rize some of my no­tions of “op­ti­miza­tion”.

Con­sider a car; say, a Toy­ota Corolla. The Corolla is made up of some num­ber of atoms; say, on the rough or­der of 1029. If you con­sider all pos­si­ble ways to ar­range 1029 atoms, only an in­finites­i­mally tiny frac­tion of pos­si­ble con­figu­ra­tions would qual­ify as a car; if you picked one ran­dom con­figu­ra­tion per Planck in­ter­val, many ages of the uni­verse would pass be­fore you hit on a wheeled wagon, let alone an in­ter­nal com­bus­tion en­g­ine.

Even re­strict­ing our at­ten­tion to run­ning ve­hi­cles, there is an as­tro­nom­i­cally huge de­sign space of pos­si­ble ve­hi­cles that could be com­posed of the same atoms as the Corolla, and most of them, from the per­spec­tive of a hu­man user, won’t work quite as well. We could take the parts in the Corolla’s air con­di­tioner, and mix them up in thou­sands of pos­si­ble con­figu­ra­tions; nearly all these con­figu­ra­tions would re­sult in a ve­hi­cle lower in our prefer­ence or­der­ing, still rec­og­niz­able as a car but lack­ing a work­ing air con­di­tioner.

So there are many more con­figu­ra­tions cor­re­spond­ing to non­ve­hi­cles, or ve­hi­cles lower in our prefer­ence rank­ing, than ve­hi­cles ranked greater than or equal to the Corolla.

Similarly with the prob­lem of plan­ning, which also in­volves hit­ting tiny tar­gets in a huge search space. Con­sider the num­ber of pos­si­ble le­gal chess moves ver­sus the num­ber of win­ning moves.

Which sug­gests one the­o­ret­i­cal way to mea­sure op­ti­miza­tion—to quan­tify the power of a mind or mindlike pro­cess:

Put a mea­sure on the state space—if it’s dis­crete, you can just count. Then col­lect all the states which are equal to or greater than the ob­served out­come, in that op­ti­miza­tion pro­cess’s im­plicit or ex­plicit prefer­ence or­der­ing. Sum or in­te­grate over the to­tal size of all such states. Divide by the to­tal vol­ume of the state space. This gives you the power of the op­ti­miza­tion pro­cess mea­sured in terms of the im­prob­a­bil­ities that it can pro­duce—that is, im­prob­a­bil­ity of a ran­dom se­lec­tion pro­duc­ing an equally good re­sult, rel­a­tive to a mea­sure and a prefer­ence or­der­ing.

If you pre­fer, you can take the re­cip­ro­cal of this im­prob­a­bil­ity (1/​1000 be­comes 1000) and then take the log­a­r­ithm base 2. This gives you the power of the op­ti­miza­tion pro­cess in bits. An op­ti­mizer that ex­erts 20 bits of power can hit a tar­get that’s one in a mil­lion.

When I think you’re a pow­er­ful in­tel­li­gence, and I think I know some­thing about your prefer­ences, then I’ll pre­dict that you’ll steer re­al­ity into re­gions that are higher in your prefer­ence or­der­ing. The more in­tel­li­gent I be­lieve you are, the more prob­a­bil­ity I’ll con­cen­trate into out­comes that I be­lieve are higher in your prefer­ence or­der­ing.

There’s a num­ber of sub­tleties here, some less ob­vi­ous than oth­ers. I’ll re­turn to this whole topic in a later se­quence. Mean­while:

* A tiny frac­tion of the de­sign space does de­scribe ve­hi­cles that we would rec­og­nize as faster, more fuel-effi­cient, safer than the Corolla, so the Corolla is not op­ti­mal. The Corolla is, how­ever, op­ti­mized, be­cause the hu­man de­signer had to hit an in­finites­i­mal tar­get in de­sign space just to cre­ate a work­ing car, let alone a car of Corolla-equiv­a­lent qual­ity. This is not to be taken as praise of the Corolla, as such; you could say the same of the Hill­man Minx. You can’t build so much as a wooden wagon by saw­ing boards into ran­dom shapes and nailing them to­gether ac­cord­ing to coin­flips.

* When I talk to a pop­u­lar au­di­ence on this topic, some­one usu­ally says: “But isn’t this what the cre­ation­ists ar­gue? That if you took a bunch of atoms and put them in a box and shook them up, it would be as­ton­ish­ingly im­prob­a­ble for a fully func­tion­ing rab­bit to fall out?” But the log­i­cal flaw in the cre­ation­ists’ ar­gu­ment is not that ran­domly re­con­figur­ing molecules would by pure chance as­sem­ble a rab­bit. The log­i­cal flaw is that there is a pro­cess, nat­u­ral se­lec­tion, which, through the non-chance re­ten­tion of chance mu­ta­tions, se­lec­tively ac­cu­mu­lates com­plex­ity, un­til a few billion years later it pro­duces a rab­bit.

* I once heard a se­nior main­stream AI type sug­gest that we might try to quan­tify the in­tel­li­gence of an AI sys­tem in terms of its RAM, pro­cess­ing power, and sen­sory in­put band­width. This at once re­minded me of a quote from Dijk­stra: “If we wish to count lines of code, we should not re­gard them as ‘lines pro­duced’ but as ‘lines spent’: the cur­rent con­ven­tional wis­dom is so fool­ish as to book that count on the wrong side of the ledger.” If you want to mea­sure the in­tel­li­gence of a sys­tem, I would sug­gest mea­sur­ing its op­ti­miza­tion power as be­fore, but then di­vid­ing by the re­sources used. Or you might mea­sure the de­gree of prior cog­ni­tive op­ti­miza­tion re­quired to achieve the same re­sult us­ing equal or fewer re­sources. In­tel­li­gence, in other words, is effi­cient op­ti­miza­tion. This is why I say that evolu­tion is stupid by hu­man stan­dards, even though we can’t yet build a but­terfly: Hu­man en­g­ineers use vastly less time/​ma­te­rial re­sources than a global ecosys­tem of mil­lions of species pro­ceed­ing through biolog­i­cal evolu­tion, and so we’re catch­ing up fast.

* The no­tion of a “pow­er­ful op­ti­miza­tion pro­cess” is nec­es­sary and suffi­cient to a dis­cus­sion about an Ar­tifi­cial In­tel­li­gence that could harm or benefit hu­man­ity on a global scale. If you say that an AI is me­chan­i­cal and there­fore “not re­ally in­tel­li­gent”, and it out­puts an ac­tion se­quence that hacks into the In­ter­net, con­structs molec­u­lar nan­otech­nol­ogy and wipes the so­lar sys­tem clean of hu­man(e) in­tel­li­gence, you are still dead. Con­versely, an AI that only has a very weak abil­ity steer the fu­ture into re­gions high in its prefer­ence or­der­ing, will not be able to much benefit or much harm hu­man­ity.

* How do you know a mind’s prefer­ence or­der­ing? If this can’t be taken for granted, then you use some of your ev­i­dence to in­fer the mind’s prefer­ence or­der­ing, and then use the in­ferred prefer­ences to in­fer the mind’s power, then use those two be­liefs to testably pre­dict fu­ture out­comes. Or you can use the Min­i­mum Mes­sage Length for­mu­la­tion of Oc­cam’s Ra­zor: if you send me a mes­sage tel­ling me what a mind wants and how pow­er­ful it is, then this should en­able you to com­press your de­scrip­tion of fu­ture events and ob­ser­va­tions, so that the to­tal mes­sage is shorter. Other­wise there is no pre­dic­tive benefit to view­ing a sys­tem as an op­ti­miza­tion pro­cess.

* In gen­eral, it is use­ful to think of a pro­cess as “op­ti­miz­ing” when it is eas­ier to pre­dict by think­ing about its goals, than by try­ing to pre­dict its ex­act in­ter­nal state and ex­act ac­tions. If you’re play­ing chess against Deep Blue, you will find it much eas­ier to pre­dict that Deep Blue will win (that is, the fi­nal board po­si­tion will oc­cupy the class of states pre­vi­ously la­beled “wins for Deep Blue”) than to pre­dict the ex­act fi­nal board po­si­tion or Deep Blue’s ex­act se­quence of moves. Nor­mally, it is not pos­si­ble to pre­dict, say, the fi­nal state of a billiards table af­ter a shot, with­out ex­trap­o­lat­ing all the events along the way.

* Although the hu­man cog­ni­tive ar­chi­tec­ture uses the same la­bel “good” to re­flect judg­ments about ter­mi­nal val­ues and in­stru­men­tal val­ues, this doesn’t mean that all suffi­ciently pow­er­ful op­ti­miza­tion pro­cesses share the same prefer­ence or­der­ing. Some pos­si­ble minds will be steer­ing the fu­ture into re­gions that are not good.

* If you came across alien ma­chin­ery in space, then you might be able to in­fer the pres­ence of op­ti­miza­tion (and hence pre­sum­ably pow­er­ful op­ti­miza­tion pro­cesses stand­ing be­hind it as a cause) with­out in­fer­ring the aliens’ fi­nal goals, by way of notic­ing the fulfill­ment of con­ver­gent in­stru­men­tal val­ues. You can look at ca­bles through which large elec­tri­cal cur­rents are run­ning, and be as­ton­ished to re­al­ize that the ca­bles are flex­ible high-tem­per­a­ture high-am­per­age su­per­con­duc­tors; an amaz­ingly good solu­tion to the sub­prob­lem of trans­port­ing elec­tric­ity that is gen­er­ated in a cen­tral lo­ca­tion and used dis­tantly. You can as­sess this, even if you have no idea what the elec­tric­ity is be­ing used for.

* If you want to take prob­a­bil­is­tic out­comes into ac­count in judg­ing a mind’s wis­dom, then you have to know or in­fer a util­ity func­tion for the mind, not just a prefer­ence rank­ing for the op­ti­miza­tion pro­cess. Then you can ask how many pos­si­ble plans would have equal or greater ex­pected util­ity. This as­sumes that you have some prob­a­bil­ity dis­tri­bu­tion, which you be­lieve to be true; but if the other mind is smarter than you, it may have a bet­ter prob­a­bil­ity dis­tri­bu­tion, in which case you will un­der­es­ti­mate its op­ti­miza­tion power. The chief sign of this would be if the mind con­sis­tently achieves higher av­er­age util­ity than the av­er­age ex­pected util­ity you as­sign to its plans.

* When an op­ti­miza­tion pro­cess seems to have an in­con­sis­tent prefer­ence rank­ing—for ex­am­ple, it’s quite pos­si­ble in evolu­tion­ary biol­ogy for allele A to beat out allele B, which beats allele C, which beats allele A—then you can’t in­ter­pret the sys­tem as perform­ing op­ti­miza­tion as it churns through its cy­cles. In­tel­li­gence is effi­cient op­ti­miza­tion; churn­ing through prefer­ence cy­cles is stupid, un­less the in­terim states of churn­ing have high ter­mi­nal util­ity.

* For do­mains out­side the small and for­mal, it is not pos­si­ble to ex­actly mea­sure op­ti­miza­tion, just as it is not pos­si­ble to do ex­act Bayesian up­dates or to perfectly max­i­mize ex­pected util­ity. Nonethe­less, op­ti­miza­tion can be a use­ful con­cept, just like the con­cept of Bayesian prob­a­bil­ity or ex­pected util­ity—it de­scribes the ideal you’re try­ing to ap­prox­i­mate with other mea­sures.