# Optimization

“How­ever many ways there may be of be­ing al­ive, it is cer­tain that there are vastly more ways of be­ing dead.”
-- Richard Dawkins

In the com­ing days, I ex­pect to be asked: “Ah, but what do you mean by ‘in­tel­li­gence’?” By way of un­tan­gling some of my de­pen­dency net­work for fu­ture posts, I here sum­ma­rize some of my no­tions of “op­ti­miza­tion”.

Con­sider a car; say, a Toy­ota Corolla. The Corolla is made up of some num­ber of atoms; say, on the rough or­der of 1029. If you con­sider all pos­si­ble ways to ar­range 1029 atoms, only an in­finites­i­mally tiny frac­tion of pos­si­ble con­figu­ra­tions would qual­ify as a car; if you picked one ran­dom con­figu­ra­tion per Planck in­ter­val, many ages of the uni­verse would pass be­fore you hit on a wheeled wagon, let alone an in­ter­nal com­bus­tion en­g­ine.

Even re­strict­ing our at­ten­tion to run­ning ve­hi­cles, there is an as­tro­nom­i­cally huge de­sign space of pos­si­ble ve­hi­cles that could be com­posed of the same atoms as the Corolla, and most of them, from the per­spec­tive of a hu­man user, won’t work quite as well. We could take the parts in the Corolla’s air con­di­tioner, and mix them up in thou­sands of pos­si­ble con­figu­ra­tions; nearly all these con­figu­ra­tions would re­sult in a ve­hi­cle lower in our prefer­ence or­der­ing, still rec­og­niz­able as a car but lack­ing a work­ing air con­di­tioner.

So there are many more con­figu­ra­tions cor­re­spond­ing to non­ve­hi­cles, or ve­hi­cles lower in our prefer­ence rank­ing, than ve­hi­cles ranked greater than or equal to the Corolla.

Similarly with the prob­lem of plan­ning, which also in­volves hit­ting tiny tar­gets in a huge search space. Con­sider the num­ber of pos­si­ble le­gal chess moves ver­sus the num­ber of win­ning moves.

Which sug­gests one the­o­ret­i­cal way to mea­sure op­ti­miza­tion—to quan­tify the power of a mind or mindlike pro­cess:

Put a mea­sure on the state space—if it’s dis­crete, you can just count. Then col­lect all the states which are equal to or greater than the ob­served out­come, in that op­ti­miza­tion pro­cess’s im­plicit or ex­plicit prefer­ence or­der­ing. Sum or in­te­grate over the to­tal size of all such states. Divide by the to­tal vol­ume of the state space. This gives you the power of the op­ti­miza­tion pro­cess mea­sured in terms of the im­prob­a­bil­ities that it can pro­duce—that is, im­prob­a­bil­ity of a ran­dom se­lec­tion pro­duc­ing an equally good re­sult, rel­a­tive to a mea­sure and a prefer­ence or­der­ing.

If you pre­fer, you can take the re­cip­ro­cal of this im­prob­a­bil­ity (1/​1000 be­comes 1000) and then take the log­a­r­ithm base 2. This gives you the power of the op­ti­miza­tion pro­cess in bits. An op­ti­mizer that ex­erts 20 bits of power can hit a tar­get that’s one in a mil­lion.

When I think you’re a pow­er­ful in­tel­li­gence, and I think I know some­thing about your prefer­ences, then I’ll pre­dict that you’ll steer re­al­ity into re­gions that are higher in your prefer­ence or­der­ing. The more in­tel­li­gent I be­lieve you are, the more prob­a­bil­ity I’ll con­cen­trate into out­comes that I be­lieve are higher in your prefer­ence or­der­ing.

There’s a num­ber of sub­tleties here, some less ob­vi­ous than oth­ers. I’ll re­turn to this whole topic in a later se­quence. Mean­while:

* A tiny frac­tion of the de­sign space does de­scribe ve­hi­cles that we would rec­og­nize as faster, more fuel-effi­cient, safer than the Corolla, so the Corolla is not op­ti­mal. The Corolla is, how­ever, op­ti­mized, be­cause the hu­man de­signer had to hit an in­finites­i­mal tar­get in de­sign space just to cre­ate a work­ing car, let alone a car of Corolla-equiv­a­lent qual­ity. This is not to be taken as praise of the Corolla, as such; you could say the same of the Hill­man Minx. You can’t build so much as a wooden wagon by saw­ing boards into ran­dom shapes and nailing them to­gether ac­cord­ing to coin­flips.

* When I talk to a pop­u­lar au­di­ence on this topic, some­one usu­ally says: “But isn’t this what the cre­ation­ists ar­gue? That if you took a bunch of atoms and put them in a box and shook them up, it would be as­ton­ish­ingly im­prob­a­ble for a fully func­tion­ing rab­bit to fall out?” But the log­i­cal flaw in the cre­ation­ists’ ar­gu­ment is not that ran­domly re­con­figur­ing molecules would by pure chance as­sem­ble a rab­bit. The log­i­cal flaw is that there is a pro­cess, nat­u­ral se­lec­tion, which, through the non-chance re­ten­tion of chance mu­ta­tions, se­lec­tively ac­cu­mu­lates com­plex­ity, un­til a few billion years later it pro­duces a rab­bit.

* I once heard a se­nior main­stream AI type sug­gest that we might try to quan­tify the in­tel­li­gence of an AI sys­tem in terms of its RAM, pro­cess­ing power, and sen­sory in­put band­width. This at once re­minded me of a quote from Dijk­stra: “If we wish to count lines of code, we should not re­gard them as ‘lines pro­duced’ but as ‘lines spent’: the cur­rent con­ven­tional wis­dom is so fool­ish as to book that count on the wrong side of the ledger.” If you want to mea­sure the in­tel­li­gence of a sys­tem, I would sug­gest mea­sur­ing its op­ti­miza­tion power as be­fore, but then di­vid­ing by the re­sources used. Or you might mea­sure the de­gree of prior cog­ni­tive op­ti­miza­tion re­quired to achieve the same re­sult us­ing equal or fewer re­sources. In­tel­li­gence, in other words, is effi­cient op­ti­miza­tion. This is why I say that evolu­tion is stupid by hu­man stan­dards, even though we can’t yet build a but­terfly: Hu­man en­g­ineers use vastly less time/​ma­te­rial re­sources than a global ecosys­tem of mil­lions of species pro­ceed­ing through biolog­i­cal evolu­tion, and so we’re catch­ing up fast.

* The no­tion of a “pow­er­ful op­ti­miza­tion pro­cess” is nec­es­sary and suffi­cient to a dis­cus­sion about an Ar­tifi­cial In­tel­li­gence that could harm or benefit hu­man­ity on a global scale. If you say that an AI is me­chan­i­cal and there­fore “not re­ally in­tel­li­gent”, and it out­puts an ac­tion se­quence that hacks into the In­ter­net, con­structs molec­u­lar nan­otech­nol­ogy and wipes the so­lar sys­tem clean of hu­man(e) in­tel­li­gence, you are still dead. Con­versely, an AI that only has a very weak abil­ity steer the fu­ture into re­gions high in its prefer­ence or­der­ing, will not be able to much benefit or much harm hu­man­ity.

* How do you know a mind’s prefer­ence or­der­ing? If this can’t be taken for granted, then you use some of your ev­i­dence to in­fer the mind’s prefer­ence or­der­ing, and then use the in­ferred prefer­ences to in­fer the mind’s power, then use those two be­liefs to testably pre­dict fu­ture out­comes. Or you can use the Min­i­mum Mes­sage Length for­mu­la­tion of Oc­cam’s Ra­zor: if you send me a mes­sage tel­ling me what a mind wants and how pow­er­ful it is, then this should en­able you to com­press your de­scrip­tion of fu­ture events and ob­ser­va­tions, so that the to­tal mes­sage is shorter. Other­wise there is no pre­dic­tive benefit to view­ing a sys­tem as an op­ti­miza­tion pro­cess.

* In gen­eral, it is use­ful to think of a pro­cess as “op­ti­miz­ing” when it is eas­ier to pre­dict by think­ing about its goals, than by try­ing to pre­dict its ex­act in­ter­nal state and ex­act ac­tions. If you’re play­ing chess against Deep Blue, you will find it much eas­ier to pre­dict that Deep Blue will win (that is, the fi­nal board po­si­tion will oc­cupy the class of states pre­vi­ously la­beled “wins for Deep Blue”) than to pre­dict the ex­act fi­nal board po­si­tion or Deep Blue’s ex­act se­quence of moves. Nor­mally, it is not pos­si­ble to pre­dict, say, the fi­nal state of a billiards table af­ter a shot, with­out ex­trap­o­lat­ing all the events along the way.

* Although the hu­man cog­ni­tive ar­chi­tec­ture uses the same la­bel “good” to re­flect judg­ments about ter­mi­nal val­ues and in­stru­men­tal val­ues, this doesn’t mean that all suffi­ciently pow­er­ful op­ti­miza­tion pro­cesses share the same prefer­ence or­der­ing. Some pos­si­ble minds will be steer­ing the fu­ture into re­gions that are not good.

* If you came across alien ma­chin­ery in space, then you might be able to in­fer the pres­ence of op­ti­miza­tion (and hence pre­sum­ably pow­er­ful op­ti­miza­tion pro­cesses stand­ing be­hind it as a cause) with­out in­fer­ring the aliens’ fi­nal goals, by way of notic­ing the fulfill­ment of con­ver­gent in­stru­men­tal val­ues. You can look at ca­bles through which large elec­tri­cal cur­rents are run­ning, and be as­ton­ished to re­al­ize that the ca­bles are flex­ible high-tem­per­a­ture high-am­per­age su­per­con­duc­tors; an amaz­ingly good solu­tion to the sub­prob­lem of trans­port­ing elec­tric­ity that is gen­er­ated in a cen­tral lo­ca­tion and used dis­tantly. You can as­sess this, even if you have no idea what the elec­tric­ity is be­ing used for.

* If you want to take prob­a­bil­is­tic out­comes into ac­count in judg­ing a mind’s wis­dom, then you have to know or in­fer a util­ity func­tion for the mind, not just a prefer­ence rank­ing for the op­ti­miza­tion pro­cess. Then you can ask how many pos­si­ble plans would have equal or greater ex­pected util­ity. This as­sumes that you have some prob­a­bil­ity dis­tri­bu­tion, which you be­lieve to be true; but if the other mind is smarter than you, it may have a bet­ter prob­a­bil­ity dis­tri­bu­tion, in which case you will un­der­es­ti­mate its op­ti­miza­tion power. The chief sign of this would be if the mind con­sis­tently achieves higher av­er­age util­ity than the av­er­age ex­pected util­ity you as­sign to its plans.

* When an op­ti­miza­tion pro­cess seems to have an in­con­sis­tent prefer­ence rank­ing—for ex­am­ple, it’s quite pos­si­ble in evolu­tion­ary biol­ogy for allele A to beat out allele B, which beats allele C, which beats allele A—then you can’t in­ter­pret the sys­tem as perform­ing op­ti­miza­tion as it churns through its cy­cles. In­tel­li­gence is effi­cient op­ti­miza­tion; churn­ing through prefer­ence cy­cles is stupid, un­less the in­terim states of churn­ing have high ter­mi­nal util­ity.

* For do­mains out­side the small and for­mal, it is not pos­si­ble to ex­actly mea­sure op­ti­miza­tion, just as it is not pos­si­ble to do ex­act Bayesian up­dates or to perfectly max­i­mize ex­pected util­ity. Nonethe­less, op­ti­miza­tion can be a use­ful con­cept, just like the con­cept of Bayesian prob­a­bil­ity or ex­pected util­ity—it de­scribes the ideal you’re try­ing to ap­prox­i­mate with other mea­sures.

• Without a nar­row enough prior on the space of pos­si­ble prefer­ences, we might ex­plain any be­hav­ior as the op­ti­miza­tion of some prefer­ences. It is in­ter­est­ing that this doesn’t seem much of a prob­lem in prac­tice.

• Robin, the cir­cum­stances un­der which a Bayesian will come to be­lieve that a sys­tem is op­ti­miz­ing, are the same cir­cum­stances un­der which a mes­sage-length min­i­mizer will send a mes­sage de­scribing the sys­tem’s “prefer­ences”: namely, when your be­liefs about its prefer­ences are ca­pa­ble of mak­ing ante facto pre­dic­tions—or at least be­ing more sur­prised by some out­comes than by oth­ers.

Most of the things we find it use­ful to de­scribe as op­ti­miz­ers, have prefer­ences that are sta­ble over a longer timescale than the re­peated ob­ser­va­tions we make of them. An in­duc­tive sus­pi­cion of such sta­bil­ity is enough of a prior to deal with aliens (or evolu­tion). Also, differ­ent things of the same class of­ten have similar prefer­ences, like mul­ti­ple hu­mans.

• Without a nar­row enough prior on the space of pos­si­ble prefer­ences,

But we’ve already de­vel­oped com­plex mod­els of what ‘in­tel­li­gent’ be­ings ‘should’ pre­fer—based on our own prefer­ences. Try to ex­plore be­yond that in­tu­itive model, and we be­come very un­com­fortable.

When spec­u­lat­ing about real in­tel­li­gences, alien or ar­tifi­cial, it be­comes very im­por­tant that we jus­tify the re­stric­tions we place on the set of po­ten­tial prefer­ences we con­sider they might have.

• Do you think it is pos­si­ble to cap­ture the ideal­ized util­ity func­tion of hu­mans merely from fac­tual knowl­edge about the world dy­nam­ics? Is it pos­si­ble to find the con­cept of “right” with­out spec­i­fy­ing even the min­i­mal in­struc­tions on how to find it in hu­mans, by merely mak­ing AI search for a sim­ple util­ity func­tion un­der­ly­ing the fac­tual ob­ser­va­tions (given that it won’t be re­motely sim­ple)? I started from this po­si­tion sev­eral months back, but grad­u­ally shifted to re­quiring at least some an­thro­po­mor­phic di­rec­tions ini­ti­at­ing the ex­trac­tion of “right”, es­tab­lish­ing a path by which rep­re­sen­ta­tion of “right” in hu­mans trans­lates into rep­re­sen­ta­tion of util­ity com­pu­ta­tion in AI. One of the stronger prob­lems for me is that it’s hard to dis­t­in­guish be­tween ob­serv­ing ter­mi­nal and in­stru­men­tal op­ti­miza­tion performed in the cur­rent cir­cum­stances, and you can’t start re­it­er­at­ing to­wards vo­li­tion with­out know­ing even ap­prox­i­mately right di­rec­tion of trans­for­ma­tions to the agent (or whole en­vi­ron­ment, if agent is ini­tially un­defined). It would be in­ter­est­ing if it’s un­nec­es­sary, al­though of course plenty of safe­guards are still in or­der, if there is an over­ar­ch­ing mechanisms to wash them out even­tu­ally.

• If you at­tempt to quan­tify the “power” of an op­ti­mi­sa­tion pro­cess—with­out any at­tempt to fac­tor in the num­ber of eval­u­a­tions re­quired, the time taken, or the re­sources used—the “best” al­gorithm is usu­ally an ex­haus­tive search.

• Eli: I think that your anal­y­sis here, and the longer anal­y­sis pre­sented in “knowa­bil­ity of FAI” misses a very im­por­tant point. The sin­gu­lar­ity is a fun­da­men­tally differ­ent pro­cess than play­ing chess or build­ing a sa­loon car. The im­por­tant dis­tinc­tion is that in build­ing a car, the car-maker’s on­tol­ogy is perfectly ca­pa­ble of rep­re­sentng all of the high-level prop­er­ties of the de­sired state, but the I sti­ga­tors of the sin­gu­lar­ity are, by defi­ni­tion lack­ing a suffi­ciently com­plex rep­re­sen­ta­tion sys­tem to rep­re­sent any of the im­por­tant prop­er­ties of the de­sired state: post sin­gu­lar­ity earth. You have had the in­sight re­quired to see this: you posted about ” dreams of XML in a uni­verse of quan­tum me­chan­ics” a cou­ple of posts back. I posted about this on my blog: “on­tolo­gies, ap­prox­i­ma­tions and fun­da­men­tal­ists” too.

It suffices to say that an op­ti­miza­tion pro­cess which takes place with re­spect to a fixed back­ground on­tol­ogy or set of states is fun­da­men­tally differ­ent to a pro­cess which I might call vari-op­ti­miza­tion, where op­ti­miza­tion and on­tol­ogy change hap­pen at the same time. The sin­gu­lar­ity (whether an AI sin­gu­lar­ity or non AI) will be of the lat­ter type.

• Roko, see “Op­ti­miza­tion and the Sin­gu­lar­ity” and “DNA Con­se­quen­tial­ism and Protein Re­in­force­ment” on the con­cept of cross-do­main op­ti­miza­tion. Yes, this is part of what dis­t­in­guishes hu­man gen­eral in­tel­li­gence and nat­u­ral se­lec­tion, from mod­ern-day nar­row-do­main Ar­tifi­cial In­tel­li­gences and non­pri­mate or­ganisms. How­ever, this doesn’t pre­sent any challenge to my crite­rion of op­ti­miza­tion, ex­cept for the pos­si­bil­ity of need­ing to re­frame your ter­mi­nal val­ues for a new on­tol­ogy (which I’ve also dis­cussed in my metaethics, un­der “Un­nat­u­ral Cat­e­gories”). In­stru­men­tally speak­ing, and as­sum­ing away the prob­lem of un­nat­u­ral cat­e­gories, you’re just find­ing new paths to the same goals, lead­ing through por­tions of re­al­ity that you may not have sus­pected ex­isted. Sort of like hu­mans dig­ging miles un­der the ground to ob­tain geother­mal en­ergy in or­der to light their homes—they’re steer­ing the fu­ture into the same re­gion as did the maker of can­dles, but wend­ing through parts of re­al­ity and laws of physics that an­cient can­dle-mak­ers didn’t sus­pect ex­isted.

• Eliezer, this par­tic­u­lar point you made is of con­cern to me: “* When an op­ti­miza­tion pro­cess seems to have an in­con­sis­tent prefer­ence rank­ing—for ex­am­ple, it’s quite pos­si­ble in evolu­tion­ary biol­ogy for allele A to beat out allele B, which beats allele C, which beats allele A—then you can’t in­ter­pret the sys­tem as perform­ing op­ti­miza­tion as it churns through its cy­cles. In­tel­li­gence is effi­cient op­ti­miza­tion; churn­ing through prefer­ence cy­cles is stupid, un­less the in­terim states of churn­ing have high ter­mi­nal util­ity.”

You see, it seems quite likely to me that hu­mans eval­u­ate util­ity in such a cir­cu­lar way un­der many cir­cum­stances, and there­fore aren’t perform­ing any op­ti­miza­tions. Ask mid­dle school girls to rank boyfriend pren­fer­ence and you find Billy beats Joey who beats Micky who beats Billy… Now, when you ask an AI to carry out an op­ti­miza­tion of hu­man util­ity based on ob­serv­ing how peo­ple op­ti­mize their own util­ity as ev­i­dence, what do you sup­pose will hap­pen? Cer­tainly hu­mans op­ti­mize some­things, some­times, but op­ti­miza­tions of some things are at odds with oth­ers. Think how some peo­ple want both se­cu­rity and ad­ven­ture. A man might have one (say se­cu­rity), be happy for a time, get bored, then move on to the other and re­peat the cy­cle. Is opimiza­tion a flux of the two states? Or the one that gives the most util­ity over the other? I sup­pose you could take an in­te­gral of util­ity over time and find which set of states = max util­ity over time. How are we go­ing to be­gin to define util­ity? “We like it! But it has to be real, no wire-head­ing.” Now throw in the com­pli­ca­tion of differ­ent peo­ple hav­ing util­ity func­tions at odds with each other. Not ev­ery­one can be king of the world, no mat­ter how much util­ity they will de­rive from this po­si­tion. Now ask the ma­chine to be effi­cient- do it as eas­ily as pos­si­ble, so that eas­ier solu­tions are fa­vored over more difficult “ex­pen­sive” ones.

Even if we avoid all the pit­falls of ‘mi­s­un­der­stand­ing’ the ini­tial com­mand to ‘op­ti­mize util­ity’, what gives you rea­son to as­sume you or I or any of the small, small sub­seg­ment of the pop­u­la­tion that reads this blog is go­ing to like what the vec­tor sum of all hu­man prefer­ences, util­ities, etc. coughs up?

• Lara Foster, I would be in­ter­ested to hear a re­ally solid ex­am­ple of non­tran­si­tive util­ity prefer­ences, if you can think of one.

• @Lara_Foster: You see, it seems quite likely to me that hu­mans eval­u­ate util­ity in such a cir­cu­lar way un­der many cir­cum­stances, and there­fore aren’t perform­ing any op­ti­miza­tions.

Eliezer touches on that is­sue in “Op­ti­miza­tion and the Sin­gu­lar­ity”:

Nat­u­ral se­lec­tion prefers more effi­cient repli­ca­tors. Hu­man in­tel­li­gences have more com­plex prefer­ences. Nei­ther evolu­tion nor hu­mans have con­sis­tent util­ity func­tions, so view­ing them as “op­ti­miza­tion pro­cesses” is un­der­stood to be an ap­prox­i­ma­tion.

By the way, Ask mid­dle school girls to rank boyfriend prefer­ence and you find Billy beats Joey who beats Micky who beats Billy...

Would you mind peek­ing into your mind and ex­plain­ing why that arises? :-) Is it just a spe­cial case of the phe­nomenon you de­scribed in the rest of your post?

• Those prone to envy bias can tend to think that “the grass is greener on the other side of the fence”. That idea can lead to cir­cu­lar prefer­ences. Such prefer­ences should not be very com­mon—evolu­tion should weed them out.

• Does it make sense to speak of a re­ally pow­er­ful op­ti­mi­sa­tion pro­cess un­der this defi­ni­tion. Con­sider the man build­ing the car, how good the car will be is de­pen­dent upon him, but also the gen­eral so­ciety around him. Have a per­son with the same ge­net­ics grow up in the early 1900s and to­day and they will be able to hit rad­i­cally differ­ent points in op­ti­mi­sa­tion space, not due to their own abil­ity but due to what has been dis­cov­ered and the in­for­ma­tion and econ­omy around them.

Another ex­am­ple, give me a work­shop, the in­ter­net and a junk­yard and I might be able to whip up a mo­tor ve­hi­cle, strand me on the desert is­land, the best I could come up with is prob­a­bly a lit­ter. Maybe a wooden bi­cy­cle.

Similarly for ideas, New­tons oft quoted com­ment about shoulder stand­ing, in­di­cates that rely­ing on other op­ti­mi­sa­tion pro­cesses is also very use­ful.

• I wouldn’t as­sume a pro­cess seem­ing to churn through prefer­ence cy­cles to have an in­con­sis­tent prefer­ence rank­ing, it could be effi­ciently op­ti­miz­ing if each state pro­vides diminish­ing re­turns. If ev­ery few hours a jailer offers ei­ther food, wa­ter or a good book, you don’t pick the same choice each time!

• Will, it’s for that rea­son, and also for the sake of de­scribing nat­u­ral se­lec­tion, that I speak of “op­ti­miza­tion pro­cesses” rather than just say­ing “op­ti­mizer”. For hu­mans to pro­duce a nu­clear power plant re­quires more than one hu­man and more than one crew of hu­mans; it takes an econ­omy and a his­tory of sci­ence, though a good deal of ex­tra effort is con­cen­trated at the end of the prob­lem.

• So can we ever stan­dard­ise mat­ters to say, for ex­am­ple, whether it re­quires ‘bet­ter’ or ‘more’ op­ti­mi­sa­tion pro­cesses to build a wagon in 1850 than a car in 1950?

• If you go around a cy­cle—and you are worse off than you would have been if you had stayed still—you may have cir­cu­lar prefer­ences. If it hap­pens re­peat­edly, a re­view is definitely in or­der.

• I am sus­pi­cious of at­tempts to define in­tel­li­gence for the fol­low­ing rea­son. Too of­ten, they lead the definer down a nar­row and ul­ti­mately fruitless path. If you define in­tel­li­gence as the abil­ity to perform some func­tion XYZ, then you can sit down and start try­ing to hack to­gether a sys­tem that does XYZ. Al­most in­vari­ably this will re­sult in a sys­tem that achieves some su­perfi­cial imi­ta­tion of XYZ and very lit­tle else.

Rather than at­tempt­ing to define in­tel­li­gence and move in a de­ter­mined path to­ward that goal, we should look around for novel in­sights and ex­plore their im­pli­ca­tions.

Imag­ine if New­ton had fol­lowed the ap­proach of “define physics and then move to­ward it”. He may have de­cided that physics is the abil­ity to build large struc­tures (cer­tainly an un­der­stand­ing of physics is helpful or re­quired for this). He might then have spent all his time in­ves­ti­gat­ing the ma­te­rial prop­er­ties of var­i­ous kinds of stone—use­ful per­haps, but misses the big pic­ture. In­stead he looked around in the most un­likely places to find some­thing in­ter­est­ing that had very lit­tle im­me­di­ate prac­ti­cal ap­pli­ca­tion. That should be our mind­set in pur­su­ing AI: the sci­en­tist’s, rather than the en­g­ineer’s, ap­proach.

• A ques­tion, would you con­sider com­put­ers as part of the dom­i­nant op­ti­mi­sa­tion pro­cesses on earth already?

It is just that you of­ten com­pare neu­rons to sili­con as one of the things that will be very differ­ent for AI. But as we already use sili­con as part of our op­ti­mi­sa­tion pro­cesses (mod­el­ling pro­tein fold­ing, weather sys­tems, data min­ing). The jump to pure sili­con op­ti­mi­sa­tion pro­cesses might not be as huge as you sug­gest with the com­par­i­son of firing rates of neu­rons and pro­ces­sor speed.

I sup­pose I am hav­ing trou­ble lo­cal­is­ing an op­ti­mi­sa­tion pro­cess and rat­ing its “power”. Con­sider two iden­ti­cal com­puter sys­tems with soft­ware that if on the sur­face would en­able it to be a RPOP, both of them locked un­der­ground on the moon one has suffi­cient en­ergy (chem­i­cal and heat gra­di­ents) to boot­strap to fu­sion or oth­er­wise get it­self out of the moon, the other one doesn’t. They ob­vi­ously will have a differ­ent abil­ity to af­fect the fu­ture. Should I say that the en­ergy re­serves are part of the op­ti­mi­sa­tion pro­cess rather than sep­a­rat­ing them away, so I can still say they are equally pow­er­ful? How much of the uni­verse do you need to con­sider part of the pro­cess so that pow­er­ful­ness(pro­cess) gives a unique an­swer?

One last ques­tion, if you cre­ate friendly AI, should I con­sider you as pow­er­ful an op­ti­mi­sa­tion pro­cess as it?

• I don’t think char­ac­ter­is­ing the power of an op­ti­miser by us­ing the size of the tar­get re­gion rel­a­tive to the size of the to­tal space is enough. A tiny tar­get in a gi­gan­tic space is triv­ial to find if the space has a very sim­ple struc­ture with re­spect to your prefer­ences. For ex­am­ple, a large smooth space with a gra­di­ent that points to­wards the op­ti­mum. Con­versely, a big­ger tar­get on a smaller space can be prac­ti­cally im­pos­si­ble to find if there is lit­tle struc­ture, or if the struc­ture is de­cep­tive.

• @Eliezer: I think your defi­ni­tion of “op­ti­miza­tion pro­cess” is a very good one, I just don’t think that the tech­nolog­i­cal sin­gu­lar­ity [will nec­ces­sar­ily be]/​[ought to be] an in­stance of it.

Eliezer: ” … you’re just find­ing new paths to the same goals, lead­ing through por­tions of re­al­ity that you may not have sus­pected ex­isted.”

• This may be a point that we dis­agree on quite fun­da­men­tally. Has it oc­curred to you that one might in­tro­duce ter­mi­nal val­ues in a new, richer on­tol­ogy which were not even pos­si­ble to state in the old one? Surely: you’re aware that most of the things that an adult hu­man con­sid­ers to be of ter­mi­nal value are not state­able in the on­tol­ogy of a 3 year old (“Loy­alty to Liber­tar­i­anism”, “Math­e­mat­i­cal Ele­gance”, “Fine Liter­a­ture”, “Sex­u­al­ity”, …); that most things a hu­man con­sid­ers to be of ter­mi­nal value are not state­able in the on­tol­ogy of a chimp.

I think that it is the pos­si­bil­ity of find­ing new value-states which were sim­ply uni­mag­in­able to an ear­lier ver­sion of one­self that at­tracts me to the tran­shu­man­ist cause; if you cast the sin­gu­lar­ity as an op­ti­miza­tion pro­cess you rule out this pos­si­bil­ity from the start. An “op­ti­miza­tion pro­cess” based ver­sion of the sin­guilar­ity will land us in some­thing like Iain M Banks’ Cul­ture, where hu­man drives and de­sires are su­per­sat­u­rated by ad­vanced tech­nol­ogy, but noth­ing re­ally new is done.

Fur­ther­more, my de­sire to ex­pe­rience value-states that are sim­ply uni­mag­in­able to the cur­rent ver­sion of me is sim­ply not state­able as an op­ti­miza­tion prob­lem: for op­ti­miza­tion always takes place over a known set of states (as you ex­plained well in the OP).

• Shane, that seems dis­tantly like try­ing to com­pare two po­si­tions in differ­ent co­or­di­nate sys­tems, with­out trans­form­ing one into the other first. Surely there is a trans­form that would con­vert the “hard” space into the terms of the “easy” space, so that the size of the tar­gets could be com­pared ap­ples to ap­ples.

• Will P: A ques­tion, would you con­sider com­put­ers as part of the dom­i­nant op­ti­mi­sa­tion pro­cesses on earth already?

Will, to the ex­tent that you can draw a cat­e­gory around dis­crete op­ti­mi­sa­tion pro­cesses, I’d say a qual­ified ‘yes’. Com­put­ers, in­so­far as they are built and used to ap­proach hu­man ter­mi­nal val­ues, are a part of the hu­man op­ti­mi­sa­tion pro­cess. Hu­man­ity is far more effi­cient in its pro­cesses than a hun­dred years back. The vast ma­jor­ity of this has some­thing to do with sili­con.

In re­sponse to your com­put­ers-on-the-moon (great coun­ter­fac­tual, by the way), I think I’d end up judg­ing op­ti­mi­sa­tion by its re­sults. That said, I sup­pose how you mea­sure effi­ciency de­pends on what that query is dis­guis­ing. In­tel­li­gence/​g? Re­pro­duc­tive speed? TIT-FOR-TAT-ness?

I read re­cently that the ne­an­derthals, de­spite hav­ing the larger brain cav­i­ties, may well have gone un­der be­cause our own an­ces­tors sim­ply bred faster. Who had the bet­ter op­ti­mi­sa­tion pro­cess there?

The ques­tion of whether ‘com­puter-op­ti­mi­sa­tion’ will ever be a sep­a­rate pro­cess from ‘hu­man-op­ti­mi­sa­tion’ de­pends largely on your point of view. It seems as though a hu­man-built com­puter should never spon­ta­neously dream up its own ter­mi­nal value. How­ever, feel free to dis­agree with me when your DNA is be­ing bro­ken down to cre­ate molec­u­lar smiley faces.

• Andy:

Sure, you can trans­form a prob­lem in a hard co­or­di­nate space into an easy one. For ex­am­ple, sim­ply or­der the points in terms of their de­sir­a­bil­ity. That makes find­ing the op­ti­mum triv­ial: just point at the first el­e­ment! The prob­lem is that once you have trans­formed the hard prob­lem into an easy one, you’ve es­sen­tially already solved the op­ti­mi­sa­tion prob­lem and thus it no longer tests the power of the op­ti­miser.

• Surely there is a trans­form that would con­vert the “hard” space into the terms of the “easy” space, so that the size of the tar­gets could be com­pared ap­ples to ap­ples.

But isn’t this the same as com­put­ing a differ­ent mea­sure (i.e. not the count­ing mea­sure) on the “hard” space? If so, you could nor­mal­ize this to a prob­a­bil­ity mea­sure, and then com­pute its Kul­lback-Leibler di­ver­gence to ob­tain a mea­sure of in­for­ma­tion gain.

• That should be our mind­set in pur­su­ing AI: the sci­en­tist’s, rather than the en­g­ineer’s, ap­proach.
Scien­tists do not pre­define the re­sults they wish to find. Pre­defin­ing is what Eliezer’s Friendly AI is all about.

• Shane, I was ba­si­cally agree­ing with you with re­gard to prob­lem spaces: nor­mal­iz­ing space size isn’t enough, you’ve also got to nor­mal­ize what­ever else makes them in­com­pa­rable. How­ever, let’s not con­fuse prob­lem space with state space. Eliezer fo­cuses on the lat­ter, which I think is pretty triv­ial com­pared to what you’re al­lud­ing to.

• It strikes me as odd to define in­tel­li­gence in terms of abil­ity to shape the world; among other things, this im­plies that if you am­pu­tate a man’s limbs, he im­me­di­ately be­comes much less in­tel­li­gent.

• Nomin­ull, I dis­t­in­guished be­tween op­ti­miza­tion power and in­tel­li­gence. Op­ti­miza­tion power is abil­ity to shape the world. In­tel­li­gence is effi­ciently us­ing time/​CPU/​RAM/​sen­sory/​mo­tor re­sources to op­ti­mize. Cut off a per­son’s limbs and they be­come less pow­er­ful; but if they can ma­nipu­late the world to their bid­ding with­out their limbs, why, they must be very in­tel­li­gent.

• John Maxwell- I thought the se­cu­rity/​ad­ven­trure ex­am­ple was good, but that the way I por­trayed it might make it seem that ever-al­ter­nat­ing IS the an­swer. Here­goes: A man lives as a bo­hemian out on the street, no­mad­i­cally day to day solv­ing his prob­lems of how to get food and shelter. It seems to him that he would be bet­ter off look­ing for a se­cure life, and thus gets a job to make money. Work­ing for money for a se­cure life is difficult and tiring and it seems to him that he will be bet­ter off once he has the money and is se­cure. Now he’s worked a long time and has the money and is se­cure, which he now finds is bor­ing both in com­par­i­son to work­ing and liv­ing a bo­hemian life with un­cer­tainty in it. Peo­ple do value un­cer­tainty and ‘au­then­tic­ity’ to a very high de­gree. Thus Be­ing Se­cure is > Work­ing to be se­cure > Not be­ing se­cure > be­ing se­cure.

Now, Eliezer would ap­pro­pri­ately point out that the man only got trapped in this loop, be­cause he didn’t ac­tu­ally know what would make him hap­piest, but as­sumed with­out hav­ing the ex­pe­rience. But, that be­ing said, do we think this fel­low would have been satis­fied be­ing told to start with ‘Don’t bother work­ing son, this is bet­ter for you, trust me!’ There’s no ob­vi­ous rea­son to me why the fAI will al­low peo­ple the au­ton­omy they so de­sire to pur­sue their own mis­takes un­less the fi­nal calcu­la­tion of hu­man util­ity de­ter­mines that it wins out, and this is du­bi­ous… I’m say­ing that I don’t care if what in truth max­i­mizes util­ity is for ev­ery­one to be­lieve they’re 19th cen­tury god-fear­ing farm­ers, or to be on a cir­cu­lar magic quest the mem­ory of the ear­liest day of which dis­ap­pears each day, such that it re­plays for eter­nity, or what­ever simu­la­tion the fAI de­cides on for post-sin­gu­lar­ity hu­man­ity, I think I’d rather be free of it to fuck up my own life. Me and many oth­ers.

I guess this goes to an­other more im­por­tant prob­lem than hu­man non­lin­ear prefer­ence- Why should we trust an AI that max­i­mizes hu­man util­ity, even if it un­der­stands what that means? Why should we, from where we sit now, like what hu­man vo­li­tion (a col­lec­tion of non-lin­ear prefer­ences) ex­trap­o­lates to, and what value do we place on our own au­ton­omy?

• Op­ti­miza­tion power re­ally ought to have some­thing to do with the abil­ity to solve op­ti­miza­tion prob­lems. The pos­si­ble sub­se­quent im­pact of those solu­tions on the world seems like a side is­sue.

• Thus Be­ing Se­cure is > Work­ing to be se­cure > Not be­ing se­cure > be­ing se­cure.

As judged at differ­ent times, un­der differ­ent cir­cum­stances (hav­ing less or more money, be­ing less or more burned out). This doesn’t sound like a “real” in­tran­si­tive prefer­ence.

what­ever simu­la­tion the fAI de­cides on for post-sin­gu­lar­ity hu­man­ity, I think I’d rather be free of it to fuck up my own life. Me and many oth­ers.… Why should we trust an AI that max­i­mizes hu­man util­ity, even if it un­der­stands what that means?

But then, your free­dom is a fac­tor in de­cid­ing what’s best for you. It sounds like you’re think­ing of an FAI as a well-in­ten­tioned but ex­tremely ar­ro­gant hu­man, who can’t re­sist the temp­ta­tion to med­dle where it ra­tio­nally shouldn’t.

• It’s not about re­sist­ing temp­ta­tion to med­dle, but about what will, in fact, max­i­mize hu­man util­ity. The AI will not care whether util­ity is max­i­mized by us or by it, as long as it is max­i­mized (un­less you want to pro­gram in ‘au­ton­omy’ as an ax­iom, but I’m sure there are other prob­lems with that). I think there is a high prob­a­bil­ity that, given its power, the fAI will de­ter­mine that it can best max­i­mize hu­man util­ity by tak­ing away hu­man au­ton­omy. It might give hu­mans the illu­sion of au­ton­omy in some cir­cum­stances, and low and be­hold these peo­ple will be ‘hap­pier’ than non-delu­sional peo­ple would be. Heck, what’s to keep it from putting ev­ery­one in their own in­di­vi­d­ual simu­la­tion? I was as­sum­ing some ax­iom that stated, ‘no wire-head­ing’, but it’s very hard for me to even know what that means in a post-sin­gu­lar­ity con­text. I’m very skep­ti­cal of hand­ing over con­trol of my life to any dic­ta­to­rial source of power, no mat­ter how ‘friendly’ it’s pro­grammed to be. Now, if Eliezer is con­viced it’s a choice be­tween his cre­ation as dic­ta­tor vs some­one else’s de­stroy­ing the uni­verse, then it is un­der­stand­able why he is work­ing to­wards the best dic­ta­tor he can sur­mise… But I would rather not have ei­ther.

• Oh, come on, Lara, did you re­ally think I hadn’t thought of that? One of the rea­sons why Friendly AI isn’t triv­ial is that you need to de­scribe hu­man val­ues like au­ton­omy—“I want to op­ti­mize my own life, not have you do it for me”—whose de­ci­sion-struc­ture is non­triv­ial, e.g., you wouldn’t want an AI choos­ing the ex­act life-course for you that max­i­mized your au­ton­omy.

• What I always won­der is why we need to pre­serve hu­man val­ues like au­ton­omy if we could pro­duce bet­ter re­sults with­out it? For ex­am­ple, if an AI could com­pute the ab­solute best way to perform a spe­cific ac­tion then why is it a good thing to be able to choose a differ­ent way to perform the ac­tion?

• ‘Bet­ter’ is ex­actly why we’d want to re­tain (some) au­ton­omy! I per­son­ally wouldn’t want an AI to tell me ex­actly how to live my life.

• Oh, come on, Eliezer, of course you thought of it. ;) How­ever, it might not have been some­thing that both­ered you, as in- A) You didn’t be­lieve ac­tu­ally hav­ing au­ton­omy mat­tered as long as peo­ple feel like they do (ie a Ma­trix/​Nexus situ­a­tion). I have heard this ar­gued. Would it mat­ter to you if you found out your whole life was a simu­la­tion? Some say no. I say yes. Mat­ter of taste per­haps?

B) OR You find it self ev­i­dent that ‘real’ au­ton­omy would be ex­trap­o­lated by the AI as some­thing es­sen­tial to hu­man hap­piness, such that an in­tel­li­gence ob­serv­ing peo­ple and max­i­miz­ing our util­ity wouldn’t need to be told ‘al­low au­ton­omy.’ This I would dis­agree with.

C) OR You rec­og­nize that this is a prob­lem with a non-ob­vi­ous solu­tion to an AI, and thus in­tend to deal with it some­how in code ahead of time, be­fore start­ing the vo­li­tion ex­trap­o­lat­ing AI. Your re­sponse in­di­cates you feel this way. How­ever, I am con­cerned even be­yond set­ting an ax­io­matic func­tion for ‘al­low au­ton­omy’ in a pro­gram. There are prob­a­bly an in­finite num­ber of ways that an AI can find ways to carry out its stated func­tion that will some­how ‘game’ our own sys­tem and lead to sub­op­ti­mal or out­right re­pug­nant re­sults (ie ev­ery­one be­ing trapped in a per­ma­nent quest- maybe the AI avoids the prob­lem of ‘it has to be real’ by ac­tu­ally cre­at­ing a magic ring that needs to be thrown into a vol­cano ev­ery 6 years or so). You don’t need me tel­ling you that! Max­i­miz­ing util­ity while de­lud­ing us about re­al­ity is only one. It seems im­pos­si­ble that we could ax­io­mat­i­cally safe­guard against all pos­si­bil­ities. As­si­mov was a pretty smart cookie, and his ‘3 laws’ are cer­tainly not suffi­cient. ‘Eliezer’s mil­lion lines of code’ might cover a much larger range of AI failures, but how could you ever be sure? The whole pro­ject just seems in­sanely dan­ger­ous. Or are you go­ing to ad­dress safety con­cerns in an­other post in this se­ries?

• What Mor­pheus and his crew gave Neo in the Ma­trix movie is not more au­ton­omy, IMHO, but rather a much more com­plete model of re­al­ity.

• Shane, this is an ad­vanced topic; it would be cov­ered un­der the topic of try­ing to com­pute the de­gree of op­ti­miza­tion of the op­ti­mizer, and the topic of choos­ing a mea­sure on the state space.

First, if you look at parts of the prob­lem in a par­tic­u­lar or­der ac­cord­ing to your search pro­cess, that’s some­what like hav­ing a mea­sure that gives large chunks of mass to the first op­tions you search. If you were look­ing for your keys, then, all else be­ing equal, you would search first in the places where you thought you were most likely to find your keys (or the eas­iest places to check, prob­a­bil­ity di­vided by cost, but for­get that for the mo­ment) - so there’s some­thing like a mea­sure, in this case a prob­a­bil­ity mea­sure, that cor­re­sponds to where you look first. Think of turn­ing it the other way around, and say­ing that the points of largest mea­sure cor­re­spond to the first places you search, whether be­cause the solu­tion is most likely to be there, or be­cause the cost is low­est. Th­ese are the solu­tions we call “ob­vi­ous” or “straight­for­ward” as if they had high prob­a­bil­ity.

Se­cond, sup­pose you were smarter than you are now and a bet­ter pro­gram­mer, tran­shu­manly so. Then for you, cre­at­ing a chess pro­gram like Deep Blue (or one of the mod­ern more effi­cient pro­grams) might be as easy as com­put­ing the Fibonacci se­quence. But the chess pro­gram would still be just as pow­er­ful as Deep Blue. It would be just as pow­er­ful an op­ti­mizer. Only to you, it would seem like an “ob­vi­ous solu­tion” so you wouldn’t give it much credit, any more than you credit gra­di­ent de­scent on a prob­lem with a global min­i­mum—though that might seem much harder to Archimedes than to you; the New­ton-Raph­son method was a brilli­ant in­no­va­tion, once upon a time.

If you see a way to solve an op­ti­miza­tion prob­lem us­ing a very sim­ple pro­gram, then it will seem to you like the difficulty of the prob­lem is only the difficulty of writ­ing that pro­gram. But it may be wiser to draw a dis­tinc­tion be­tween the ob­ject level and the meta level. Kas­parov ex­erted con­tin­u­ous power to win mul­ti­ple chess games. The pro­gram­mers of Deep Blue ex­erted a con­stant amount of effort to build it, and then they could win as many chess games as they liked by press­ing a but­ton. It is a mis­take to com­pare the effort ex­erted by Kas­parov to the effort ex­erted by the pro­gram­mers; you should com­pare Kas­parov to Deep Blue, and say that Deep Blue was a more pow­er­ful op­ti­mizer than Kas­parov. The pro­gram­mers you would only com­pare to nat­u­ral se­lec­tion, say, and maybe you should in­clude in that the econ­omy be­hind them that built the com­put­ing hard­ware.

But this just goes to show that what we con­sider difficulty isn’t always the same as ob­ject-level op­ti­miza­tion power. Once the pro­gram­mers built Deep Blue, it would have been just a press of the but­ton for them to turn Deep Blue on, but when Deep Blue was run­ning, it would still have been ex­ert­ing op­ti­miza­tion power. And you don’t find it difficult to reg­u­late your body’s breath­ing and heart­beat and other prop­er­ties, but you’ve got a whole medulla and any num­ber of gene reg­u­la­tory net­works con­tribut­ing to a con­tin­u­ous op­ti­miza­tion of your body. So what we per­ceive as difficulty is not the same as op­ti­miza­tion-power-in-the-world—that’s more a func­tion of what hu­mans con­sider ob­vi­ous or effortless, ver­sus what they have to think about and ex­am­ine mul­ti­ple op­tions in or­der to do.

We could also de­scribe the op­ti­mizer in less con­crete and more prob­a­bil­is­tic terms, so that if the en­vi­ron­ment is not cer­tain, the op­ti­mizer has to ob­tain its end un­der mul­ti­ple con­di­tions. In­deed, if this is not the case, then we might as well model the sys­tem by think­ing in terms of a sin­gle lin­ear chain of cause and effect, which would not ar­rive at the same des­ti­na­tion if per­turbed any­where along its way—so then there is no point in de­scribing the sys­tem as hav­ing a goal.

We could say that op­ti­miza­tion isn’t re­ally in­ter­est­ing un­til it has to cross mu­ti­ple do­mains or un­known do­mains, the way we con­sider hu­man in­tel­li­gence and nat­u­ral se­lec­tion as more in­ter­est­ing op­ti­miza­tions than beavers build­ing a dam. Th­ese may also be rea­sons why you feel that sim­ple prob­lems don’t re­flect much difficulty, or that the kind of op­ti­miza­tion performed isn’t com­men­su­rate with the work your in­tel­li­gence per­ceives as “work’.

Even so, I would main­tain the view of an op­ti­miza­tion pro­cess as some­thing that squeezes the fu­ture into a par­tic­u­lar re­gion, across a range of start­ing con­di­tions, so that it’s sim­pler to un­der­stand the des­ti­na­tion than the path­way. Even if the pro­gram that does this seems re­ally straight­for­ward to a hu­man AI re­searcher, the pro­gram it­self is still squeez­ing the fu­ture—it’s work­ing even if you aren’t. Or maybe you want to sub­sti­tute a differ­ent mea­sure on the state space than the equiprob­a­ble one—but at that point you’re bring­ing your own in­tel­li­gence into the prob­lem. There’s a lot of prob­lems that look sim­ple to hu­mans, but it isn’t always easy to make an AI solve them.

• Be­ing Se­cure is > Work­ing to be se­cure > Not be­ing se­cure > be­ing secure

It is not much of a loop un­less you re­peat it. If the re-Bo­hemian be­comes a re-office worker, we have a loop. Other­wise, we have an ex­per­i­ment that did not work. The ac­tual prefer­ences de­scribed sounded more like: (Inac­cu­rate) ex­pected value of be­ing se­cure > Not be­ing se­cure > (Ac­tual value of) be­ing se­cure > Work­ing to be se­cure where the Bo­hemian was will­ing to en­dure the low­est point to reach the high­est point, only to dis­cover his in­cor­rect ex­pec­ta­tions. I ex­pect, how­ever, that the re-Bo­hemian pe­riod shortly af­ter liqui­dat­ing ev­ery­thing from the se­cu­rity pe­riod will be a lot of fun.

• Eli, most of what you say above isn’t new to me—I’ve already en­coun­tered these things in my work on defin­ing ma­chine in­tel­li­gence. More­over, none of this has much im­pact on the fact that mea­sur­ing the power of an op­ti­miser sim­ply in terms of the rel­a­tive size of a tar­get sub­space to the search space doesn’t work: some­times tiny tar­gets in mas­sive spaces are triv­ial to solve, and some­times big­ger tar­gets in mod­er­ate spaces are prac­ti­cally im­pos­si­ble. The sim­ple num­ber-of-bits-of-op­ti­mi­sa­tion-power method you de­scribe in this post doesn’t take this into ac­count. As far as I can see, the only way you could deny this is if you were a strong NFL the­o­rem be­liever.

• More­over, none of this has much im­pact on the fact that mea­sur­ing the power of an op­ti­miser sim­ply in terms of the rel­a­tive size of a tar­get sub­space to the search space doesn’t work: some­times tiny tar­gets in mas­sive spaces are triv­ial to solve, and some­times big­ger tar­gets in mod­er­ate spaces are prac­ti­cally im­pos­si­ble.

I thought I de­scribed my at­ti­tude to­ward this above: The con­cept of a prob­lem’s difficulty, the amount of men­tal effort it feels like you need to ex­ert to solve it, should not be con­fused with the op­ti­miza­tion power ex­erted in solv­ing it, which should not be con­fused with the in­tel­li­gence used in solv­ing it. What is “triv­ial to solve” de­pends on how in­tel­li­gent you start out; “how much you can ac­com­plish” de­pends on the re­sources you start with; and whether I bother to de­scribe some­thing as an “op­ti­miza­tion pro­cess” at all will de­pend on whether it achieves the “same goal” across mul­ti­ple oc­ca­sions, con­di­tions, and start­ing points.

• Eliezer—Con­sider max­i­miz­ing y in the search space y = - vec­tor_length(x). You can make this space as large as you like, by in­creas­ing the range or the di­men­sion­al­ity of x. But it does not get any more difficult, whether you mea­sure by difficulty, power needed, or in­tel­li­gence needed.

• Re: op­ti­miza­tion power

The prob­lem I see is that—if you are mak­ing up ter­minol­ogy—it would be nice if the name re­flected what was be­ing mea­sured.

Op­ti­mi­sa­tion power sug­gests some­thing use­ful—but the pro­posed met­ric con­tains no refer­ence to the num­ber of tri­als, the num­ber of tri­als in se­ries on the crit­i­cal path—or most of the other com­mon ways of mea­sur­ing the worth of op­ti­mi­sa­tion pro­cesses. It seems to be more a func­tion of the size of the prob­lem space than any­thing else—in which case, why “power” and not, say “fac­tor”.

Be­fore christen­ing the no­tion, there are some ba­sic ques­tions: Is the pro­posed met­ric any use? What is the point of it?

• Eli, you pro­pose this num­ber of bits met­ric as a way “to quan­tify the power of a mind”. Surely then, some­thing with a very high value in your met­ric should be a “pow­er­ful mind”?

It’s easy to come up with a wide range of op­ti­mi­sa­tion prob­lems, as Phil Goetz did above, where a very sim­ple al­gorithm on very mod­est hard­ware would achieve mas­sive scores with re­spect to your mind power met­ric. And yet, this is clearly not a “pow­er­ful mind” in any rea­son­able sense.

• Re: I would rather not have ei­ther.

Choose no su­per­in­tel­li­gence, then—but don’t ex­pect re­al­ity to pay your wishes much at­ten­tion.