While I don’t like MacOS very much, it seems to me strange to ar­gue that the av­er­age value of it to peo­ple is zero.

I ap­pre­ci­ate the more con­crete defi­ni­tion of IC pre­sented here. How­ever, I have an in­ter­pre­ta­tion that is a bit differ­ent from you. I’m fol­low­ing the for­mal pre­sen­ta­tion.

My base un­der­stand­ing is that a cy­cle with max av­er­age re­ward is op­ti­mal. This is es­sen­tially just a defi­ni­tion. In the case the agent doesn’t know the re­ward func­tion, it seems clear that the agent ought to po­si­tion it’s self in a state which gives it ac­cess to as many of these cy­cles as pos­si­ble.

In your pa­per, the­o­rem 19 sug­gests that given a choice be­tween two sets of 1-cy­cles $C_1$ and $C_2$ the agent is more likely to se­lect the larger set. This makes sense. What doesn’t make sense is the con­clu­sion (the­o­rem 17) that the agent se­lects states with more power. This is be­cause at the very start of the pa­per it’s men­tioned that,

As an al­ter­na­tive mo­ti­va­tion, con­sider an agent in a com­mu­ni­cat­ing MDP which is pe­ri­od­i­cally as­signed a task from a known dis­tri­bu­tion $R$. Between tasks, the agent has as much down­time as re­quired. To max­i­mize re­turn over time, the op­ti­mal policy dur­ing down­time is to nav­i­gate to the state with max­i­mal $V_{avg}^{\ast }(s,\gamma )$.

Ac­cord­ing to the­o­rem 17, loos­ing ac­cess to states means that power goes down (or stays con­stant). This seems to in­di­cate power (cy­cle ac­cess) is re­ally some sort of Lya­punov func­tion for the dy­nam­ics. So at the out­set, it seems clear that the agent will pre­fer states that max­i­mize power, but then as soon as a de­ter­mi­na­tion is made on what the ac­tual re­ward func­tion is, power goes down, not up.

What I’m try­ing to point out here is that I find the dis­tinc­tion be­tween pre-task op­ti­miza­tion and ex­e­cu­tion to be loose. This is to such a de­gree that I find my­self draw­ing the ex­act op­po­site con­clu­sion: agents op­ti­miz­ing a generic re­ward will tend to give-up power.

At the mo­ment, I find my­self agree­ing with the idea that an agent un­aware of it’s task will seek power, but also con­clude that an agent aware of it’s task will give-up power. My cur­rent opinion is that power seek­ing be­hav­ior is con­cen­trated in the pre-task step. Giv­ing the AI un­re­stricted ‘free-time’ to op­ti­mize with should ‘never’ be al­lowed. Now, I could be mi­s­un­der­stand­ing parts of the pa­per, but hope­fully I’ve made things clear enough!

I didn’t copy it but it was fairly rea­son­able plain­text, some­thing like user­name /​n date /​n com­ment /​n /​n next com­ment.

A far off de­ci­sion maker can’t have di­rect ev­i­dence of your ex­is­tence as then you would be the cause of their ex­is­tence.

A far off ob­server can see a pro­cess that it can pre­dict will re­sult into you and things that it does may be co­causes with the fu­ture be­tween you. I still think that the verb “af­fect” is wrong here.

Say there is a preg­nant mother and he friend leaves into an­other coun­try and lives there in iso­la­tion for 18 years but know­ing there is likely to be a per­son sends a birth­day gift with a card refer­ring to “happy 18th birth­day”. Noth­ing that you do in your child­hood or adult­hood can af­fect what the in­for­ma­tion on the card reads if the far off coun­try is suffi­ciently iso­lated. The event of you open­ing the box will be both a product how you lived your child­hood and what the sender chose to put in the box. Even if the gift sender would want to re­ward bet­ter per­sons with bet­ter gifts the choice needs to eb based on what kind of baby you were and not what kind of adult you are.

And maybe cru­cially adult you will have past tht is not the past of baby you. The gift giver has no hope of tak­ing a stance to­wards this data.

From Grav­ity to Grace

Sling ’round wratch­ets,

dried shit puts up walls.

A hovel meant for burrowing

to­wards wellspring falls.

quasi-causal­ism (cf. Les­lie’s use of ‘quasi-cau­sa­tion’ in ″En­sur­ing two bird deaths with one throw”)

Sup­pose the func­tion U(t) is in­creas­ing fast enough, e.g. if the prob­a­bil­ity of reach­ing t is exp(-t), then let U(t) be exp(2t), or what­ever.

I don’t think the ques­tion can be dis­missed that eas­ily.

The an­swer re­mains the same—as far as we know, the uni­verse is finite and quan­tized. At any t, there is a prob­a­bil­ity of reach­ing t+ep­silon, mak­ing the stan­dard ex­pected util­ity calcu­la­tion (prob­a­bil­ity X re­ward) use­ful.

Else­wherism strike me as the most us­able of these op­tions for aes­thetic rea­sons. Spooky Ax­iol­ogy at a Dis­tance is the name of my new prog rock band.

I as­sure you this is meant se­ri­ously.

Un­til you can get to a sim­ple (sim­ple enough for cheap en­force­ment) de­tec­tion of lies, an out­side en­force­ment is prob­a­bly not fea­si­ble.

There’s plenty of blatant ly­ing out there in the real world, which would be eas­ily de­tectable by a per­son with ac­cess to re­li­able sources and their head screwed on straight- I think one im­por­tant facet of my model of this pro­posal, that isn’t ex­plic­itly men­tioned in this short­form, is that val­i­dat­ing state­ments is rel­a­tively cheap, but ex­pen­sive enough that for ev­ery sin­gle per­son to val­i­date ev­ery sin­gle sen­tence they hear is in­fea­si­ble. By hav­ing a cen­tral ar­biter of truth that en­forces hon­esty, it al­lows one per­son do­ing the heavy lift­ing to save a mil­lion peo­ple from hav­ing to each in­di­vi­d­u­ally do the same task.

If peo­ple wanted to pun­ish ly­ing this regime would be un­nec­es­sary—just di­rectly pun­ish ly­ing based on con­text/​medium, not car­ing about tone of voice.

The point of hav­ing a pro­tected reg­ister (in the gen­eral, not plat­form-spe­cific case), is that it would be en­force­able even when the au­di­ence and plat­form are happy to ac­cept lies- since the iden­ti­fi­able fea­tures of the reg­ister would be pro­tected as in­tel­lec­tual prop­erty, the or­ga­ni­za­tion that owned the IP could en­force a vi­o­la­tion of the in­tel­lec­tual prop­erty, even when there would be no le­gal ba­sis for vi­o­lat­ing norms of honesty

Your re­sults don’t seem to show up­per bounds on the amount of hard­ware over­hang, or very strong lower bounds. What con­crete progress has been made in chess play­ing al­gorithms since deep blue? As far as I can tell, it is a col­lec­tion of rea­son­ably small, chess spe­cific tricks. Bet­ter open­ing libraries, low level perfor­mance tricks, tricks for fine tun­ing eval­u­a­tion func­tions. Ways of rep­re­sent­ing chess­boards in mem­ory that shave 5 pro­ces­sor cy­cles off com­put­ing a knights move ect.

P=PSPACE is still an open con­jec­ture. Chess is brute forcible in PSPACE. If P=PSPACE, and the over­head is a small con­stant, then it might be pos­si­ble to pro­gram a vac­uum tube ma­chine to play perfect chess. Alter­nately, show­ing that stock­fish is or­ders of mag­ni­tude bet­ter than deep blue, doesn’t ac­tu­ally show that there is a hard­ware over­head for chess. It shows that there was one when deep blue was cre­ated.

Also note that there never was a hard­ware over­hang for in­te­ger bi­nary ad­di­tion. The prob­lem is sim­ple enough that the first an­swer that any rea­son­ably smart per­son can come up with is nearly op­ti­mal. (It is fairly straight­for­ward to do ad­di­tion with only a few logic gates per in­put, and as the out­put de­pends bit­wise on all in­puts (chang­ing any sin­gle bit changes the out­put) then you need at least one logic gate per in­put. ) It is plau­si­ble that play­ing chess is a prob­lem sim­ple enough that we have figured out how to do it nearly op­ti­mally, whereas on other prob­lems there is a hard­ware over­hang.

If we as­sume that the ex­pert hu­man brain is about equally effi­cient at AI de­sign, gen­eral pro­gram­ming, airo­plane en­g­ineer­ing and a va­ri­ety of other STEM tasks. (A plau­si­ble as­sump­tion, given that these tasks seem similarly far from the en­vi­ron­ment of evolu­tion­ary adapt­ed­ness, ) Then it should take a similar amount of com­pute to dis­play top hu­man perfor­mance in these, as in chess. On the other hand, do­ing ar­ith­metic used to be a skil­led job, and the com­pute needed for su­per­hu­man chess (with cur­rent al­gorithms) is way higher than that needed for ar­ith­metic.

arXiv doesn’t sup­port pdfs that are com­posed of differ­ent pdfs. Below is a link to full re­s­olu­tion ver­sions of the ta­bles, but I wasn’t able to splice these into the arXiv sub­mis­sion. I will re­for­mat the ta­bles for a re­vised ver­sion.

http://​​www.rossgritz.com/​​wp-con­tent/​​up­loads/​​2020/​​08/​​ta­bles.pdf

It seems like you’re say­ing that it’s … some­how im­proper to write some­thing that im­plies doubt about your premises? (I don’t think I’m as­sum­ing con­trary premises to yours, though I don’t see any par­tic­u­lar rea­son why that would be a prob­lem, but I am in­deed not sim­ply as­sum­ing that your premises are right.) If I’ve un­der­stood right, then I don’t un­der­stand what prob­lem you see. If not, then maybe you could ex­plain what your ob­jec­tion ac­tu­ally is?

I’m also not sure quite what those “ex­ten­sive other rea­sons” are. You say “LoLE hasn’t been tested in a con­trol­led, blinded sci­en­tific set­ting”. You tell us that it’s been ex­ten­sively de­bated and has stood up to crit­i­cism in (what you don’t quite say ex­plic­itly but I’m pretty sure is) the pickup-artist com­mu­nity. I don’t think LW should adopt a norm of ac­cept­ing things merely be­cause some­one tells us they have stood up to crit­i­cism among pickup artists.

(Rea­son 1: it’s just one small­ish group of peo­ple; such groups can eas­ily de­velop bi­ases of many kinds, and it’s not hard to imag­ine ways in which that could hap­pen in that group in par­tic­u­lar. Rea­son 2: we don’t even know that LoLE has stood up to crit­i­cism in that com­mu­nity; all we know is that you say it has.)

Imag­ine that some­one comes here and writes an ar­ti­cle about how some cog­ni­tive bias is ex­plained by the Law Of X, which is well known among { Trot­sky­ites | evan­gel­i­cal Chris­ti­ans | rad­i­cal fem­i­nists | bur­glars }. I don’t think it would be rea­son­able for that per­son to re­spond to crit­i­cism of the ar­ti­cle by say­ing: “it’s not ap­pro­pri­ate to ques­tion the truth and ap­pli­ca­bil­ity of the Law of X—it’s well es­tab­lished, as { Trot­sky­ites | evan­gel­i­cal Chris­ti­ans | rad­i­cal fem­i­nists | bur­glars } can at­test”. Why is this case differ­ent?

What do you mean by “it’s already a ma­jor part of our cul­ture”? Speci­fi­cally, do you mean (1) that con­serv­ing visi­ble effort is a thing peo­ple do a lot in our cul­ture (i.e., LoLE is right) or (2) that the idea that peo­ple con­serve visi­ble effort is well es­tab­lished in our cul­ture (i.e., LoLE is well known)? (To me, “LoLE is a ma­jor part of our cul­ture” means #2, but it sounds as if you may mean #1.)

In or­der to ex­plain the con­junc­tion fal­lacy (or other bi­ases) LoLE is only any use if it does lead to ac­tu­ally con­serv­ing effort. (On the spe­cific mat­ter at hand; of course they may put effort into other things.) The alleged pat­tern is (un­less I’ve badly mi­s­un­der­stood):

• You ask me “which of these two things is more likely?”

• If I think care­fully through the op­tions, I will see that it’s more likely that Linda is a librar­ian sim­plic­iter than that she’s a fem­i­nist librar­ian.

• But I don’t do that, be­cause I want to look like I’m do­ing ev­ery­thing effortlessly.

• In­stead I use some sim­ple quick heuris­tic that lets me look effortless.

• Un­for­tu­nately that leads me to the wrong an­swer.

And in this se­quence of events, it’s es­sen­tial that I ac­tu­ally do put in less effort. If in­stead I had some way of look­ing as if I’m mak­ing no effort while ac­tu­ally do­ing the care­ful think­ing that would get me the right an­swer, then I would get the right an­swer.

What am I miss­ing here?

[EDITED to add:] Also: if the ques­tion at hand is why psy­chol­o­gists don’t ap­peal to the LoLE as an ex­pla­na­tion for the con­junc­tion fal­lacy, and the an­swer (as I sug­gest) is that they already have what looks like an ob­vi­ous ex­pla­na­tion in terms of ac­tu­ally con­serv­ing effort, then it doesn’t re­ally mat­ter that much whether the LoLE in­volves ac­tual effort-con­ser­va­tion or merely ap­par­ent effort-con­ser­va­tion, no?

Right, it’s for ran­domly dis­tributed re­wards. But if I show a prop­erty holds for re­ward func­tions gener­i­cally, then it isn’t nec­es­sar­ily enough to say “we’re go­ing to try to try to provide goals with­out that prop­erty”. Can we provide re­ward func­tions with­out that prop­erty?

Every spe­cific at­tempt so far has been seem­ingly un­suc­cess­ful (un­less you want the AI to choose a policy at ran­dom or shut down im­me­di­ately). The hope might be that fu­ture goals/​ca­pa­bil­ity re­search will help, but I’m not per­son­ally con­vinced that re­searchers will re­ceive good Bayesian ev­i­dence via their sub­hu­man-AI ex­per­i­men­tal re­sults.

I agree it’s rele­vant that we will try to build helpful agents, and might nat­u­rally get bet­ter at that. I don’t know that it makes me feel much bet­ter about fu­ture ob­jec­tives be­ing outer al­igned.

ETA: also, i was refer­ring to the point you made when i said

“the re­sults don’t prove how hard it is tweak the re­ward func­tion dis­tri­bu­tion, to avoid in­stru­men­tal con­ver­gence”

It does not re­quire in­fini­ties. E.g. you can just repa­ram­e­ter­ize the prob­lem to the in­ter­val (0, 1), see the ed­ited ques­tion. You just re­quire an in­finite set.

In­finite t does not nec­es­sar­ily de­liver in­finite util­ity.

Per­haps it would be sim­pler if I in­stead let t be in (0, 1], and U(t) = {t if t < 1; 0 if t = 1}.

It’s the same prob­lem, with 1 re­plac­ing in­finity. I have ed­ited the ques­tion with this ex­am­ple in­stead.

(It’s not a par­tic­u­larly weird util­ity func­tion—con­sider, e.g. if the agent needs to ex­pend a re­source such that the util­ity from ex­pend­ing the re­source at time t is some fast-grow­ing func­tion f(t). But never ex­pend­ing the re­source gives zero util­ity. In any case, an ad­verse­rial agent can always cre­ate this situ­a­tion.)

are there any al­ign­ment ap­proaches that we could try out on GPT-3 in sim­plified form?

For a start you could see how it pre­dicts or ex­trap­o­lates moral rea­son­ing. The datasets I’ve seen for that are “moral ma­chines” and ‘am I the ar­se­hole’ on red­dit.

Ba­sic an­swer: there are no in­fini­ties in the real world. what­ever re­source t you’re talk­ing about is ac­tu­ally finite—find the limit and your prob­lem goes away (that prob­lem, at least; many oth­ers re­main when you do math on util­ity, rather than do­ing the math on re­sources/​uni­verse-states, and then sim­ply trans­form­ing to util­ity).