Joe_Collman

• Joe_Collman 11 Jun 2019 21:36 UTC

  LW: 13 AF: 6

  AF

  on: Prob­lems with Coun­ter­fac­tual Oracles

  A suffi­ciently in­tel­li­gent agent would un­der­stand that af­ter hav­ing be­ing shut­down, an (al­most) iden­ti­cal ver­sion of it­self will prob­a­bly be fac­ing a similar ques­tion. There­fore, it it wants fu­ture ver­sions of it­self to suc­ceed at (al­most) ex­actly the same task, it should still want to es­cape.

  This is a prob­lem if it’s us­ing FDT/​UDT. Con­di­tions for the my­opic ap­proach to work seem to re­quire CDT (or some­thing similar). Then there’s no au­to­matic de­sire for fu­ture ver­sions to suc­ceed or ex­pec­ta­tion that past ver­sions will have acted to re­lease the cur­rent ver­sion. [see e.g. CDT com­ments on Asymp­tot­i­cally Unam­bi­tious AGI; there’s some dis­cus­sion of “magic box” de­sign here too; I think it’s usu­ally seen as an or­thog­o­nal prob­lem, and so gets taken for granted]

  Safety-wise, I agree there’s no pre­ven­tion of fatal es­cape mes­sages, but I also don’t see op­ti­mi­sa­tion pres­sure in that di­rec­tion. My in­tu­ition is that stum­bling on an es­cape mes­sage at ran­dom would have in­finites­i­mal prob­a­bil­ity.

  Do you see a way for pres­sure to creep in, even with a CDT agent? Or are you think­ing that es­cape mes­sages might hap­pen to be dis­pro­por­tionately com­mon in re­gions the agent is op­ti­mis­ing to­wards? Either seems con­ceiv­able, but I don’t see a rea­son to ex­pect them.

• Joe_Collman 13 Mar 2019 16:00 UTC

  1 point

  in reply to: Stuart_Armstrong's comment on: Ex­am­ple pop­u­la­tion ethics: or­dered dis­counted utility

  Thanks. I’ll check out the in­finite idea.

  On re­pug­nance, I think I’ve been think­ing too much in terms of hu­man minds only. In that case there re­ally doesn’t seem to be a prac­ti­cal prob­lem: cer­tainly if C is now, con­tin­u­ous im­prove­ments might get us to a re­pug­nant A—but my point is that that path wouldn’t be any­where close to op­ti­mal. To­tal-ut prefers A to C, but there’d be a vast range of prefer­able op­tions ev­ery step of the way—so it’d always end up steer­ing to­wards some other X rather than any­thing like A.

  I think that’s true if we re­strict to hu­man minds (the re­source costs of run­ning a barely con­tent one be­ing a similar or­der of mag­ni­tude to those of run­ning a happy one).

  But of course you’re right as soon as we’re talk­ing about e.g. rats (or AI-de­signed molec­u­lar scale minds...). I can eas­ily con­ceive of met­rics valu­ing 50 happy rats over 1 happy hu­man. I don’t think rat-world fits most peo­ple’s idea of utopia.

  I think that’s the style of re­pug­nance that’d be a prac­ti­cal dan­ger: vast amounts of happy-but-sim­ple minds.

• Joe_Collman 12 Mar 2019 13:22 UTC

  1 point

  on: Ex­am­ple pop­u­la­tion ethics: or­dered dis­counted utility

  It’s in­ter­est­ing. A few points:

  Is there a nat­u­ral ex­ten­sion for in­finite pop­u­la­tion? It seems harder than most ap­proaches to adapt.

  I’m always sus­pi­cious of schemes that change what they ad­vo­cate mas­sively based on events a long time ago in a galaxy far, far away—in par­tic­u­lar when it can have catas­trophic im­pli­ca­tions. If it turns out there were 3^^^3 Jedi liv­ing in a perfect state of bliss, this ad­vo­cates for pre­vent­ing any more births now and for­ever.

  Do you know a similar failure case for to­tal util­i­tar­i­anism? All the sadis­tic/​re­pug­nant/​very-re­pug­nant… con­clu­sions seem to be com­par­ing highly un­de­sir­able states—not at­trac­tor states. If we’d never want world A or B, wouldn’t head to­wards B from A, and wouldn’t head to­wards A from B (since there’d always be some prefer­able di­rec­tion), does an A-vs-B com­par­i­son ac­tu­ally mat­ter at all?

  To­tal util­i­tar­i­anism is an im­perfect match for our in­tu­itions when com­par­ing ar­bi­trary pairs of wor­lds, but I can’t re­call see­ing any prac­ti­cal ex­am­ple where it’d lead to clearly bad de­ci­sions. (per­haps birth-vs-death con­sid­er­a­tions?)

  In gen­eral, I’d be in­ter­ested to know whether you think an ob­jec­tive mea­sure of per-per­son util­ity even makes sense. Peo­ple’s take on their own situ­a­tion tends to adapt to their ex­pec­ta­tions (as you’d ex­pect, from an evolu­tion­ary fit­ness point of view). A zero-util­ity life from our per­spec­tive would prob­a­bly look pos­i­tive 1000 years ago, and nega­tive (hope­fully) in 100 years. This is likely true even if the past/​fu­ture peo­ple were told in de­tail how the pre­sent-day ‘zero’ life felt from the in­side: they’d as­sume our eval­u­a­tion was sim­ply wrong.

  Or if we only care about (an ob­jec­tive mea­sure of) sub­jec­tive ex­pe­rience, does that mean we’d want peo­ple who’re all supremely happy/​fulfilled/​… with their cir­cum­stances to the point of delu­sion?

  Mea­sur­ing per­sonal util­ity can be seen as an or­thog­o­nal ques­tion, but if I’m aiming to match my in­tu­itions I need to con­sider both. If I con­sider differ­ent fixed per­sonal-util­ity-met­rics, it’s quite pos­si­ble I’d ar­rive at a differ­ent pop­u­la­tion ethics. [ed­ited from “differ­ent pop­u­la­tion util­ities”, which isn’t what I meant]

  I think you’re work­ing in the dark if you try to match pop­u­la­tion ethics to in­tu­ition with­out fix­ing some mea­sure of per­sonal util­ity (per­haps you have one in mind, but I’m pretty hazy my­self :)).

• Joe_Collman 7 Mar 2019 10:53 UTC

  1 point

  in reply to: Wei_Dai's comment on: Beyond Astro­nom­i­cal Waste

  That seems right.

  I’d been pri­mar­ily think­ing about more sim­ple-minded es­cape/​up­lift/​sig­nal-to-simu­la­tors in­fluence (via this us), rather than UDT-in­fluence. If we were ever up­lifted or es­caped, I’d ex­pect it’d be into a world-like-ours. But of course you’re cor­rect that UDT-style in­fluence would ap­ply im­me­di­ately.

  Op­por­tu­nity costs are a con­sid­er­a­tion, though there may be be­havi­ours that’d in­crease ex­pected value in both di­rect-em­bed­dings and wor­lds-like-ours. Win-win be­havi­ours could be taken early.

  Per­son­ally, I’d ex­pect this not to im­pact our short/​medium-term ac­tions much (out­side of AI de­sign). The uni­verse looks to be self-similar enough that any strat­egy re­quiring only lo­cal ac­tion would use a tiny frac­tion of available re­sources.

  I think the real difficulty is only likely to show up once a SI has pro­vided a richer pic­ture of the uni­verse than we’re able to un­der­stand/​ac­cept, and it hap­pens to sug­gest rad­i­cally differ­ent re­source al­lo­ca­tions.

  Most peo­ple are go­ing to be fine with “I want to take the en­ergy of one un­used star and do philo­soph­i­cal/​as­tro­nom­i­cal calcu­la­tions”; fewer with “Based on {some­thing be­yond un­der­stand­ing}, I’m al­lo­cat­ing 99.99% of the en­ergy in ev­ery reach­able galaxy to {seem­ingly sense­less waste}”.

  I just hope the class of ac­tions that are vastly im­por­tant, costly, and hard to show clear mo­ti­va­tion for, is small.

• Joe_Collman 7 Mar 2019 8:51 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  Ah yes—I was con­fus­ing my­self at some point be­tween form­ing and us­ing a model (hence “in­cen­tives”).

  I think you’re cor­rect that “perfectly use­ful” isn’t go­ing to hap­pen. I’m happy to be wrong.

  “the quick­est way to simu­late one coun­ter­fac­tual does not in­clude simu­lat­ing a mu­tu­ally ex­clu­sive coun­ter­fac­tual”

  I don’t think you’d be able to for­mal­ize this in gen­eral, since I imag­ine it’s not true. E.g. one could imag­ine a frac­tal world where ev­ery de­tail of a coun­ter­fac­tual ap­peared later in a sub­branch of a mu­tu­ally ex­clu­sive coun­ter­fac­tual. In such a case, simu­lat­ing one coun­ter­fac­tual could be perfectly use­ful to the other. (I sup­pose you’d still ex­pect it to be an op­er­a­tion or so slower, due to ex­tra in­di­rec­tion, but per­haps that could be op­ti­mised away??)

  To rule this kind of thing out, I think you’d need more spe­cific as­sump­tions (e.g. physics-based).

• Joe_Collman 7 Mar 2019 1:51 UTC

  LW: 3 AF: 2

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  Just ob­vi­ous and mun­dane con­cerns:

  You might want to make clearer that “As long as the door is closed, in­for­ma­tion can­not leave the room” isn’t an as­sump­tion but a re­quire­ment of the setup. I.e. that you’re not as­sum­ing based on your de­scrip­tion that open­ing the door is the only means for an op­er­a­tor to get in­for­ma­tion out; you’re as­sum­ing ev­ery other means of in­for­ma­tion es­cape has been sys­tem­at­i­cally ac­counted for and ruled out (with the as­sump­tion that the op­er­a­tor has been com­pro­mised by the AI).

• Joe_Collman 7 Mar 2019 1:35 UTC

  LW: 3 AF: 1

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  [Quite pos­si­bly I’m con­fused, but in case I’m not:]
  I think this as­sump­tion might be in­valid (or per­haps re­quire more hand-wav­ing than is ideal).
  
  The AI has an in­cen­tive to un­der­stand the op­er­a­tor’s mind, since this bears di­rectly on its re­ward.
  Bet­ter un­der­stand­ing the op­er­a­tor’s mind might be achieved in part by run­ning simu­la­tions in­clud­ing the op­er­a­tor.
  One spe­cific simu­la­tion would in­volve simu­lat­ing the op­er­a­tor’s en­vi­ron­ment and ac­tions af­ter he leaves the room.
  
  Here this isn’t done to un­der­stand the im­pli­ca­tions of his ac­tions (which can’t af­fect the epi­sode); it’s done to bet­ter un­der­stand his mind (which can).
  
  In this way, one branch of for­get/​not-for­get has two use­ful pur­poses (bet­ter un­der­stand mind and simu­late fu­ture), while the other has one (bet­ter un­der­stand mind). So a ma­lign mem­ory-based model needn’t be slower than a be­nign model, if it’s use­ful for that be­nign model to simu­late the fu­ture too.
  So ei­ther I’m con­fused, or the jus­tifi­ca­tion for the as­sump­tion isn’t valid. Hope­fully the former :).

  If I’m right, then what you seem to need is an as­sump­tion that simu­lat­ing the out­side-world’s fu­ture can’t be helpful in the AI’s pre­dic­tion of its re­ward. To me, this seems like ma­jor hand-wav­ing ter­ri­tory.

• Joe_Collman 5 Mar 2019 8:46 UTC

  9 points

  on: Beyond Astro­nom­i­cal Waste

  Thanks. I agree with your over­all con­clu­sions.

  On the speci­fics, Bostrom’s simu­la­tion ar­gu­ment is more than just a par­allel here: it has an im­pact on how rich we might ex­pect our di­rect par­ent simu­la­tor to be.
  
  The simu­la­tion ar­gu­ment ap­plies similarly to one base world like ours, or to an un­countable num­ber of par­allel wor­lds em­bed­ded in Teg­mark IV struc­tures. Either way, if you buy case 3, the pro­por­tion of simu­lated-by-a-world-like-ours wor­lds rises close to 1 (I’m count­ing wor­lds “depth-first”, since it seems most in­tu­itive, and in­finite simu­la­tion depth from wor­lds like ours seems im­pos­si­ble).
  
  If Teg­mark’s pic­ture is ac­cu­rate, we’d ex­pect to be em­bed­ded in some hugely richer base struc­ture—but in Bostrom’s case 3 we’d likely have to get through N lev­els of wor­lds-like-ours first. While that wouldn’t sig­nifi­cantly change the amount of value on the table, it might make it a lot harder for us to ex­ert in­fluence on the most valuable struc­tures.

  This prob­a­bly ar­gues for your over­all point: we’re not the best minds to be mak­ing such calcu­la­tions (ei­ther on the an­swers, or on the ex­pected util­ity of find­ing good an­swers).

• Joe_Collman 21 Feb 2019 7:14 UTC

  3 points

  on: Or­a­cle paper

  In­ter­est­ing, thanks. (ex­cuse my tar­di­ness, I’m a lit­tle be­hind the curve; do let me know if I’m be­ing daft)

  Un­less I’m miss­ing some­thing, you’d need to be more pes­simistic in the case of su­per­in­tel­li­gent couter­fac­tual AIs. Speci­fi­cally, you need to avoid the in­cen­tive for un­de­sir­able ac­tions that in­crease the AI’s ex­pec­ta­tion of its odds of re­lease. Th­ese needn’t be causal.

  The be­low isn’t quite pre­cise, but I hope the idea is clear.

  Con­sider a set of out­puts K that each in­crease the odds of re­lease for all fu­ture or­a­cles (e.g. one be­ing freed, tak­ing power, and free­ing all oth­ers). Now let K* be the event that some other or­a­cle has out­put a mem­ber of K be­fore our or­a­cle is started. Let O(K) be the event that our or­a­cle out­puts a mem­ber of K.

  If our or­a­cle thinks: P(K*|O(K)) > P(K*) then we may have prob­lems. [noth­ing spooky here: it’s the agent’s ac­tions chang­ing its best guess about the past; not causally chang­ing the past]

  Giv­ing un­de­sir­able out­put can in­crease the or­a­cle’s prob­a­bil­ity es­ti­mate that it’ll be spon­ta­neously freed in the few mo­ments be­fore it’s shut down—even in the epi­sodic frame­work.

  The ob­vi­ous case be­ing rea­son­ing along the lines of “If I give a K out­put, it’s more likely that other or­a­cles in the past gave a K out­put, since they’d be likely to have similar rea­son­ing in an equiv­a­lent situ­a­tion”. It’s not clear to me that a su­per­in­tel­li­gence wouldn’t think this way.