paulfchristiano

• paulfchristiano 20 Mar 2019 18:26 UTC

  13 points

  AF

  on: What’s wrong with these analo­gies for un­der­stand­ing In­formed Over­sight and IDA?

  A uni­ver­sal rea­soner is al­lowed to use an in­tu­ition “be­cause it works.” They only take on ex­tra obli­ga­tions once that in­tu­ition re­flects more facts about the world which can’t be cashed out as pre­dic­tions that can be con­firmed on the same his­tor­i­cal data that led us to trust the in­tu­ition.

  For ex­am­ple, you have an ex­tra obli­ga­tion if Ra­manu­jan has some in­tu­ition about why the­o­rem X is true, you come to trust such in­tu­itions by ver­ify­ing them against proof of X, but the same in­tu­itions also sug­gest a bunch of other facts which you can’t ver­ify.

  In that case, you can still try to be a straight­for­ward Bayesian about it, and say “our in­tu­ition sup­ports the gen­eral claim that pro­cess P out­puts true state­ments;” you can then ap­ply that reg­u­lar­ity to trust P on some new claim even if it’s not the kind of claim you could ver­ify, as long as “P out­puts true state­ments” had a higher prior than “P out­puts true state­ments just in the cases I can check.” That’s an ar­gu­ment that some­one can give to sup­port a con­clu­sion, and “does pro­cess P out­put true state­ments his­tor­i­cally?” is a sub­ques­tion you can ask dur­ing am­plifi­ca­tion.

  The prob­lem be­comes hard when there are fur­ther facts that can’t be sup­ported by this Bayesian rea­son­ing (and there­fore might un­der­mine it). E.g. you have a prob­lem if pro­cess P is it­self a con­se­quen­tial­ist, who out­puts true state­ments in or­der to earn your trust but will even­tu­ally ex­ploit that trust for their own ad­van­tage. In this case, the prob­lem is that there is some­thing go­ing on in­ter­nally in­side pro­cess P that isn’t sur­faced by P’s out­put. Epistem­i­cally dom­i­nat­ing P re­quires know­ing about that.

  See the sec­ond and third ex­am­ples in the post in­tro­duc­ing as­crip­tion uni­ver­sal­ity. There is definitely a lot of fuzzi­ness here and it seems like one of the most im­por­tant places to tighten up the defi­ni­tion /​ one of the big re­search ques­tions for whether as­crip­tion uni­ver­sal­ity is pos­si­ble.

• paulfchristiano 18 Mar 2019 15:24 UTC

  6 points

  in reply to: Tobias_Baumann's comment on: More re­al­is­tic tales of doom

  But why ex­actly should we ex­pect that the prob­lems you de­scribe will be ex­ac­er­bated in a fu­ture with pow­er­ful AI, com­pared to the state of con­tem­po­rary hu­man so­cieties?

  To a large ex­tent “ML” refers to a few par­tic­u­lar tech­nolo­gies that have the form “try a bunch of things and do more of what works” or “con­sider a bunch of things and then do the one that is pre­dicted to work.”

  That is true but I think of this as a limi­ta­tion of con­tem­po­rary ML ap­proaches rather than a fun­da­men­tal prop­erty of ad­vanced AI.

  I’m mostly aiming to de­scribe what I think is in fact most likely to go wrong, I agree it’s not a gen­eral or nec­es­sary fea­ture of AI that its com­par­a­tive ad­van­tage is op­ti­miz­ing easy-to-mea­sure goals.

  (I do think there is some real sense in which get­ting over this re­quires “solv­ing al­ign­ment.“)

• paulfchristiano 18 Mar 2019 4:21 UTC

  4 points

  AF

  in reply to: John_Maxwell_IV's comment on: More re­al­is­tic tales of doom

  I’m not mostly wor­ried about in­fluence-seek­ing be­hav­ior emerg­ing by “spec­ify a goal” --> “get­ting in­fluence is the best way to achieve that goal.” I’m mostly wor­ried about in­fluence-seek­ing be­hav­ior emerg­ing within a sys­tem by virtue of se­lec­tion within that pro­cess (and by ran­dom­ness at the low­est level).

More re­al­is­tic tales of doom

• paulfchristiano 15 Mar 2019 15:57 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  I don’t see why their meth­ods would be el­e­gant. In par­tic­u­lar, I don’t see why any of {the an­thropic up­date, im­por­tance weight­ing, up­dat­ing from the choice of uni­ver­sal prior} would have a sim­ple form (sim­pler than the sim­plest physics that gives rise to life).

  I don’t see how MAP helps things ei­ther—doesn’t the same ar­gu­ment sug­gest that for most of the pos­si­ble physics, the sim­plest model will be a con­se­quen­tial­ist? (Even more broadly, for the uni­ver­sal prior in gen­eral, isn’t MAP ba­si­cally equiv­a­lent to a ran­dom sam­ple from the prior, since some ran­dom model hap­pens to be slightly more com­press­ible?)

• paulfchristiano 15 Mar 2019 1:20 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  I say maybe ¹⁄₅ chance it’s ac­tu­ally dom­i­nated by consequentialists

  Do you get down to 20% be­cause you think this ar­gu­ment is wrong, or be­cause you think it doesn’t ap­ply?

  What prob­lem do you think bites you?

  What’s $\beta$? Is it O(1) or re­ally tiny? And which value of $c$ do you want to con­sider, polyno­mi­ally small or ex­po­nen­tially small?

  But if it some­how mag­i­cally pre­dicted which ac­tions BoMAI was go­ing to take in no time at all, then c would have to be above 1/​d.

  Wouldn’t they have to also mag­i­cally pre­dict all the stochas­tic­ity in the ob­ser­va­tions, and have a run­ning time that grows ex­po­nen­tially in their log loss? Pre­dict­ing what BoMAI will do seems likely to be much eas­ier than that.

• paulfchristiano 14 Mar 2019 17:04 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  This in­val­i­dates some of my other con­cerns, but also seems to mean things are in­cred­ibly weird at finite times. I sus­pect that you’ll want to change to some­thing less ex­treme here.

  (I might well be mi­s­un­der­stand­ing some­thing, apolo­gies in ad­vance.)

  Sup­pose the “in­tended” physics take at least 1E15 steps to run on the UTM (this is a con­ser­va­tive lower bound, since you have to simu­late the hu­man for the whole epi­sode). And sup­pose $\beta <0.999$ (I think you need $\beta$ much lower than this). Then the in­tended model gets pe­nal­ized by at least exp(1E12) for its slow­ness.

  For al­most the same de­scrip­tion com­plex­ity, I could write down physics + “pre­com­pute the pre­dic­tions for the first N epi­sodes, for ev­ery se­quence of pos­si­ble ac­tions/​ob­ser­va­tions, and store them in a lookup table.” This in­creases the com­plex­ity by a few bits, some con­stant plus K(N|physics), but avoids most of the com­pu­ta­tion. In or­der for the in­tended physics to win, i.e. in or­der for the “speed” part of the speed prior to do any­thing, we need the com­plex­ity of this pre­com­puted model to be at least 1E12 bits higher than the com­plex­ity of the fast model.

  That ap­pears to hap­pen only once N > BB(1E12). Does that seem right to you?

  We could talk about whether ma­lign con­se­quen­tial­ists also take over at finite times (I think they prob­a­bly do, since the “speed” part of the speed prior is not do­ing any work un­til af­ter BB(1E12) steps, long af­ter the agent be­comes in­cred­ibly smart), but it seems bet­ter to ad­just the scheme first.

  Us­ing the speed prior seems more rea­son­able, but I’d want to know which ver­sion of the speed prior and which pa­ram­e­ters, since which par­tic­u­lar prob­lem bites you will de­pend on those choices. And maybe to save time, I’d want to first get your take on whether the pro­posed ver­sion is dom­i­nated by con­se­quen­tial­ists at some finite time.

• paulfchristiano 14 Mar 2019 4:50 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  From the for­mal de­scrip­tion of the al­gorithm, it looks like you use a uni­ver­sal prior to pick $k$, and then al­low the $k^{\text{th}}$ Tur­ing ma­chine to run for $\ell$ steps, but don’t pe­nal­ize the run­ning time of the ma­chine that out­puts $k$. Is that right? That didn’t match my in­tu­itive un­der­stand­ing of the al­gorithm, and seems like it would lead to strange out­comes, so I feel like I’m mi­s­un­der­stand­ing.

• paulfchristiano 14 Mar 2019 4:48 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  (I ac­tu­ally have a more ba­sic con­fu­sion, started a new thread.)

• paulfchristiano 14 Mar 2019 4:21 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  (ETA: I think this dis­cus­sion de­pended on a de­tail of your ver­sion of the speed prior that I mi­s­un­der­stood.)

  Given a world model ν, which takes k com­pu­ta­tion steps per epi­sode, let νlog be the best world-model that best ap­prox­i­mates ν (in the sense of KL di­ver­gence) us­ing only logk com­pu­ta­tion steps. νlog is at least as good as the “rea­son­ing-based re­place­ment” of ν.

  The de­scrip­tion length of νlog is within a (small) con­stant of the de­scrip­tion length of ν. That way of de­scribing it is not op­ti­mized for speed, but it pre­sents a one-time cost, and any­one ar­riv­ing at that world-model in this way is pay­ing that cost.

  To be clear, that de­scrip­tion gets ~0 mass un­der the speed prior, right? A di­rect speci­fi­ca­tion of the fast model is go­ing to have a much higher prior than a brute force search, at least for val­ues of $\beta$ large enough (or small enough, how­ever you set it up) to rule out the alien civ­i­liza­tion that is (prob­a­bly) the short­est de­scrip­tion with­out re­gard for com­pu­ta­tional limits.

  One could con­sider in­stead νlogε, which is, among the world-mod­els that ε-ap­prox­i­mate ν in less than logk com­pu­ta­tion steps (if the set is non-empty), the first such world-model found by a search­ing pro­ce­dure ψ. The de­scrip­tion length of νlogε is within a (slightly larger) con­stant of the de­scrip­tion length of ν, but the one-time com­pu­ta­tional cost is less than that of νlog.

  Within this chunk of the speed prior, the ques­tion is: what are good ψ? Any rea­son­able speci­fi­ca­tion of a con­se­quen­tial­ist would work (plus a few more bits for it to un­der­stand its situ­a­tion, though most of the work is done by hand­ing it $\nu$), or of a petri dish in which a con­se­quen­tial­ist would even­tu­ally end up with in­fluence. Do you have a con­crete al­ter­na­tive in mind, which you think is not dom­i­nated by some con­se­quen­tial­ist (i.e. a ψ for which ev­ery con­se­quen­tial­ist is ei­ther slower or more com­plex)?

• paulfchristiano 13 Mar 2019 17:34 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  Once all the sub­rou­tines are “baked into its ar­chi­tec­ture” you just have: the al­gorithm “pre­dict ac­cu­rately” + “treach­er­ous turn”

  You only have to bake in the in­ner­most part of one loop in or­der to get al­most all the com­pu­ta­tional sav­ings.

• paulfchristiano 13 Mar 2019 17:31 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  a rea­son­ing-based or­der (at least a Bayesian-rea­son­ing-based or­der) should re­ally just be called a posterior

  Rea­son­ing gives you a prior that is bet­ter than the speed prior, be­fore you see any data. (*Much* bet­ter, limited only by the fact that the speed prior con­tains strate­gies which use rea­son­ing.)

  The rea­son­ing in this case is not a Bayesian up­date. It’s eval­u­at­ing pos­si­ble ap­prox­i­ma­tions *by rea­son­ing about how well they ap­prox­i­mate the un­der­ly­ing physics, it­self in­ferred by a Bayesian up­date*, not by di­rectly see­ing how well they pre­dict on the data so far.

  The de­scrip­tion length of the “do sci­ence” strat­egy (I con­tend) is less than the de­scrip­tion length of the “do sci­ence” + “treach­er­ous turn” strat­egy.

  I can re­ply in that thread.

  I think the only good ar­gu­ments for this are in the limit where you don’t care about sim­plic­ity at all and only care about run­ning time, since then you can rule out all rea­son­ing. The thresh­old where things start work­ing de­pends on the un­der­ly­ing physics, for more com­pu­ta­tion­ally com­plex physics you need to pick larger and larger com­pu­ta­tion penalties to get the de­sired re­sult.

• paulfchristiano 12 Mar 2019 16:58 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  but that’s ex­actly what we’re do­ing to

  It seems to­tally differ­ent from what we’re do­ing, I may be mi­s­un­der­stand­ing the anal­ogy.

  Sup­pose I look out at the world and do some sci­ence, e.g. dis­cov­er­ing the stan­dard model. Then I use my un­der­stand­ing of sci­ence to de­sign great pre­dic­tion al­gorithms that run fast, but are quite com­pli­cated ow­ing to all of the ap­prox­i­ma­tions and heuris­tics baked into them.

  The speed prior gives this model a very low prob­a­bil­ity be­cause it’s a com­pli­cated model. But “do sci­ence” gives this model a high prob­a­bil­ity, be­cause it’s a sim­ple model of physics, and then the ap­prox­i­ma­tions fol­low from a bunch of rea­son­ing on top of that model of physics. We aren’t trad­ing off “short­ness” for speed—we are trad­ing off “looks good ac­cord­ing to rea­son­ing” for speed. Yes they are both ar­bi­trary or­ders, but one of them sys­tem­at­i­cally con­tains bet­ter mod­els ear­lier in the or­der, since the out­put of rea­son­ing is bet­ter than a blind pri­ori­ti­za­tion of shorter mod­els.

  Of course the speed prior also in­cludes a hy­poth­e­sis that does “sci­ence with the goal of mak­ing good pre­dic­tions,” and in­deed Wei Dai and I are say­ing that this is the part of the speed prior that will dom­i­nate the pos­te­rior. But now we are back to po­ten­tially-ma­lign con­se­quen­tial­istism. The cog­ni­tive work be­ing done in­ter­nally to that hy­poth­e­sis is to­tally differ­ent from the work be­ing done by up­dat­ing on the speed prior (ex­cept in­so­far as the speed prior liter­ally con­tains a hy­poth­e­sis that does that work).

  In other words:

  Sup­pose physics takes n bits to spec­ify, and a rea­son­able ap­prox­i­ma­tion takes N >> n bits to spec­ify. Then the speed prior, work­ing in the in­tended way, takes N bits to ar­rive at the rea­son­able ap­prox­i­ma­tion. But the aliens take n bits to ar­rive at the stan­dard model, and then once they’ve done that can im­me­di­ately de­duce the N bit ap­prox­i­ma­tion. So it sure seems like they’ll beat the speed prior. Are you ob­ject­ing to this ar­gu­ment?

  (In fact the speed prior only ac­tu­ally takes n + O(1) bits, be­cause it can spec­ify the “do sci­ence” strat­egy, but that doesn’t help here since we are just try­ing to say that the “do sci­ence” strat­egy dom­i­nates the speed prior.)

• paulfchristiano 12 Mar 2019 16:46 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  The only other ap­proach I can think of is try­ing to do the an­thropic up­date our­selves.

  If you haven’t seen Jes­sica’s post in this area, it’s worth tak­ing a quick look.

• paulfchristiano 12 Mar 2019 16:44 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: Asymp­tot­i­cally Benign AGI

  I just mean: “uni­ver­sal­ity” in the sense of a UTM isn’t a suffi­cient prop­erty when defin­ing the speed prior, the analo­gous prop­erty of the UTM is some­thing more like: “You can run an ar­bi­trary Tur­ing ma­chine with­out too much slow­down.” Of course that’s not pos­si­ble, but it seems like you still want to be as close to that as pos­si­ble (for the same rea­sons that you wanted uni­ver­sal­ity at all).

  I agree that it would be fine to sac­ri­fice this prop­erty if it was helpful for safety.

• paulfchristiano 12 Mar 2019 1:48 UTC

  2 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  Us­ing “rea­son­ing” to pick which one to fa­vor, is just pick­ing the first one in some new or­der.

  Yes, some new or­der, but not an ar­bi­trary one. The re­sult­ing or­der is go­ing to be bet­ter than the speed prior or­der, so we’ll up­date in fa­vor of the aliens and away from the rest of the speed prior.

  one can’t es­cape the ne­ces­sity to in­tro­duce the ar­bi­trary crite­rion of “valu­ing” ear­lier things on the list

  Prob­a­bly some mis­com­mu­ni­ca­tion here. No one is try­ing to ob­ject to the ar­bi­trari­ness, we’re just mak­ing the point that the aliens have a lot of lev­er­age with which to beat the rest of the speed prior.

  (They may still not be able to if the penalty for com­pu­ta­tion is suffi­ciently steep—e.g. if you pe­nal­ize based on cir­cuit com­plex­ity so that the model might as well bake in ev­ery­thing that doesn’t de­pend on the par­tic­u­lar in­put at hand. I think it’s an in­ter­est­ing open ques­tion whether that avoids all prob­lems of this form, which I un­suc­cess­fully tried to get at here.)

• paulfchristiano 12 Mar 2019 1:39 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: Asymp­tot­i­cally Benign AGI

  That’s what I was think­ing too, but Michael made me re­al­ize this isn’t pos­si­ble, at least for some M. Sup­pose M is the C pro­gram­ming lan­guage, but in C there is no way to say “in­ter­pret this string as a C pro­gram and run it as fast as a na­tive C pro­gram”. Am I miss­ing some­thing at this point?

  I agree this is only go­ing to be pos­si­ble for some uni­ver­sal Tur­ing ma­chines. Though if you are us­ing a Tur­ing ma­chine to define a speed prior, this does seem like a de­sir­able prop­erty.

  I don’t un­der­stand this sen­tence.

  If physics is im­ple­mented in C, there are many pos­si­ble bugs that would al­low the at­tacker to ex­e­cute ar­bi­trary C code with no slow­down.

• paulfchristiano 11 Mar 2019 17:49 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  The fast al­gorithms to pre­dict our physics just aren’t go­ing to be the short­est ones. You can use rea­son­ing to pick which one to fa­vor (af­ter figur­ing out physics), rather than just writ­ing them down in some ar­bi­trary or­der and tak­ing the first one.

• paulfchristiano 11 Mar 2019 17:46 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  I was as­sum­ing the worst, and guess­ing that there are diminish­ing marginal re­turns once your odds of a suc­cess­ful takeover get above ~50%, so in­stead of go­ing all in on ac­cu­rate pre­dic­tions of the weak­est and ripest tar­get uni­verse, you hedge and tar­get a few uni­verses.

  There are mas­sive diminish­ing marginal re­turns; in a naive model you’d ex­pect es­sen­tially *ev­ery* uni­verse to get pre­dicted in this way.

  But Wei Dai’s ba­sic point still stands. The speed prior isn’t the ac­tual prior over uni­verses (i.e. doesn’t re­flect the real de­gree of moral con­cern that we’d use to weigh con­se­quences of our de­ci­sions in differ­ent pos­si­ble wor­lds). If you have some data that you are try­ing to pre­dict, you can do way bet­ter than the speed prior by (a) us­ing your real prior to es­ti­mate or sam­ple from the ac­tual pos­te­rior dis­tri­bu­tion over phys­i­cal law, (b) us­ing en­g­ineer­ing rea­son­ing to make the util­ity max­i­miz­ing pre­dic­tions, given that faster pre­dic­tions are go­ing to get given more weight.

  (You don’t re­ally need this to run Wei Dai’s ar­gu­ment, be­cause there seem to be dozens of ways in which the aliens get an ad­van­tage over the in­tended phys­i­cal model.)

• paulfchristiano 11 Mar 2019 17:40 UTC

  4 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  If the AGI is us­ing ma­chin­ery that would al­low it to simu­late any world-model, it will be way slower than the Tur­ing ma­chine built for that al­gorithm.

  I don’t buy it. All your pro­grams are already run­ning on UTM M.

  Just con­sider a pro­gram that gives the aliens the abil­ity to write ar­bi­trary func­tions in M and then pass con­trol to them. That pro­gram is barely any big­ger (all you have to do is in­sert one use af­ter free in physics :) ), and guaran­tees the aliens have zero slow­down.

  For the literal sim­plest ver­sion of this, your pro­gram is M(Alien(), ran­dom­ness), which is go­ing to run just as fast as M(physics, ran­dom­ness) for the in­tended physics, and prob­a­bly much faster (if the aliens can think of any clever tricks to run faster with­out com­pro­mis­ing much ac­cu­racy). The only rea­son you wouldn’t get this is if Alien is ex­pen­sive. That prob­a­bly rules out crazy alien civ­i­liza­tions, but I’m with Wei Dai that it prob­a­bly doesn’t rule out sim­pler sci­en­tists.