rohinmshah

• rohinmshah 13 Dec 2019 0:51 UTC

  LW: 18 AF: 8

  AF

  on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  A year later, I con­tinue to agree with this post; I still think its pri­mary ar­gu­ment is sound and im­por­tant. I’m some­what sad that I still think it is im­por­tant; I thought this was an ob­vi­ous-once-pointed-out point, but I do not think the com­mu­nity ac­tu­ally be­lieves it yet.

  I par­tic­u­larly agree with this sen­tence of Daniel’s re­view:

  I think the post is im­por­tant, be­cause it con­strains the types of valid ar­gu­ments that can be given for ‘freak­ing out about goal-di­rect­ed­ness’, for lack of a bet­ter term.”

  “Con­strain­ing the types of valid ar­gu­ments” is ex­actly the right way to de­scribe the post. Many re­sponses to the post have been of the form “this is miss­ing the point of EU max­i­miza­tion ar­gu­ments”, and yes, the post is de­liber­ately miss­ing that point. The post is not say­ing that ar­gu­ments for AI risk are wrong, just that they are based on in­tu­itions and not math. While I do think that we are likely to build goal-di­rected agents, I do not think the VNM the­o­rem and similar ar­gu­ments sup­port that claim: they sim­ply de­scribe how a goal-di­rected agent should think.

  How­ever, talks like AI Align­ment: Why It’s Hard, and Where to Start and posts like Co­her­ent de­ci­sions im­ply con­sis­tent util­ities seem to claim that “VNM and similar the­o­rems” im­plies “goal-di­rected agents”. While there has been some dis­agree­ment over whether this claim is ac­tu­ally pre­sent, it doesn’t re­ally mat­ter—read­ers come away with that im­pres­sion. I see this post as cor­rect­ing that claim; it would have been ex­tremely use­ful for me to read this post a lit­tle over two years ago, and anec­do­tally I have heard that oth­ers have found it use­ful as well.

  I am some­what wor­ried that if read­ers who read this post in iso­la­tion will get the wrong im­pres­sion, since it re­ally was meant as part of the se­quence. For ex­am­ple, I think Bran­gus’ com­ment post is propos­ing an in­ter­pre­ta­tion of “goal-di­rect­ed­ness” that I pro­posed and ar­gued against in the pre­vi­ous post (see also my re­sponse, which mostly quotes the pre­vi­ous post). Similarly, I some­times hear the coun­ter­ar­gu­ment that there will be eco­nomic pres­sures to­wards goal-di­rected AI, even though this po­si­tion is com­pat­i­ble with the post and ad­dressed in the next post. I’m not sure how to solve this though, with­out just hav­ing both the pre­vi­ous and next posts ap­pended to this post. (Part of the prob­lem is that differ­ent peo­ple have differ­ent re­sponses to the post, so it’s hard to ad­dress all of them with­out adding a ton of words.)

• rohinmshah 11 Dec 2019 8:16 UTC

  8 points

  on: What are the best ar­gu­ments and/​or plans for do­ing work in “AI policy”?

  If we don’t have such a clear dis­tinc­tion, then there’s not much that we can do, ex­cept ban AI, or ML en­tirely (or maybe ban AI above a cer­tain com­pute thresh­old, or op­ti­miza­tion thresh­old), which seems like a non-starter.

  Idk, if hu­man­ity as a whole could have a jus­tified 90% con­fi­dence that AI above a cer­tain com­pute thresh­old would kill us all, I think we could ban it en­tirely. Like, why on earth not? It’s in ev­ery­body’s in­ter­est to do so. (Note that this is not the case with cli­mate change, where it is in ev­ery­one’s in­ter­est for them to keep emit­ting while oth­ers stop emit­ting.)

  This seems prob­a­bly true even if it was 90% con­fi­dence that there is some thresh­old over which AI would kill us all, that we don’t yet know. In this case I imag­ine some­thing more like a di­rect ban on most peo­ple do­ing it, and some re­search that very care­fully ex­plores what the thresh­old is.

  This is only any use at all if gov­ern­ments can eas­ily iden­tify tractable re­search pro­grams that ac­tu­ally con­tribute to AI safety, in­stead of have “AI safety” as a cool tagline. I guess that you imag­ine that that will be the case in the fu­ture? Or maybe you think that it doesn’t mat­ter if they fund a bunch of ter­rible, pointless re­search if some “real” re­search also gets funded?

  A com­mon way in which this is done is to get ex­perts to help al­lo­cate fund­ing, which seems like a rea­son­able way to do this, and prob­a­bly bet­ter than the cur­rent mechanisms ex­cept­ing Open Phil (cur­rent mechanism = how well you can con­vince ran­dom donors to give you money).

  What? It seems like this is only pos­si­ble if the tech­ni­cal prob­lem is solved and known to be solved. At that point, the prob­lem is solved

  In the world where the al­igned ver­sion is not com­pet­i­tive, a gov­ern­ment can unilat­er­ally pay the price of not be­ing com­pet­i­tive be­cause it has many more re­sources.

  Also there are other prob­lems you might care about, like how the AI sys­tem might be used. You may not be too happy if any­one can “buy” a su­per­in­tel­li­gent AI from the com­pany that built it; this makes ar­bi­trary hu­mans gen­er­ally more able to im­pact the world; if you have a group of not-very-al­igned agents mak­ing big changes to the world and pos­si­bly fight­ing with each other, things will plau­si­bly go badly at some point.

  Again, if there are ex­ist­ing, leg­ible stan­dards of what’s safe and what isn’t this seems good. But with­out such stan­dards I don’t know how this helps?

  Tel­ling what is /​ isn’t safe seems de­cid­edly eas­ier than mak­ing an ar­bi­trary agent safe; it feels like we will be able to be con­ser­va­tive about this. But this is mostly an in­tu­ition.

  
  

  I think a gen­eral re­sponse to your in­tu­ition is that I don’t see tech­ni­cal solu­tions as the only op­tions; there are other ways we could be safe (1, 2).

  Cruxes:

  • We’re go­ing to have clear, leg­ible things that en­sure safety (which might be “never build sys­tems of this type”).

  • Govern­ments are much more com­pe­tent than you cur­rently be­lieve (I don’t know what you be­lieve, but prob­a­bly I think they are more com­pe­tent than you do)

  • We have so lit­tle ev­i­dence /​ ar­gu­ment so far, that just the model un­cer­tainty means that we can’t con­clude “it is unim­por­tant to think about how we could use the re­sources of the most pow­er­ful ac­tors in the world”.

• rohinmshah 10 Dec 2019 9:25 UTC

  LW: 4 AF: 3

  AF

  on: The Para­dox of Robustness

  Btw, as a meta-point, my un­der­stand­ing of your key claim is:

  Some­times, get­ting more of one nec­es­sar­ily means get­ting less of the other. Hence, the “para­dox.”

  My im­pres­sion af­ter read­ing your com­ment is that you’re ac­tu­ally say­ing that if you op­ti­mize for one, the other one might go down, which I cer­tainly agree with, but for a much sim­pler rea­son: in gen­eral, if you make a change that isn’t tar­geted at a vari­able Y, then that change is roughly equally likely to in­crease or de­crease Y.

  When we de­scribe the be­hav­ior of a sys­tem, we typ­i­cally op­er­ate at vary­ing lev­els of ab­strac­tion. I’m not mak­ing an ar­gu­ment about the frag­ility of the sub­strate that the sys­tem is on, but rather the frag­ility of the parts that we typ­i­cally use to de­scribe the sys­tem at an ap­pro­pri­ate level of ab­strac­tion.

  When we de­scribe the func­tion­al­ity of an ar­tifi­cial neu­ral net­work, we tend to speak about model weights and com­pu­ta­tional graphs, which do tol­er­ate slight mod­ifi­ca­tions. On the other hand, when we de­scribe the func­tion­al­ity of A* search, we tend to speak about sin­gle lines of code that do stuff, which gen­er­ally don’t tol­er­ate slight mod­ifi­ca­tions.

  This seems to be a fact about which pro­gram­ming lan­guage you choose, as op­posed to what al­gorithm you’re us­ing. I could in the­ory im­ple­ment A* in neu­ral net weights by hand, and then it would be ro­bust. Similarly, I could write out a learned neu­ral net in Python (one line of Python for ev­ery flop in the model), and it would no longer be ro­bust.

  (I think more broadly the ro­bust­ness you’re iden­ti­fy­ing in­volves mak­ing “small” changes where “small” is defined in terms of a “dis­tance” defined by the pro­gram­ming lan­guage; I want a “dis­tance” defined on al­gorithms, be­cause that seems more rele­vant to talk­ing about prop­er­ties of AI. That dis­tance should not de­pend on what pro­gram­ming lan­guage you use to im­ple­ment the al­gorithm.)

  Con­sider the scalar defi­ni­tion of ro­bust­ness: how well did you do dur­ing your perfor­mance off some train­ing dis­tri­bu­tion? In this case, many hu­mans are brit­tle, since they are not do­ing well ac­cord­ing to in­clu­sive fit­ness. Even within their own lives, hu­mans don’t pur­sue the goals they set for them­selves 10 years ago. There’s a lot of ways in which hu­mans are brit­tle in this sense.

  I claim the neu­ral net of your ex­am­ple wouldn’t be brit­tle in this way, since you pos­tu­lated that it was trained on the ac­tual dis­tri­bu­tion of en­vi­ron­ments it would en­counter.

  I’m not sure I un­der­stand the differ­ence be­tween a log­i­cal agent en­coun­ter­ing a sud­den, un­pre­dictable change to its en­vi­ron­ment and a log­i­cal agent en­ter­ing a regime where its op­er­at­ing as­sump­tions turned out to be false.

  A log­i­cal agent for solv­ing mazes could be an agent that fol­lows a wall. If you de­ploy such an agent in a maze with a loop, then it can cir­cle for­ever. It seems like a type er­ror to call this a sud­den, un­pre­dictable change—I wouldn’t re­ally as­cribe be­liefs to this agent at all.

• rohinmshah 10 Dec 2019 7:05 UTC

  LW: 10 AF: 7

  AF

  on: The Para­dox of Robustness

  I don’t un­der­stand this post. Some con­fu­sions:

  Given a hard-coded agent that ex­plic­itly com­puted the con­se­quences of its ac­tions, and then took the ac­tion which max­i­mized ex­pected value ac­cord­ing to its util­ity func­tion, we would ob­serve pre­cisely the op­po­site be­hav­ior. Mu­tate a sin­gle line of code and the func­tion­al­ity of this agent would al­most cer­tainly be bro­ken. How­ever, the agent is still ro­bust in the al­ign­ment sense, as its val­ues will never drift.

  Isn’t this true of ev­ery com­puter pro­gram? This sounds like an ar­gu­ment that AI can never be ro­bust in func­tion­al­ity, which seems to prove too much. (If you ac­tu­ally mean this, I think your use of the word “ro­bust” has di­verged from the prop­erty I care about.)

  Log­i­cal sys­tems can be ro­bust be­cause their be­hav­ior is very pre­dictable off-dis­tri­bu­tion, but they are not always ro­bust in the sense of be­ing able to adapt to sud­den, un­pre­dictable changes.

  I would like to see an ex­am­ple of be­ing un­able to adapt to sud­den, un­pre­dictable changes; that doesn’t match my ex­pla­na­tion of why log­i­cal sys­tems fail: I would say that they make as­sump­tions that turn out to be false, with a par­tic­u­larly com­mon case be­ing the failure of the de­signer to con­sider some par­tic­u­lar edge case, and there are enough edge cases that these failures be­come too com­mon.

  they must look some­thing like a mesa op­ti­mizer. Thus, if the cur­rent learn­ing paradigm is to cre­ate gen­eral in­tel­li­gence, we will nec­es­sar­ily en­counter prob­lems en­demic to the “old” type of AI.

  In the pre­vi­ous para­graph, I thought you ar­gued that log­i­cal /​ “old” AI has ro­bust speci­fi­ca­tion but not ro­bust func­tion­al­ity. But the worry with mesa op­ti­miz­ers is the ex­act op­po­site; that they will have ro­bust func­tion­al­ity but not ro­bust speci­fi­ca­tion.

  We might ex­pect it to perform ex­tremely well and be quite ro­bust to changes in its en­vi­ron­ment, but be brit­tle in the sense of hav­ing no ex­plicit prin­ci­ples which un­der­lie its pur­suit of goals.

  What does “brit­tle” mean here? You can say the same of hu­mans; are hu­mans “brit­tle”?

• rohinmshah 9 Dec 2019 8:57 UTC

  7 points

  on: What are the best ar­gu­ments and/​or plans for do­ing work in “AI policy”?

  What? I feel like I must be mi­s­un­der­stand­ing, be­cause it seems like there are broad cat­e­gories of things that gov­ern­ments can do that are helpful, even if you’re only wor­ried about the risk of an AI op­ti­miz­ing against you. I guess I’ll just list some, and you can tell me why none of these work:

  • Fund­ing safety research

  • Build­ing al­igned AIs themselves

  • Creat­ing laws that pre­vent races to the bot­tom be­tween com­pa­nies (e.g. “no AI with >X com­pute may be de­ployed with­out first con­duct­ing a com­pre­hen­sive re­view of the chance of the AI ad­ver­sar­i­ally op­ti­miz­ing against hu­man­ity”)

  • Mon­i­tor­ing AI sys­tems (e.g. “we will cre­ate a board of AI in­ves­ti­ga­tors; ev­ery­one mak­ing pow­er­ful AI sys­tems must be eval­u­ated once a year”)

  I don’t think there’s a con­crete plan that I would want a gov­ern­ment to start on to­day, but I’d be sur­prised if there weren’t such plans in the fu­ture when we know more (both from more re­search, and the AI risk prob­lem is clearer).

  You can also look at the pa­pers un­der the cat­e­gory “AI strat­egy and policy” in the Align­ment Newslet­ter database.

• rohinmshah 8 Dec 2019 5:55 UTC

  2 points

  on: [AN #76]: How dataset size af­fects ro­bust­ness, and bench­mark­ing safe ex­plo­ra­tion by mea­sur­ing con­straint violations

  Thanks!

• rohinmshah 8 Dec 2019 5:50 UTC

  10 points

  on: Com­ment on Co­her­ence ar­gu­ments do not im­ply goal di­rected behavior

  I think this for­mu­la­tion of goal-di­rect­ed­ness is pretty similar to one I sug­gested in the post be­fore the co­her­ence ar­gu­ments post (In­tu­itions about goal-di­rected be­hav­ior, sec­tion “Our un­der­stand­ing of the be­hav­ior”). I do think this is an im­por­tant con­cept to ex­plain our con­cep­tion of goal-di­rect­ed­ness, but I don’t think it can be used as an ar­gu­ment for AI risk, be­cause it proves too much. For ex­am­ple, for many peo­ple with­out tech­ni­cal ex­per­tise, the best model they have for a lap­top is that it is pur­su­ing some goal (at least, many of my rel­a­tives fre­quently an­thro­po­mor­phize their lap­tops). Should they worry that their lap­tops are go­ing to take over the world?

  For a deeper re­sponse, I’d recom­mend In­tu­itions about goal-di­rected be­hav­ior. I’ll quote some of the rele­vant parts here:

  There is a gen­eral pat­tern in which as soon as we un­der­stand some­thing, it be­comes some­thing lesser. As soon as we un­der­stand rain­bows, they are rel­e­gated to the “dull cat­a­logue of com­mon things”. This sug­gests a some­what cyn­i­cal ex­pla­na­tion of our con­cept of “in­tel­li­gence”: an agent is con­sid­ered in­tel­li­gent if we do not know how to achieve the out­comes it does us­ing the re­sources that it has (in which case our best model for that agent may be that it is pur­su­ing some goal, re­flect­ing our ten­dency to an­thro­po­mor­phize). That is, our eval­u­a­tion about in­tel­li­gence is a state­ment about our epistemic state.

  [… four ex­am­ples …]

  To the ex­tent that the Misspeci­fied Goal ar­gu­ment re­lies on this in­tu­ition, the ar­gu­ment feels a lot weaker to me. If the Misspeci­fied Goal ar­gu­ment rested en­tirely upon this in­tu­ition, then it would be as­sert­ing that be­cause we are ig­no­rant about what an in­tel­li­gent agent would do, we should as­sume that it is op­ti­miz­ing a goal, which means that it is go­ing to ac­cu­mu­late power and re­sources and lead to catas­tro­phe. In other words, it is ar­gu­ing that as­sum­ing that an agent is in­tel­li­gent defi­ni­tion­ally means that it will ac­cu­mu­late power and re­sources. This seems clearly wrong; it is pos­si­ble in prin­ci­ple to have an in­tel­li­gent agent that nonethe­less does not ac­cu­mu­late power and re­sources.

  Also, the ar­gu­ment is not say­ing that in prac­tice most in­tel­li­gent agents ac­cu­mu­late power and re­sources. It says that we have no bet­ter model to go off of other than “goal-di­rected”, and then pushes this model to ex­treme sce­nar­ios where we should have a lot more un­cer­tainty.

  See also the sum­mary of that post:

  “From the out­side”, it seems like a goal-di­rected agent is char­ac­ter­ized by the fact that we can pre­dict the agent’s be­hav­ior in new situ­a­tions by as­sum­ing that it is pur­su­ing some goal, and as a re­sult it is ac­quires power and re­sources. This can be in­ter­preted ei­ther as a state­ment about our epistemic state (we know so lit­tle about the agent that our best model is that it pur­sues a goal, even though this model is not very ac­cu­rate or pre­cise) or as a state­ment about the agent (pre­dict­ing the be­hav­ior of the agent in new situ­a­tions based on pur­suit of a goal ac­tu­ally has very high pre­ci­sion and ac­cu­racy). Th­ese two views have very differ­ent im­pli­ca­tions on the val­idity of the Misspeci­fied Goal ar­gu­ment for AI risk.

  What links here?

  • rohinmshah's comment on Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior by rohinmshah (13 Dec 2019 0:51 UTC; 18 points)

• rohinmshah 8 Dec 2019 5:20 UTC

  LW: 4 AF: 3

  AF

  on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  I pretty strongly agree with this re­view (and jtbc it was writ­ten with­out any in­put from me, even though Daniel and I are both at CHAI).

  I think of ‘co­her­ence ar­gu­ments’ as in­clud­ing things like ‘it’s not pos­si­ble for you to agree to give me a limitless num­ber of dol­lars in re­turn for noth­ing’, which does im­ply some de­gree of ‘goal-di­rec­tion’.

  Yeah, maybe I should say “co­her­ence the­o­rems” to be clearer about this? (Like, it isn’t a the­o­rem that I shouldn’t give you limitless num­ber of dol­lars in re­turn for noth­ing; maybe I think that you are more ca­pa­ble than me and fully al­igned with me, and so you’d do a bet­ter job with my money. Or maybe I value your hap­piness, and the best way to pur­chase it is to give you money no strings at­tached.)

  Re­sponses from out­side this camp

  Fwiw, I do in fact worry about goal-di­rect­ed­ness, but (I think) I know what you mean. (For oth­ers, I think Daniel is refer­ring to some­thing like “the MIRI camp”, though that is also not an ac­cu­rate poin­ter, and it is true that I am out­side that camp.)

  My re­sponses to the ques­tions:

  1. The ones in Will hu­mans build goal-di­rected agents?, but if you want ar­gu­ments that aren’t about hu­mans, then I don’t know.

  2. Depends on the dis­tri­bu­tion over util­ity func­tions, the ac­tion space, etc, but e.g. if it uniformly se­lects a nu­meric re­ward value for each pos­si­ble tra­jec­tory (state-ac­tion se­quence) where the ac­tions are low-level (e.g. hu­man mus­cle con­trol), as­tro­nom­i­cally low.

  3. That will prob­a­bly be a good model for some (many?) pow­er­ful AI sys­tems that hu­mans build.

  4. I don’t know. (I think it de­pends quite strongly on the way in which we train pow­er­ful AI sys­tems.)

  5. Not likely at low lev­els of in­tel­li­gence, plau­si­ble at higher lev­els of in­tel­li­gence, but re­ally the ques­tion is not speci­fied enough.

• rohinmshah 5 Dec 2019 17:07 UTC

  LW: 3 AF: 2

  AF

  on: [AN #76]: How dataset size af­fects ro­bust­ness, and bench­mark­ing safe ex­plo­ra­tion by mea­sur­ing con­straint violations

  It goes un­der many names, such as trans­fer learn­ing, ro­bust­ness to dis­tri­bu­tional shift /​ data shift, and out-of-dis­tri­bu­tion gen­er­al­iza­tion. Each one has (to me) slightly differ­ent con­no­ta­tions, e.g. trans­fer learn­ing sug­gests that the re­searcher has a clear idea of the dis­tinc­tion be­tween the first and sec­ond set­ting (and so you “trans­fer” from the first to the sec­ond), whereas if in RL you change which part of the state space you’re in as you act, I would be more likely to call that dis­tri­bu­tional shift rather than trans­fer learn­ing.

• rohinmshah 2 Dec 2019 23:35 UTC

  LW: 6 AF: 4

  AF

  on: Ver­ifi­ca­tion and Transparency

  Planned newslet­ter sum­mary:

  This post points out that ver­ifi­ca­tion and trans­parency have similar goals. Trans­parency pro­duces an arte­fact that al­lows the user to an­swer ques­tions about the sys­tem un­der in­ves­ti­ga­tion (e.g. “why did the neu­ral net pre­dict that this was a ten­nis ball?”). Ver­ifi­ca­tion on the other hand al­lows the user to pose a ques­tion, and then au­to­mat­i­cally an­swers that ques­tion (e.g. “is there an ad­ver­sar­ial ex­am­ple for this image?”).

• rohinmshah 2 Dec 2019 4:18 UTC

  LW: 4 AF: 3

  AF

  on: Clar­ify­ing “AI Align­ment”

  Crys­tal­lized my view of what the “core prob­lem” is (as I ex­plained in a com­ment on this post). I think I had in­tu­itions of this form be­fore, but at the very least this post clar­ified them.

• rohinmshah 2 Dec 2019 3:57 UTC

  2 points

  on: Ar­gu­ments about fast takeoff

  Thanks, nom­i­nated!

• rohinmshah 2 Dec 2019 3:56 UTC

  5 points

  on: Will AI See Sud­den Progress?

  Up­dated me quite strongly to­wards con­tin­u­ous take­off (from a po­si­tion of ig­no­rance)

• rohinmshah 2 Dec 2019 2:50 UTC

  2 points

  on: Ar­gu­ments about fast takeoff

  Up­dated me quite strongly to­wards con­tin­u­ous take­off (from a po­si­tion of ig­no­rance). (I would also nom­i­nate the AI Im­pacts post, but I don’t think it ever got cross-posted.)

• rohinmshah 1 Dec 2019 18:08 UTC

  6 points

  on: Use­ful Does Not Mean Secure

  Tbc, I wasn’t mod­el­ing Eliezer /​ Nate /​ MIRI as say­ing “there will be pow­er­ful ad­ver­sar­ial op­ti­miza­tion, and so we need se­cu­rity”—it is in fact quite clear that we’re all aiming for “no pow­er­ful ad­ver­sar­ial op­ti­miza­tion in the first place”. I was re­spond­ing to the ar­gu­ments in this post.

  (and if he sus­pected this was easy, he wouldn’t want to build in the as­sump­tion that it’s easy as a pre­req­ui­site for a safety ap­proach work­ing; he’d want a safety ap­proach that’s ro­bust to the sce­nario where it’s easy to ac­ci­den­tally end up with vastly more qual­ity-ad­justed op­ti­miza­tion than in­tended).

  I agree that’s de­sir­able all else equal, but such an ap­proach would likely re­quire more time and effort (a lot more time and effort on my model). It’s an em­piri­cal ques­tion whether we have that time /​ effort to spare (and also per­haps it’s bet­ter to get AGI ear­lier to e.g. re­duce other x-risks, in which case the marginal safety from not rely­ing on the as­sump­tion may not be worth it).

  (I men­tioned co­or­di­na­tion on not build­ing AGI above—I think that might be fea­si­ble if the “global epistemic state” was that build­ing AGI is likely to kill us all, but seems quite in­fea­si­ble if our epistemic state is “ev­ery­thing we know sug­gests this will work, but it could fail if we some­how end up with more op­ti­miza­tion”.)

  Note that on my model, the kind of para­noia Eliezer is point­ing to with “AI safety mind­set” or se­cu­rity mind­set is some­thing he be­lieves you need in or­der to pre­vent ad­ver­sar­i­al­ness and the other bad byprod­ucts of “your sys­tem de­votes large amounts of thought to things and thinks in re­ally weird ways”.

  [...]

  The par­allel to cryp­tog­ra­phy is that in AI al­ign­ment we deal with sys­tems that perform in­tel­li­gent searches through a very large search space, and which can pro­duce weird con­texts that force the code down un­ex­pected paths. This is be­cause the weird edge cases are places of ex­tremes, and places of ex­tremes are of­ten the place where a given ob­jec­tive func­tion is op­ti­mized. Like com­puter se­cu­rity pro­fes­sion­als, AI al­ign­ment re­searchers need to be very good at think­ing about edge cases.

  It’s much eas­ier to make code that works well on the path that you were vi­su­al­iz­ing than to make code that works on all the paths that you weren’t vi­su­al­iz­ing. AI al­ign­ment needs to work on all the paths you weren’t vi­su­al­iz­ing.

  [...]

  The prob­lem comes from “lots of weird, ex­treme-state-in­stan­ti­at­ing, loop­hole-find­ing op­ti­miza­tion”

  Thanks, this is helpful for un­der­stand­ing MIRI’s po­si­tion bet­ter. (I prob­a­bly should have figured it out from Nate and Scott’s posts, but I don’t think I ac­tu­ally did.)

  Broadly, my hope is that we ac­tu­ally see non-ex­is­ten­tially-catas­trophic failures caused by AIs go­ing down “the paths you weren’t vi­su­al­iz­ing”, and this causes you to start vi­su­al­iz­ing the path. Ob­vi­ously all else equal it’s bet­ter if we vi­su­al­ize it in the first place.

  I think I also have a differ­ent pic­ture of what pow­er­ful op­ti­miza­tion will look like—the pa­per­clip max­i­mizer doesn’t seem like a good model for the sort of thing we’re likely to build. An ap­proach based on some ex­plic­itly rep­re­sented goal is go­ing to be dead in the wa­ter well be­fore it be­comes even hu­man-level in­tel­li­gent, be­cause it will ig­nore “com­mon sense rules” again and again (c.f. most speci­fi­ca­tion gam­ing ex­am­ples). In­stead, our AI sys­tems are go­ing to need to un­der­stand com­mon sense rules some­how, and the re­sult­ing sys­tem is not go­ing to look like it’s ruth­lessly pur­su­ing some sim­ple goal.

  For ex­am­ple, the re­sult­ing sys­tem may be more ac­cu­rately mod­eled as hav­ing un­cer­tainty about the goal (whether ex­plic­itly rep­re­sented or in­ter­nally learned). Weird + ex­treme states tend to only be good for a few goals, and so would not be good choices if you’re un­cer­tain about the goal. In ad­di­tion, if our AI sys­tems are learn­ing from our con­ven­tions, then they will likely pick up our risk aver­sion, which also tends to pre­vent weird + ex­treme states.

  Fi­nally, it seems like there’s a broad basin of cor­rigi­bil­ity, that pre­vents weird + ex­treme states that hu­mans would rate as bad. It’s not hard to figure out that hu­mans don’t want to die, so any weird + ex­treme state you cre­ate has to re­spect that con­straint. And there are many other such easy-to-learn con­straints.

• rohinmshah 1 Dec 2019 1:39 UTC

  LW: 6 AF: 4

  AF

  on: Use­ful Does Not Mean Secure

  Planned newslet­ter sum­mary:

  Re­cently, I sug­gested the fol­low­ing broad model: The way you build things that are use­ful and do what you want is to un­der­stand how things work and put them to­gether in a de­liber­ate way. If you put things to­gether ran­domly, they ei­ther won’t work, or will have un­in­tended side effects. Un­der this model, rel­a­tive to do­ing noth­ing, it is net pos­i­tive to im­prove our un­der­stand­ing of AI sys­tems, e.g. via trans­parency tools, even if it means we build pow­er­ful AI sys­tems sooner (which re­duces the time we have to solve al­ign­ment).

  This post pre­sents a coun­ter­ar­gu­ment: while un­der­stand­ing helps us make _use­ful_ sys­tems, it need not help us build _se­cure_ sys­tems. We need se­cu­rity be­cause that is the only way to get use­ful sys­tems in the pres­ence of pow­er­ful ex­ter­nal op­ti­miza­tion, and the whole point of AGI is to build sys­tems that are more pow­er­ful op­ti­miz­ers than we are. If you take an already-use­ful AI sys­tem, and you “make it more pow­er­ful”, this in­creases the in­tel­li­gence of both the use­ful parts and the ad­ver­sar­ial parts. At this point, the main point of failure is if the ad­ver­sar­ial parts “win”: you now have to be ro­bust against ad­ver­saries, which is a se­cu­rity prop­erty, not a use­ful­ness prop­erty.

  Un­der this model, trans­parency work need not be helpful: if the trans­parency tools al­low you to de­tect some kinds of bad cog­ni­tion but not oth­ers, an ad­ver­sary sim­ply makes sure that all of its ad­ver­sar­ial cog­ni­tion is the kind you can’t de­tect. Ro­hin’s note: Or, if you use your trans­parency tools dur­ing train­ing, you are se­lect­ing for mod­els whose ad­ver­sar­ial cog­ni­tion is the kind you can’t de­tect. Then, trans­parency tools could in­crease un­der­stand­ing and shorten the time to pow­er­ful AI sys­tems, _with­out_ im­prov­ing se­cu­rity.

  Planned opinion:

  I cer­tainly agree that in the pres­ence of pow­er­ful ad­ver­sar­ial op­ti­miz­ers, you need se­cu­rity to get your sys­tem to do what you want. How­ever, we can just not build pow­er­ful ad­ver­sar­ial op­ti­miz­ers. My preferred solu­tion is to make sure our AI sys­tems are try­ing to do what we want , so that they never be­come ad­ver­sar­ial in the first place. But if for some rea­son we can’t do that, then we could make sure AI sys­tems don’t be­come too pow­er­ful, or not build them at all. It seems very weird to in­stead say “well, the AI sys­tem is go­ing to be ad­ver­sar­ial and way more pow­er­ful, let’s figure out how to make it se­cure”—that should be the last ap­proach, if none of the other ap­proaches work out. (More de­tails in this com­ment.) Note that MIRI doesn’t aim for se­cu­rity be­cause they ex­pect pow­er­ful ad­ver­sar­ial op­ti­miza­tion—they aim for se­cu­rity be­cause _any_ op­ti­miza­tion <@leads to ex­treme out­comes@>(@Op­ti­miza­tion Am­plifies@). (More de­tails in this com­ment.)

  (If you want to com­ment on my opinion, please do so as a re­ply to the other com­ment I made.)

  ETA: Added a sen­tence about MIRI’s be­liefs to the opinion.

  What links here?

  • Rob Bensinger's comment on Use­ful Does Not Mean Secure by Ben Pace (1 Dec 2019 7:41 UTC; 15 points)

• rohinmshah 30 Nov 2019 5:44 UTC

  LW: 11 AF: 8

  AF

  on: Use­ful Does Not Mean Secure

  they ex­pect the field to en­gage with this prob­lem and do it well.

  It’s a lit­tle more dis­junc­tive:

  • Maybe the prob­lem is too difficult. Then we could co­or­di­nate to avoid the prob­lem. This could mean not build­ing pow­er­ful AI sys­tems at all, limit­ing the types of AI sys­tems we build, etc.

  • Maybe it’s not ac­tu­ally a prob­lem. Hu­mans seem kinda sorta goal-di­rected, and fre­quently face ex­is­ten­tial angst over what the mean­ing of life is. Maybe pow­er­ful AI sys­tems will similarly be very ca­pa­ble but not have ex­plicit goals.

  • Maybe there isn’t differ­en­tially pow­er­ful op­ti­miza­tion. We build some­what-smarter-than-hu­man AI sys­tems, and these AI sys­tems en­able us to be­come more ca­pa­ble our­selves (just as the In­ter­net has made us more ca­pa­ble), and our ca­pa­bil­ities in­crease alongside the AI’s ca­pa­bil­ities, and there’s never any op­ti­miza­tion that’s way more pow­er­ful than us.

  • Maybe there isn’t pow­er­ful ad­ver­sar­ial op­ti­miza­tion. We figure out how to solve the al­ign­ment prob­lem for small in­creases in in­tel­li­gence; we use this to de­sign the first smarter-than-hu­man sys­tems, they use it to de­sign their suc­ces­sors, etc. to ar­bi­trary lev­els of ca­pa­bil­ities.

  • Maybe hacks are enough. Every time we no­tice a prob­lem, we just ap­ply a “band-aid” fix, noth­ing that seems prin­ci­pled. But this turns out to be enough. (I don’t like this ar­gu­ment, be­cause it’s un­clear what a “band-aid” fix is—for some defi­ni­tions of “band-aid” I’d feel con­fi­dent that “band-aids” would not be enough. But there’s some­thing along these lines.)

  Yann is im­plic­itly tak­ing the stance that there will not be pow­er­ful ad­ver­sar­ial pres­sures ex­ploit­ing such un­fore­seen differ­ences in the ob­jec­tive func­tion and hu­man­ity’s val­ues.

  Maybe, I’m not sure. Re­gard­less of the un­der­ly­ing ex­pla­na­tion, if ev­ery­one sounded like Yann I’d be more wor­ried. (ETA: Well, re­ally I’d spend a bunch of time eval­u­at­ing the ar­gu­ment more deeply, and form a new opinion, but as­sum­ing I did that and found the ar­gu­ment un­con­vinc­ing, then I’d be more wor­ried.)

  As you take the sys­tem and make it vastly su­per­in­tel­li­gent, your pri­mary fo­cus needs to be on se­cu­rity from ad­ver­sar­ial forces, rather than pri­mar­ily on mak­ing some­thing that’s use­ful.

  I agree if you as­sume a dis­crete ac­tion that sim­ply causes the sys­tem to be­come vastly su­per­in­tel­li­gent. But we can try not to get to pow­er­ful ad­ver­sar­ial op­ti­miza­tion in the first place; if that never hap­pens then you never need the se­cu­rity. (As a re­cent ex­am­ple, re­laxed ad­ver­sar­ial train­ing takes ad­van­tage of this fact.) In the pre­vi­ous list, bul­let points 3 and 4 are ex­plic­itly about avoid­ing pow­er­ful ad­ver­sar­ial op­ti­miza­tion, and bul­let points 1 and 2 are about notic­ing whether or not we have to worry about pow­er­ful ad­ver­sar­ial op­ti­miza­tion and deal­ing with it if so. (Meta: Can we get good num­bered lists? If we have them, how do I make them?)

  Given how difficult se­cu­rity is, it seems bet­ter to aim for one of those sce­nar­ios. In prac­tice, I do think any plan that in­volves build­ing pow­er­ful AI sys­tems will re­quire some amount of se­cu­rity-like thought—for ex­am­ple, if you’re hop­ing to de­tect ad­ver­sar­ial op­ti­miza­tion to stop it from aris­ing, you need a lot of con­fi­dence in the de­tec­tor. But there isn’t liter­ally strong ad­ver­sar­ial op­ti­miza­tion work­ing against the de­tec­tor—it’s more that if there’s a “ran­dom” failure, that turns into ad­ver­sar­ial op­ti­miza­tion, and so it be­comes hard to cor­rect the failure. So it seems more ac­cu­rate to say that we need very low rates of failure—but in the ab­sence of ad­ver­sar­ial op­ti­miza­tion.

  (Btw, this en­tire com­ment is pred­i­cated on con­tin­u­ous take­off; if I were con­vinced of dis­con­tin­u­ous take­off I’d ex­pect my be­liefs to change rad­i­cally, and to be much less op­ti­mistic.)

  What links here?

  • [AN #76]: How dataset size af­fects ro­bust­ness, and bench­mark­ing safe ex­plo­ra­tion by mea­sur­ing con­straint violations by rohinmshah (4 Dec 2019 18:10 UTC; 13 points)

• rohinmshah 30 Nov 2019 0:54 UTC

  LW: 4 AF: 3

  AF

  on: Utility un­cer­tainty vs. ex­pected in­for­ma­tion gain

  Fwiw, I talk about util­ity un­cer­tainty be­cause that’s what the me­chan­i­cal change to the code looks like—in­stead of hav­ing a known re­ward func­tion, you have a dis­tri­bu­tion over re­ward func­tions. It’s cer­tainly true that this only makes a differ­ence as long as there is still in­for­ma­tion to be gained about the util­ity func­tion.

• rohinmshah 30 Nov 2019 0:45 UTC

  LW: 5 AF: 4

  AF

  on: What are we as­sum­ing about util­ity func­tions?

  the dis­cus­sion was whether those agents will be broadly goal-di­rected at all, a weaker con­di­tion than be­ing a util­ity maximizer

  Uh, that chap­ter was claiming that “be­ing a util­ity max­i­mizer” is vac­u­ous, and there­fore “goal-di­rected” is a stronger con­di­tion than “be­ing a util­ity max­i­mizer”.

• rohinmshah 27 Nov 2019 2:58 UTC

  4 points

  on: No, it’s not The In­cen­tives—it’s you

  Tbc, in the grand­par­ent I was re­spond­ing to the spe­cific sen­tence I quoted, which seems to me to be mak­ing a bold claim that I think is false. It’s of course pos­si­ble that the cor­rect ac­tion is still “leave academia”, but for a differ­ent rea­son, like the one you gave.

  Re:

  it should not be ‘joined’ but in­stead ‘quit, and get to work on an al­ter­na­tive’.

  That de­pends pretty strongly on what the al­ter­na­tive is. Sup­pose your goal is for more in­ves­ti­ga­tion of spec­u­la­tive ideas that may or may not pan out, so that hu­man­ity figures out true and use­ful things about the world. It’s not clear to me that you can do sig­nifi­cantly bet­ter than cur­rent academia, even if you as­sume that ev­ery­one will switch from academia to your new in­sti­tu­tion.

  And of course, peo­ple in academia are se­lected for be­ing good at aca­demic jobs, and may not be good at build­ing in­sti­tu­tions. Or they may hate all the politick­ing that would be re­quired for an al­ter­na­tive. Or they might not par­tic­u­larly care about im­pact, and just want to do re­search be­cause it’s fun. All of which are rea­sons you might join academia rather than quit and work on an al­ter­na­tive, and it’s “morally fine”.