evhub

What I’ll be do­ing at MIRI

• evhub 6 Nov 2019 18:54 UTC

  6 points

  on: [AN #72]: Align­ment, ro­bust­ness, method­ol­ogy, and sys­tem build­ing as re­search pri­ori­ties for AI safety

  Asya’s opinion on “Norms, Re­wards, and the In­ten­tional Stance” ap­pears to have ac­ci­den­tally been re­placed by Ro­hin’s opinion on the “Ought Progress Up­date.”

• evhub 5 Nov 2019 7:40 UTC

  LW: 10 AF: 6

  AF

  in reply to: johnswentworth's comment on: But ex­actly how com­plex and frag­ile?

  That may be the crux. I’m gen­er­ally of the mind­set that “can’t guaran­tee/​ver­ify” im­plies “com­pletely use­less for AI safety”. Ver­ify­ing that’s it’s safe is the whole point of AI safety re­search. If we were hop­ing to make some­thing that just hap­pened to be safe even though we couldn’t guaran­tee it be­fore­hand or dou­ble-check af­ter­wards, that would just be called “AI”

  Surely “the whole point of AI safety re­search” is just to save the world, no? If the world ends up be­ing saved, does it mat­ter whether we were able to “ver­ify” that or not? From my per­spec­tive, as a util­i­tar­ian, it seems to me that the only rele­vant ques­tion is how some par­tic­u­lar in­ter­ven­tion/​re­search/​etc. af­fects the prob­a­bil­ity of AI be­ing good for hu­man­ity (or the EV, to be pre­cise). It cer­tainly seems quite use­ful to be able to ver­ify lots of stuff to achieve that goal, but I think it’s worth be­ing clear that ver­ifi­ca­tion is an in­stru­men­tal goal not a ter­mi­nal one—and that there might be other pos­si­ble ways to achieve that ter­mi­nal goal (un­der­stand­ing em­piri­cal ques­tions, for ex­am­ple, as Ro­hin wanted to do in this thread). At the very least, I cer­tainly wouldn’t go around say­ing that ver­ifi­ca­tion is “the whole point of AI safety re­search.”

• evhub 5 Nov 2019 1:44 UTC

  LW: 8 AF: 6

  AF

  in reply to: rohinmshah's comment on: More vari­a­tions on pseudo-alignment

  Perfectly rea­son­able for you to not re­ply like you said, though I think it’s worth­while for me to at least clar­ify one point:

  I don’t think a com­pe­tent-at-hu­man-level sys­tem doesn’t know about de­cep­tion, and I don’t think a com­pe­tent-at-be­low-hu­man-level sys­tem can cause ex­tinc­tion-level catastrophe

  A model which sim­ply “doesn’t know about de­cep­tion” isn’t the only (or even the pri­mary) situ­a­tion I’m imag­in­ing. The ex­am­ple I gave in the post was a situ­a­tion in which the model hadn’t yet “figured out that de­cep­tion is a good strat­egy,” which could be:

  • be­cause it didn’t know about de­cep­tion,

  • be­cause it thought that de­cep­tion wouldn’t work,

  • be­cause it thought it was fully al­igned,

  • be­cause the train­ing pro­cess con­strained its thoughts such that it wasn’t able to even think about de­cep­tion,

  or some other rea­son. I don’t nec­es­sar­ily want to take a stand on which of these pos­si­bil­ities I think is the most likely, as I think that will vary de­pend­ing on the train­ing pro­cess. Rather, I want to point to the gen­eral prob­lem that a lot of these sorts of pos­si­bil­ities ex­ist such that, es­pe­cially if you ex­pect ad­ver­saries in the en­vi­ron­ment, I think it will be quite difficult to elimi­nate all of them.

  What links here?

  • rohinmshah's comment on More vari­a­tions on pseudo-alignment by evhub (5 Nov 2019 15:44 UTC; 5 points)

• evhub 4 Nov 2019 23:48 UTC

  LW: 20 AF: 12

  AF

  on: Will trans­parency help catch de­cep­tion? Per­haps not

  I agree that once you have a fully de­cep­tively al­igned model that’s plot­ting against you and try­ing to figure out how to fool your trans­parency tools, there’s not much hope left for you to be able to de­tect and rem­edy that is­sue. Im­por­tantly, how­ever, that’s not my model for how trans­parency could help you catch de­cep­tion. Rather, the idea is that by us­ing your trans­parency tools + over­seer to guide your train­ing pro­cess, you can pre­vent your train­ing pro­cess from ever en­ter­ing the regime where the model is try­ing to trick the trans­parency tools. This is es­pe­cially im­por­tant in the con­text of gra­di­ent hack­ing (as I men­tion in that post), since you have to in­spect the en­tire train­ing pro­cess if you want to know whether any gra­di­ent hack­ing oc­curred.

  Put an­other way: once you’re play­ing the game where I can hand you any model and then you have to figure out whether it’s de­cep­tive or not, you’ve already lost. In­stead, you want to be in the regime where your train­ing pro­cess is con­structed so as to steer clear of situ­a­tions in which your model might be­come de­cep­tive in the first place. I see en­forc­ing some­thing like cor­rigi­bil­ity or my­opia as a way of do­ing this: be­fore your model can be­come de­cep­tive, it has to be­come non-my­opic, which means if you can de­tect your model start­ing to be­come non-my­opic, then you can pre­vent it from ever be­com­ing de­cep­tive.

  From “Re­laxed ad­ver­sar­ial train­ing for in­ner al­ign­ment:”

  [For the re­cur­sion prob­lem,] the re­cur­sion doesn’t ac­tu­ally have to work for all $M\in M$—rather, we only need our ac­cept­abil­ity guaran­tee to hold along any path through model space that could ac­tu­ally be tra­versed by our train­ing pro­ce­dure.

  Thus, it’s fine if our pro­ce­dure fails if given a suffi­ciently un­ac­cept­able model, as long as our pro­ce­dure can pre­vent such mod­els from ever aris­ing. For ex­am­ple, if we always used an old ver­sion of $M$ that we were con­fi­dent in for im­ple­ment­ing $\text{Amp}\left(M\right)$, then we would just need it to be the case that $M$ couldn’t get to the point where it would be able to de­ceive $\text{Amp}\left(M_{\text{old}}\right)$ in-be­tween over­sight.

More vari­a­tions on pseudo-alignment

• evhub 3 Nov 2019 20:16 UTC

  LW: 18 AF: 9

  AF

  on: But ex­actly how com­plex and frag­ile?

  So then the largest re­main­ing worry is that it will still gain power fast and cor­rec­tion pro­cesses will be slow enough that its some­what mis­al­igned val­ues will be set in for­ever. But it isn’t ob­vi­ous to me that by that point it isn’t suffi­ciently well al­igned that we would rec­og­nize its fu­ture as a won­drous utopia, just not the very best won­drous utopia that we would have imag­ined if we had re­ally care­fully sat down and imag­ined utopias for thou­sands of years. This again seems like an em­piri­cal ques­tion of the scale of differ­ent effects, un­less there is a an ar­gu­ment that some effect will be to­tally over­whelming.

  I think this ar­gu­ment mostly holds in the case of proxy al­ign­ment, but fails in the case of de­cep­tive al­ign­ment. If a model is de­cep­tively al­igned, then I don’t think there is any rea­son we should ex­pect it to be only “some­what mis­al­igned”—once a mesa-op­ti­mizer be­comes de­cep­tive, there’s no longer op­ti­miza­tion pres­sure act­ing to keep its mesa-ob­jec­tive in line with the base, which means it could be to­tally off, not just slightly wrong. Ad­di­tion­ally, a de­cep­tively al­igned mesa-op­ti­mizer might be able to do things like gra­di­ent hack­ing to sig­nifi­cantly hin­der our cor­rec­tion pro­cesses.

  Also, I think it’s worth point­ing out that de­cep­tion doesn’t just hap­pen dur­ing train­ing: it’s also pos­si­ble for a non-de­cep­tive proxy al­igned mesa-op­ti­mizer to be­come de­cep­tive dur­ing de­ploy­ment, which could throw a huge wrench in your cor­rec­tion pro­cesses story. In par­tic­u­lar, non-my­opic proxy al­igned mesa-op­ti­miz­ers “want to be de­cep­tive” in the sense that, if pre­sented with the strat­egy of de­cep­tive al­ign­ment, they will choose to take it (this is a form of sub­op­ti­mal­ity al­ign­ment). This could be es­pe­cially con­cern­ing in the pres­ence of an ad­ver­sary in the en­vi­ron­ment (a com­peti­tor AI, for ex­am­ple) that is choos­ing its out­put to cause other AIs to be­have de­cep­tively.

  What links here?

  • More vari­a­tions on pseudo-alignment by evhub (4 Nov 2019 23:24 UTC; 20 points)

• evhub 3 Nov 2019 19:45 UTC

  LW: 3 AF: 2

  AF

  in reply to: steve2152's comment on: Chris Olah’s views on AGI safety

  To me, the im­por­tant safety fea­ture of “micro­scope AI” is that the AI is not mod­el­ing the down­stream con­se­quences of its out­puts (which au­to­mat­i­cally rules out ma­nipu­la­tion and de­ceit).

  As I men­tioned in this com­ment, not mod­el­ing the con­se­quences of its out­put is ac­tu­ally ex­actly what I want to get out of my­opia.

  For the lat­ter ques­tion, what is the user in­ter­face, “Use in­ter­pretabil­ity tools & vi­su­al­iza­tions on the world-model” seems about as good an an­swer as any, and I am very happy to have Chris and oth­ers try­ing to flesh out that vi­sion.

  Yep; me too!

  I hope that they don’t stop at fea­ture ex­trac­tion, but also pul­ling out the re­la­tion­ships (causal, com­po­si­tional, etc.) that we need to do coun­ter­fac­tual rea­son­ing, plan­ning etc., and even a “search through causal path­ways to get de­sired con­se­quences” in­ter­face.

  Chris (and the rest of Clar­ity) are definitely work­ing on stuff like this!

  un­su­per­vised (a.k.a. “self-su­per­vised”) learn­ing as ofer sug­gests seems awfully safe but is it re­ally?

  I gen­er­ally agree that un­su­per­vised learn­ing seems much safer than other ap­proaches (e.g. RL), though I also agree that there are still con­cerns. See for ex­am­ple Abram’s re­cent “The Parable of Pre­dict-O-Matic” and the rest of his Par­tial Agency se­quence.

• evhub 2 Nov 2019 20:00 UTC

  LW: 20 AF: 6

  AF

  in reply to: Raemon's comment on: Chris Olah’s views on AGI safety

  Yep, I think that’s a cor­rect sum­mary of the fi­nal point.

  The main coun­ter­point that comes to mind is a pos­si­ble world where “opaque AIs” just can’t ever achieve gen­eral in­tel­li­gence, but mod­er­ately well-thought-out AI de­signs can bridge the gap to “gen­eral in­tel­li­gence/​agency” with­out be­ing re­li­able enough to be al­igned.

  Well, we know it’s pos­si­ble to achieve gen­eral in­tel­li­gence via dumb black box search—evolu­tion did it—and we’ve got lots of ev­i­dence for cur­rent black box ap­proaches be­ing quite pow­er­ful. So it seems un­likely to me that we “just can’t ever achieve gen­eral in­tel­li­gence” with black box ap­proaches, though it could be that do­ing so is much more difficult than if you have more of an un­der­stand­ing.

  Also, ease of al­ign­ing a par­tic­u­lar AI de­sign is a rel­a­tive prop­erty, not an ab­solute one. When you say trans­par­ent ap­proaches might not be “re­li­able enough to be al­igned” you could mean that they’ll be just as likely likely as black box ap­proaches to be al­igned, less likely, or that they won’t be able to meet some bench­mark thresh­old prob­a­bil­ity of safety. I would guess that trans­parency will in­crease the prob­a­bil­ity of al­ign­ment rel­a­tive to not hav­ing it, though I would say that it’s un­clear cur­rently by how much.

  The way I gen­er­ally like to think about this is that there are many pos­si­ble roads we can take to get to AGI, with some be­ing more al­ignable and some be­ing less al­ignable and some be­ing shorter and some be­ing longer. Then, the ar­gu­ment here is that trans­parency re­search opens up ad­di­tional av­enues which are more al­ignable, but which may be shorter or longer. Even if they’re shorter, how­ever, since they’re more al­ignable the idea is that even if we end up tak­ing the fastest path with­out re­gards to safety, if you can make the fastest path available to us a safer one, then that’s a win.

Chris Olah’s views on AGI safety

• evhub 31 Oct 2019 6:15 UTC

  LW: 2 AF: 1

  AF

  in reply to: Grue_Slinky's comment on: Tech­ni­cal AGI safety re­search out­side AI

  Also rele­vant is Ge­offrey Irv­ing and Amanda Askell’s “AI Safety Needs So­cial Scien­tists.”

• evhub 30 Oct 2019 21:09 UTC

  4 points

  in reply to: Raemon's comment on: [Site Up­date] Sub­scrip­tions, Book­marks, & Pingbacks

  Looks like it’s fixed now; thanks!

• evhub 30 Oct 2019 19:07 UTC

  5 points

  in reply to: jimrandomh's comment on: [Site Up­date] Sub­scrip­tions, Book­marks, & Pingbacks

  Some­thing shows up now, but they all just say “Hope­ful­lyAnony­mous” in­stead of the ac­tual users I sub­scribed to.

• evhub 30 Oct 2019 5:36 UTC

  4 points

  on: [Site Up­date] Sub­scrip­tions, Book­marks, & Pingbacks

  When I go to man­age my sub­scrip­tions, it claims that I no longer have any—what hap­pened to all of my old sub­scrip­tions?

• evhub 29 Oct 2019 22:17 UTC

  LW: 2 AF: 1

  AF

  in reply to: rohinmshah's comment on: Im­pact mea­sure­ment and value-neu­tral­ity verification

  I could imag­ine that some peo­ple mis­tak­enly think that un­al­igned AI is ac­tu­ally al­igned and so build it, or that some mal­i­cious ac­tors build AI al­igned with them, and the strat­egy-steal­ing as­sump­tion means that this is ba­si­cally fine as long as they don’t start out with too many re­sources, but this doesn’t seem like the main­line sce­nario to worry about: it seems much more rele­vant whether we can al­ign AI or not.

  That’s not the sce­nario I’m think­ing about when I think about strat­egy-steal­ing. I men­tioned this a bit in this com­ment, but when I think about strat­egy-steal­ing I’m gen­er­ally think­ing about it as an al­ign­ment prop­erty that may or may not hold for a sin­gle AI: namely, the prop­erty that the AI is equally good at op­ti­miz­ing all of the differ­ent things we might want it to op­ti­mize. If this prop­erty doesn’t hold, then you get some­thing like Paul’s go­ing out with a whim­per where our easy-to-spec­ify val­ues win out over our other val­ues.

  Fur­ther­more, I agree with you that I gen­er­ally ex­pect ba­si­cally all early AGIs to have similar al­ign­ment prop­er­ties, though I think you push a lot un­der the rug when you say they’ll all ei­ther be “al­igned” or “un­al­igned.” In par­tic­u­lar, I gen­er­ally imag­ine pro­duc­ing an AGI that is cor­rigible in that it’s try­ing to do what you want, but isn’t nec­es­sar­ily fully al­igned in the sense of figur­ing out what you want for you. In such a case, it’s very im­por­tant that your AGI not be bet­ter at op­ti­miz­ing some of your val­ues over oth­ers, as that will shift the dis­tri­bu­tion of value/​re­sources/​etc. away from the real hu­man prefer­ence dis­tri­bu­tion that we want.

  Also, value-neu­tral­ity ver­ifi­ca­tion isn’t just about strat­egy-steal­ing: it’s also about in­ner al­ign­ment, since it could help you sep­a­rate op­ti­miza­tion pro­cesses from ob­jec­tives in a nat­u­ral way that makes it eas­ier to ver­ify al­ign­ment prop­er­ties (such as com­pat­i­bil­ity with strat­egy-steal­ing, but also pos­si­bly cor­rigi­bil­ity) on those ob­jects.

• evhub 29 Oct 2019 21:37 UTC

  LW: 5 AF: 4

  AF

  in reply to: rohinmshah's comment on: Gra­di­ent hacking

  Does the agent need to have the abil­ity to change the weights in the neu­ral net that it is im­ple­mented in? If so, how does it get that abil­ity?

  No, at least not in the way that I’m imag­in­ing this work­ing. In fact, I wouldn’t re­ally call that gra­di­ent-hack­ing any­more (maybe it’s just nor­mal hack­ing at that point?).

  For your other points, I agree that they seem like in­ter­est­ing di­rec­tions to poke on for figur­ing out whether some­thing like this works or not.

• evhub 29 Oct 2019 21:31 UTC

  LW: 4 AF: 3

  AF

  in reply to: rohinmshah's comment on: Are min­i­mal cir­cuits de­cep­tive?

  A cou­ple of things. First, a min­i­mal cir­cuit is not the same as a speed-prior-min­i­mal al­gorithm. Min­i­mal cir­cuits have to be min­i­mal in width + depth, so a GLUT would definitely lose out. Se­cond, even if you’re op­er­at­ing on pure speed, I think there are sets of tasks that are large enough that a GLUT won’t work. For ex­am­ple, con­sider the task of find­ing the min­i­mum of an ar­bi­trary con­vex func­tion. Cer­tainly for the in­finite set of all pos­si­ble con­vex func­tions (on the ra­tio­nals, say), I would be pretty sur­prised if some­thing like gra­di­ent de­scent weren’t the fastest way to do that. Even if you re­strict to only finitely many con­vex func­tions, if your set is large enough it still seems hard to do bet­ter than gra­di­ent de­scent, es­pe­cially since look­ing up the solu­tion in a huge GLUT could be quite ex­pen­sive (how do you do the lookup? a hash table? a bi­nary tree? I’m not sure if those would be good enough here).

• evhub 29 Oct 2019 21:19 UTC

  LW: 2 AF: 1

  AF

  in reply to: rohinmshah's comment on: Im­pact mea­sure­ment and value-neu­tral­ity verification

  I think this com­ment by Wei Dai does a good job of clar­ify­ing what’s go­ing on with the strat­egy-steal­ing as­sump­tion. I know Wei Dai was also con­fused about the pur­pose of the strat­egy-steal­ing as­sump­tion for a while un­til he wrote that com­ment.

• evhub 23 Oct 2019 23:50 UTC

  10 points

  in reply to: Said Achmiz's comment on: What are some un­pop­u­lar (non-nor­ma­tive) opinions that you hold?

  Sup­pose you were asked to briefly de­scribe what such a place (which would, by con­struc­tion, not be Less Wrong) would be like—what would you say?

  I’m a big fan of in-per­son con­ver­sa­tion. I think it’s en­tirely pos­si­ble to save the world with­out need­ing to be able to talk about any­thing you want on­line in a pub­lic fo­rum.

• evhub 23 Oct 2019 21:01 UTC

  21 points

  in reply to: Stuart Anderson's comment on: What are some un­pop­u­lar (non-nor­ma­tive) opinions that you hold?

  I just want it to be clear to any­body who might be read­ing this and won­der­ing what sorts of be­liefs peo­ple on LessWrong hold that I do not hold any of the be­liefs in the above post and that I find the be­liefs ex­pressed there to be both fac­tu­ally in­cor­rect and morally re­pug­nant.