rohinmshah(Rohin Shah)

• rohinmshah 3 Apr 2020 23:21 UTC

  3 points

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  Sup­pose you have two events X and Y, such that X causes Y, that is, if not-X were true than not-Y would also be true.

  Now sup­pose there’s some Y’ analo­gous to Y, and you make the ar­gu­ment A: “since Y hap­pened, Y’ is also likely to hap­pen”. If that’s all you know, I agree that A is rea­son­able ev­i­dence that Y’ is likely to hap­pen. But if you then show that the analo­gous X’ is not true, while X was true, I think ar­gu­ment A pro­vides ~no ev­i­dence.

  Ex­am­ple:

  “It was rain­ing yes­ter­day, so it will prob­a­bly rain to­day.”

  “But it was cloudy yes­ter­day, and to­day it is sunny.”

  “Ah. In that case it prob­a­bly won’t rain.”

  I think con­di­tion 2 causes rac­ing causes MAD strate­gies in the case of nu­clear weapons; since con­di­tion 2 /​ rac­ing doesn’t hold in the case of AI, the fact that MAD strate­gies were used for nu­clear weapons pro­vides very lit­tle ev­i­dence about whether similar strate­gies will be used for AI.

  MAD strate­gies could still serve as some ev­i­dence for the gen­eral idea that coun­tries/​in­sti­tu­tions are some­times will­ing to do things that are risky to them­selves, and that pose very large nega­tive ex­ter­nal­ities of risks to oth­ers, for strate­gic rea­sons.

  I agree with that sen­tence in­ter­preted liter­ally. But I think you can change “for strate­gic rea­sons” to “in cases where con­di­tion 2 holds” and still cap­ture most of the cases in which this hap­pens.

• rohinmshah 3 Apr 2020 23:12 UTC

  LW: 3 AF: 3

  AF

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  Are you think­ing roughly that (a) re­turns diminish steeply from the cur­rent point, or (b) that effort will likely ramp up a lot in fu­ture and pluck a large quan­tity of the low hang­ing fruit that cur­rently re­main, such that even more ramp­ing up would face steeply diminish­ing re­turns?

  More like (b) than (a). In par­tic­u­lar, I’m think of lots of ad­di­tional effort by longter­mists, which prob­a­bly doesn’t re­sult in lots of ad­di­tional effort by ev­ery­one else, which already means that we’re scal­ing sub­lin­early. In ad­di­tion, you should then ex­pect diminish­ing marginal re­turns to more re­search, which lessens it even more.Also, a thing that I realized

  Also, I was think­ing about this re­cently, and I am pretty pes­simistic about wor­lds with dis­con­tin­u­ous take­off, which should maybe add an­other ~5 per­centage points to my risk es­ti­mate con­di­tional on no in­ter­ven­tion by longter­mists, and ~4 per­centage points to my un­con­di­tional risk es­ti­mate.

• rohinmshah 3 Apr 2020 2:10 UTC

  2 points

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  Con­di­tion 2 is nec­es­sary for race dy­nam­ics to arise, which is what peo­ple are usu­ally wor­ried about.

  Sup­pose that AI sys­tems weren’t go­ing to be use­ful for any­thing—the only effect of AI sys­tems was that they posed an x-risk to the world. Then it would still be true that “nei­ther side wants to do the thing, be­cause if they do the thing they get de­stroyed too”.

  Nonethe­less, I think in this world, no one ever builds AI sys­tems and so don’t need to worry about x-risk.

• rohinmshah 3 Apr 2020 2:07 UTC

  LW: 3 AF: 3

  AF

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  I guess I was just sug­gest­ing that your com­ments there, taken by them­selves/​out of con­text, seemed to ig­nore those im­por­tant ar­gu­ments, and thus might seem overly op­ti­mistic.

  Sure, that seems rea­son­able.

  Is this for ex­is­ten­tial risk from AI as a whole, or just “ad­ver­sar­ial op­ti­mi­sa­tion”/​”mis­al­ign­ment” type sce­nar­ios?

  Just ad­ver­sar­ial op­ti­miza­tion /​ mis­al­ign­ment. See the com­ment thread with Wei Dai be­low, es­pe­cially this com­ment.

  Like, for you, “there’s no ac­tion from longter­mists” would be a spe­cific con­straint you have to add to your world model?

  Oh yeah, definitely. (Toby does the same in The Precipice; his po­si­tion is that it’s clearer not to con­di­tion on any­thing, be­cause it’s usu­ally un­clear what ex­actly you are con­di­tion­ing on, though in per­son he did like the op­er­a­tional­iza­tion of “with­out ac­tion from longter­mists”.)

  Like, my model of the world is that for any suffi­ciently im­por­tant de­ci­sion like the de­vel­op­ment of pow­er­ful AI sys­tems, there are lots of hu­mans bring­ing many per­spec­tives to the table, which usu­ally ends up with most con­sid­er­a­tions be­ing brought up by some­one, and an over­all high level of risk aver­sion. On this model, longter­mists are one of the many groups that ar­gue for be­ing more care­ful than we oth­er­wise would be.

  I imag­ine you could also con­di­tion on some­thing like “sur­pris­ingly much ac­tion from longter­mists”, which would re­duce your es­ti­mated risk fur­ther?

  Yeah, pre­sum­ably. The 1 in 20 num­ber was very made up, even more so than the 1 in 10 num­ber. I sup­pose if our ac­tions were very suc­cess­ful, I could see us get­ting down to 1 in 1000? But if we just ex­erted a lot more effort (i.e. “sur­pris­ingly much ac­tion”), the ex­tra effort prob­a­bly doesn’t help much more than the ini­tial effort, so maybe… 1 in 25? 1 in 30?

  (All of this is very an­chored on the ini­tial 1 in 10 num­ber.)

• rohinmshah 2 Apr 2020 17:27 UTC

  LW: 3 AF: 3

  AF

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  So it seems like, un­less we ex­pect the rele­vant ac­tors to act in ac­cor­dance with some­thing close to im­par­tial al­tru­ism, we should ex­pect them to avoid risks some­what to avoid ex­is­ten­tial risks (or ex­tinc­tion speci­fi­cally), but far less than they re­ally should. (Roughly this ar­gu­ment is made in The Precipice, and I be­lieve by 80k.)

  I agree that ac­tors will fo­cus on x-risk far less than they “should”—that’s ex­actly why I work on AI al­ign­ment! This doesn’t mean that x-risk is high in an ab­solute sense, just higher than it “should” be from an al­tru­is­tic per­spec­tive. Pre­sum­ably from an al­tru­is­tic per­spec­tive x-risk should be very low (cer­tainly be­low 1%), so my 10% es­ti­mate is or­ders of mag­ni­tude higher than what it “should” be.

  Also, re: Precipice, it’s worth not­ing that Toby and I don’t dis­agree much—I es­ti­mate 1 in 10 con­di­tioned on no ac­tion from longter­mists; he es­ti­mates 1 in 5 con­di­tioned on AGI be­ing de­vel­oped this cen­tury. Let’s say that ac­tion from longter­mists can halve the risk; then my un­con­di­tional es­ti­mate would be 1 in 20, and would be very slightly higher if we con­di­tion on AGI be­ing de­vel­oped this cen­tury (be­cause we’d have less time to pre­pare), so over­all there’s a 4x differ­ence, which given the huge un­cer­tainty is re­ally not very much.

• rohinmshah 2 Apr 2020 17:18 UTC

  LW: 2 AF: 2

  AF

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  MAD-style strate­gies hap­pen when:

  1. There are two (or more) ac­tors that are in com­pe­ti­tion with each other

  2. There is a tech­nol­ogy such that if one ac­tor de­ploys it and the other ac­tor doesn’t, the first ac­tor re­mains the same and the sec­ond ac­tor is “de­stroyed”.

  3. If both ac­tors de­ploy the tech­nol­ogy, then both ac­tors are “de­stroyed”.

  (I just made these up right now; you could prob­a­bly get bet­ter ver­sions from pa­pers about MAD.)

  Con­di­tion 2 doesn’t hold for ac­ci­dent risk from AI: if any ac­tor de­ploys an un­al­igned AI, then both ac­tors are de­stroyed.

  I agree I didn’t ex­plain this well in the in­ter­view—when I said

  if the de­struc­tion hap­pens, that af­fects you too

  I should have said some­thing like

  if you de­ploy a dan­ger­ous AI sys­tem, that af­fects you too

  which is not true for nu­clear weapons (de­ploy­ing a nuke doesn’t af­fect you in and of it­self).

• rohinmshah 2 Apr 2020 17:03 UTC

  3 points

  in reply to: MichaelA's comment on: Ro­hin Shah on rea­sons for AI optimism

  Yeah I think I was prob­a­bly wrong about this (in­clud­ing what other peo­ple were talk­ing about when they said “nu­clear arms race”).

• rohinmshah 2 Apr 2020 17:02 UTC

  LW: 2 AF: 2

  AF

  in reply to: abramdemski's comment on: Think­ing About Filtered Ev­i­dence Is (Very!) Hard

  the Bayesian no­tion of be­lief doesn’t al­low us to make the dis­tinc­tion you are point­ing to

  Sure, that seems rea­son­able. I guess I saw this as the point of a lot of MIRI’s past work, and was ex­pect­ing this to be about hon­esty /​ filtered ev­i­dence some­how.

  I also think this re­sult has noth­ing to do with “you can’t have a perfect model of Carol”. Part of the point of my as­sump­tions is that they are, in­di­vi­d­u­ally, quite com­pat­i­ble with hav­ing a perfect model of Carol amongst the hy­pothe­ses.

  I think we mean differ­ent things by “perfect model”. What if I in­stead say “you can’t perfectly up­date on X and Carol-said-X , be­cause you can’t know why Carol said X, be­cause that could in the worst case re­quire you to know ev­ery­thing that Carol will say in the fu­ture”?

• rohinmshah 1 Apr 2020 19:36 UTC

  LW: 4 AF: 3

  AF

  in reply to: abramdemski's comment on: Think­ing About Filtered Ev­i­dence Is (Very!) Hard

  Yeah, I feel like while hon­esty is needed to prove the im­pos­si­bil­ity re­sult, the prob­lem arose with the as­sump­tion that the agent could effec­tively rea­son now about all the out­puts of a re­cur­sively enu­mer­able pro­cess (re­gard­less of hon­esty). Like, the way I would phrase this point is “you can’t perfectly up­date on X and Carol-said-X , be­cause you can’t have a perfect model of Carol”; this ap­plies whether or not Carol is hon­est. (See also this com­ment.)

[AN #93]: The Precipice we’re stand­ing at, and how we can back away from it

• rohinmshah 30 Mar 2020 17:48 UTC

  2 points

  in reply to: TAG's comment on: Solip­sism is Underrated

  This up­date seems like it would be ex­traor­di­nar­ily small, given our poor un­der­stand­ing of the brain, and the rel­a­tively small amount of con­certed effort that goes into un­der­stand­ing con­scious­ness.

• rohinmshah 30 Mar 2020 7:01 UTC

  LW: 2 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: Think­ing About Filtered Ev­i­dence Is (Very!) Hard

  I still don’t get it but prob­a­bly not worth dig­ging fur­ther. My cur­rent con­fu­sion is that even un­der the be­hav­iorist in­ter­pre­ta­tion, it seems like just be­liev­ing con­di­tion 2 im­plies know­ing all the things Carol would ever say (or Alice has a mis­taken be­lief). Prob­a­bly this is a con­fu­sion that would go away with enough for­mal­iza­tion /​ math, but it doesn’t seem worth do­ing that.

• rohinmshah 28 Mar 2020 19:52 UTC

  3 points

  on: Solip­sism is Underrated

  But us­ing ei­ther in­ter­pre­ta­tion, how puz­zling is the view, that the ac­tivity of these lit­tle ma­te­rial things some­how is re­spon­si­ble for con­scious qualia? This is where a lot of crit­i­cal think­ing has led many peo­ple to say things like “con­scious­ness must be what an al­gorithm im­ple­mented on a phys­i­cal ma­chine feels like from the ‘in­side.’” And this is a de­cent hy­poth­e­sis, but not an ex­plana­tory one at all. The emer­gence of con­scious­ness and qualia is just some­thing that ma­te­ri­al­ists need to ac­cept as a spooky phe­nomenon. It’s not a very satis­fy­ing solu­tion to the hard prob­lem of con­scious­ness.

  “lack of a satis­fy­ing ex­plana­tory solu­tion” does not im­ply low like­li­hood if you think that the ex­plana­tory solu­tion ex­ists but is com­pu­ta­tion­ally hard to find (which in fact seems pretty rea­son­able).

  Like, the same struc­ture of ar­gu­ment could be used to ar­gue that com­put­ers are ex­tremely low like­li­hood—how puz­zling is the view, that the ac­tivity of elec­trons mov­ing around some­how is re­spon­si­ble for prov­ing math­e­mat­i­cal the­o­rems?

  With lap­tops, we of course have a good ex­pla­na­tion of how com­pu­ta­tion arises from elec­trons, but that’s be­cause we de­signed them—it would prob­a­bly be much harder if we had no knowl­edge of lap­tops or even elec­tric­ity and then were handed a lap­top and asked to ex­plain how it could re­li­ably pro­duce true math­e­mat­i­cal the­o­rems. This seems pretty analo­gous to the situ­a­tion we find our­selves in with con­scious­ness.

• rohinmshah 28 Mar 2020 17:03 UTC

  10 points

  in reply to: Daniel Kokotajlo's comment on: When to as­sume neu­ral net­works can solve a problem

  “Hu­man Com­pat­i­ble” is mak­ing ba­si­cally the same points as “Su­per­in­tel­li­gence,” only in a dumbed-down and stream­lined man­ner, with lots of pre­sent-day ex­am­ples to illus­trate.

  I do not agree with this. I think the ar­gu­ments in Hu­man Com­pat­i­ble are more con­vinc­ing than the ones in Su­per­in­tel­li­gence (mostly be­cause they make fewer ques­tion­able as­sump­tions).

  (I agree that Stu­art prob­a­bly does agree some­what with the “Bostro­mian po­si­tion”.)

• rohinmshah 28 Mar 2020 5:53 UTC

  LW: 2 AF: 2

  AF

  in reply to: G Gordon Worley III's comment on: De­con­fus­ing Hu­man Values Re­search Agenda v1

  Some ex­am­ples of ac­tions taken by dic­ta­tors that I think were well in­ten­tioned and meant to fur­ther goals that seemed laud­able and not about power grab­bing to the dic­ta­tor but had net nega­tive out­comes for the peo­ple in­volved and the world:

  What’s your model for why those ac­tions weren’t un­done?

  
  

  To pop back up to the origi­nal ques­tion—if you think mak­ing your friend 10x more in­tel­li­gent would be net nega­tive, would you make them 10x dumber? Or per­haps it’s only good to make them 2x smarter, but af­ter that more marginal in­tel­li­gence is bad?

  It would be re­ally shock­ing if we were at the op­ti­mal ab­solute level of in­tel­li­gence, so I as­sume that you think we’re at the op­ti­mal rel­a­tive level of in­tel­li­gence, that is, the best situ­a­tion is when your friends are about as in­tel­li­gent as you are. In that case, let’s sup­pose that we in­crease/​de­crease all of your friends and your in­tel­li­gence by a fac­tor of X. For what range of X would you ex­pect this in­ter­ven­tion is net pos­i­tive?

  (I’m aware that in­tel­li­gence is not one-di­men­sional, but I feel like this is still a mostly mean­ingful ques­tion.)

  Just to be clear about my own po­si­tion, a well in­ten­tioned su­per­in­tel­li­gent AI sys­tem to­tally could make mis­takes. How­ever, it seems pretty un­likely that they’d be of the ex­is­ten­tially-catas­trophic kind. Also, the mis­take could be net nega­tive, but the AI sys­tem over­all should be net pos­i­tive.

• rohinmshah 28 Mar 2020 5:32 UTC

  LW: 4 AF: 3

  AF

  in reply to: DanielFilan's comment on: An An­a­lytic Per­spec­tive on AI Alignment

  Do you think I’m wrong?

  No, which is why I want to stop us­ing the ex­am­ple.

  (The coun­ter­fac­tual I was think­ing of was more like “imag­ine we handed a lap­top to 19th-cen­tury sci­en­tists, can they mechanis­ti­cally un­der­stand it?” But even that isn’t a good anal­ogy, it over­states the difficulty.)

• rohinmshah 28 Mar 2020 0:02 UTC

  LW: 4 AF: 3

  AF

  in reply to: johnswentworth's comment on: Align­ment as Translation

  Let me know if this anal­ogy sounds rep­re­sen­ta­tive of the strate­gies you imag­ine.

  Yeah, it does. I definitely agree that this doesn’t get around the chicken-and-egg prob­lem, and so shouldn’t be ex­pected to suc­ceed on the first try. It’s more like you get to keep try­ing this strat­egy over and over again un­til you even­tu­ally suc­ceed, be­cause if ev­ery­thing goes wrong you just un­plug the AI sys­tem and start over.

  the chicken-and-egg prob­lem is a ground truth prob­lem. If we have enough data to es­ti­mate X to within 5%, then do­ing clever things with that data is not go­ing re­duce that er­ror any fur­ther.

  I think you get “ground truth data” by try­ing stuff and see­ing whether or not the AI sys­tem did what you wanted it to do.

  (This does sug­gest that you wouldn’t ever be able to ask your AI sys­tem to do some­thing com­pletely novel with­out hav­ing a hu­man along to en­sure it’s what we ac­tu­ally meant, which seems wrong to me, but I can’t ar­tic­u­late why.)

• rohinmshah 27 Mar 2020 23:52 UTC

  LW: 4 AF: 3

  AF

  in reply to: johnswentworth's comment on: Align­ment as Translation

  Yeah, this could be a way that things are. My in­tu­ition is that it wouldn’t be this way, but I don’t have any good ar­gu­ments for it.

• rohinmshah 27 Mar 2020 23:48 UTC

  LW: 2 AF: 2

  AF

  in reply to: DanielFilan's comment on: An An­a­lytic Per­spec­tive on AI Alignment

  Yup, that seems like a pretty rea­son­able es­ti­mate to me.

  Note that my de­fault model for “what should be the in­put to es­ti­mate difficulty of mechanis­tic trans­parency” would be the num­ber of pa­ram­e­ters, not num­ber of neu­rons. If a neu­ron works over a much larger in­put (lead­ing to more pa­ram­e­ters), wouldn’t that make it harder to mechanis­ti­cally un­der­stand?

• rohinmshah 27 Mar 2020 20:44 UTC

  LW: 4 AF: 3

  AF

  in reply to: johnswentworth's comment on: Align­ment as Translation

  Any­way, sounds like value-in-the-tail is a cen­tral crux here.

  Seems some­what right to me, sub­ject to caveat be­low.

  it’s not a nec­es­sary con­di­tion—if the re­main­ing 5% of prob­lems are still ex­is­ten­tially deady and likely to come up even­tu­ally (but not of­ten enough to be caught in test­ing), then risk isn’t re­ally de­creased.

  An im­por­tant part of my in­tu­ition about value-in-the-tail is that if your first solu­tion can knock off 95% of the risk, you can then use the re­sult­ing AI sys­tem to de­sign a new AI sys­tem where you’ve trans­lated bet­ter and now you’ve elimi­nated 99% of the risk, and iter­at­ing this pro­cess you get to effec­tively no on­go­ing risk. There is of course risk dur­ing the iter­a­tion, but that risk can be rea­son­ably small.

  A similar ar­gu­ment ap­plies to eco­nomic com­pet­i­tive­ness: yes, your first agent is pretty slow rel­a­tive to what it could be, but you can make it faster and faster over time, so you only lose a lot of value dur­ing the first few ini­tial phases.

  (For the eco­nomic value part, this is mostly based on in­dus­try ex­pe­rience try­ing to au­to­mate things.)

  I have the same in­tu­ition, and strongly agree that usu­ally most of the value is in the long tail. The hope is mostly that you can ac­tu­ally keep mak­ing progress on the tail as time goes on, es­pe­cially with the help of your newly built AI sys­tems.