rohinmshah

• rohinmshah 21 Oct 2019 23:30 UTC

  3 points

  in reply to: Liam Donovan's comment on: What are the differ­ences be­tween all the iter­a­tive/​re­cur­sive ap­proaches to AI al­ign­ment?

  I’m not sure how rele­vant meta-ex­e­cu­tion is any more, I haven’t seen it dis­cussed much re­cently. So prob­a­bly you’d want to ask Paul, or some­one else who was around ear­lier than I was.

• rohinmshah 21 Oct 2019 23:29 UTC

  4 points

  in reply to: Liam Donovan's comment on: What are the differ­ences be­tween all the iter­a­tive/​re­cur­sive ap­proaches to AI al­ign­ment?

  Can you not mean­ingfully dis­cuss “this am­plifi­ca­tion pro­ce­dure is like an n-depth ap­prox­i­ma­tion of HCH at step x”, for any am­plifi­ca­tion pro­ce­dure?

  No, you can’t. E.g. If your am­plifi­ca­tion pro­ce­dure only al­lows you to ask a sin­gle sub­agent a sin­gle ques­tion, that will ap­prox­i­mate a lin­ear HCH in­stead of a tree-based HCH. If your am­plifi­ca­tion pro­ce­dure doesn’t in­voke sub­agents at all, but in­stead pro­vides more and more facts to the agent, it doesn’t look any­thing like HCH. The canon­i­cal im­ple­men­ta­tions of iter­ated am­plifi­ca­tion are try­ing to ap­prox­i­mate HCH though.

  For ex­am­ple, the in­ter­nal struc­ture of the dis­til­led agent de­scribed in Chris­ti­ano’s pa­per is un­likely to look any­thing like a tree. How­ever, my (po­ten­tially in­cor­rect?) im­pres­sion is that the agent’s ca­pa­bil­ities at step x are iden­ti­cal to an HCH tree of depth x if the un­der­ly­ing learn­ing sys­tem is ar­bi­trar­ily ca­pa­ble.

  That sounds right to me.

• rohinmshah 21 Oct 2019 6:31 UTC

  2 points

  in reply to: Liam Donovan's comment on: What are the differ­ences be­tween all the iter­a­tive/​re­cur­sive ap­proaches to AI al­ign­ment?

  I maybe think so? I’m not sure about meta-ex­e­cu­tion (as the com­ment thread above shows).

• rohinmshah 21 Oct 2019 6:29 UTC

  3 points

  in reply to: Liam Donovan's comment on: What are the differ­ences be­tween all the iter­a­tive/​re­cur­sive ap­proaches to AI al­ign­ment?

  “Depth” only ap­plies to the canon­i­cal tree-based im­ple­men­ta­tion of IDA. If you slot in other am­plifi­ca­tion or dis­til­la­tion pro­ce­dures, then you won’t nec­es­sar­ily have “depths” any more. You’ll still have re­cur­sion, and that re­cur­sion will lead to more and more ca­pa­bil­ity. Where it ends up de­pends on your ini­tial agent and how good the am­plifi­ca­tion and dis­til­la­tion pro­ce­dures are.

• rohinmshah 21 Oct 2019 6:17 UTC

  LW: 3 AF: 2

  AF

  in reply to: evhub's comment on: Re­laxed ad­ver­sar­ial train­ing for in­ner alignment

  That’s… a fair point. It does make up a sub­stan­tial por­tion of the trans­parency sec­tion, which seems like the “solu­tions” part of this post, but it isn’t the en­tire post.

  Matthew’s cer­tainly right that I tend to re­ply to things I dis­agree with, though I usu­ally try to avoid dis­agree­ing with de­tails. I’m not sure that I only dis­agree with de­tails here, but I can’t clearly ar­tic­u­late what about this feels off to me. I’ll delete the opinion al­to­gether; I’m not go­ing to put an un­clear opinion in the newslet­ter.

• rohinmshah 21 Oct 2019 5:58 UTC

  LW: 4 AF: 2

  AF

  in reply to: Charlie Steiner's comment on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  I cer­tainly would count an on­tolog­i­cal failure in the re­ward func­tion as an in­cor­rect be­lief about the re­ward func­tion.

• rohinmshah 21 Oct 2019 5:54 UTC

  2 points

  in reply to: danield's comment on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  Glad it was use­ful!

• rohinmshah 19 Oct 2019 19:33 UTC

  LW: 2 AF: 1

  AF

  in reply to: Matthew Barnett's comment on: Re­laxed ad­ver­sar­ial train­ing for in­ner alignment

  My opinion, also go­ing into the newslet­ter:

  Like Matthew, I’m ex­cited to see more work on trans­parency and ad­ver­sar­ial train­ing for in­ner al­ign­ment. I’m a some­what skep­ti­cal of the value of work that plans to de­com­pose fu­ture mod­els into a “world model”, “search” and “ob­jec­tive”: I would guess that there are many ways to achieve in­tel­li­gent cog­ni­tion that don’t eas­ily fac­tor into any of these con­cepts. It seems fine to study a sys­tem com­posed of a world model, search and ob­jec­tive in or­der to gain con­cep­tual in­sight; I’m more wor­ried about propos­ing it as an ac­tual plan.

• rohinmshah 19 Oct 2019 18:32 UTC

  LW: 12 AF: 4

  AF

  on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  I men­tioned in my opinion that I think many of my dis­agree­ments are be­cause of an im­plicit dis­agree­ment on how we build pow­er­ful AI sys­tems:

  the book has an im­plied stance to­wards the fu­ture of AI re­search that I don’t agree with: I could imag­ine that pow­er­ful AI sys­tems end up be­ing cre­ated by learn­ing alone with­out need­ing the con­cep­tual break­throughs that Stu­art out­lines.

  I didn’t ex­pand on this in the newslet­ter be­cause I’m not clear enough on the dis­agree­ment; I try to avoid writ­ing very con­fused thoughts that say wrong things about what other peo­ple be­lieve in a pub­li­ca­tion read by a thou­sand peo­ple. But that’s fine for a com­ment here!

  Rather than at­tribute a model to Stu­art, I’m just go­ing to make up a model that was in­spired by read­ing HC, but wasn’t pro­posed by HC. In this model, we get a su­per­in­tel­li­gent AI sys­tem that looks like a Bayesian-like sys­tem that ex­plic­itly rep­re­sents things like “be­liefs”, “plans”, etc. Some more de­tails:

  • Things like ‘hi­er­ar­chi­cal plan­ning’ are ex­plicit al­gorithms. Sim­ply look­ing at the al­gorithm can give you a lot of in­sight into how it does hi­er­ar­chy. You can in­spect things like “op­tions” just by look­ing at in­puts/​out­puts to the hi­er­ar­chi­cal plan­ning mod­ule. The same thing ap­plies for e.g. causal rea­son­ing.

  • Any black box deep learn­ing sys­tem is only used to provide low-level in­puts to the real ‘in­tel­li­gence’, in the same way that for hu­mans vi­sion pro­vides low-level in­puts for the rest of cog­ni­tion. We don’t need to worry about the deep learn­ing sys­tem “tak­ing over”, in the same way that we don’t worry about our vi­sion mod­ule “tak­ing over”.

  • The AI sys­tem was cre­ated by break­throughs in al­gorithms for causal rea­son­ing, hi­er­ar­chi­cal plan­ning, etc, that al­low it to deal with the com­bi­na­to­rial ex­plo­sion caused by the real world. As a re­sult, it is very cheap to run (i.e. doesn’t need a huge amount of com­pute). This is more com­pat­i­ble with a dis­con­tin­u­ous take­off, though a con­tin­u­ous take­off is pos­si­ble if the al­gorithms im­proved con­tin­u­ously over time, rather than hav­ing break­throughs.

  Some im­pli­ca­tions of this model:

  • All of the “in­tel­li­gence” is hap­pen­ing via ex­plicit al­gorithms. We only need to make sure that the al­gorithms are al­igned. So, we only have an outer al­ign­ment prob­lem; there is no in­ner al­ign­ment prob­lem.

  • Since the sys­tem is mostly Bayesian, the main challenges are to avoid mis­speci­fi­ca­tion (solu­tion: use some­thing equiv­a­lent to the Solomonoff prior) and to be com­pu­ta­tion­ally effi­cient (solu­tion: keep a small set of hy­pothe­ses, de­tect when they fail to ex­plain the data, and ex­pand to a big­ger class of hy­pothe­ses). You don’t have to worry about other forms of ro­bust­ness like ad­ver­sar­ial ex­am­ples.

• rohinmshah 19 Oct 2019 18:07 UTC

  LW: 8 AF: 4

  AF

  on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  If you’re cu­ri­ous about how I se­lect what goes in the newslet­ter: I al­most put in this crit­i­cal re­view of the book, in the spirit of pre­sent­ing both sides of the ar­gu­ment. I didn’t put it in be­cause I couldn’t un­der­stand it.

  My best guess right now is that the au­thor is ar­gu­ing that “we’ll never get su­per­in­tel­li­gence”, pos­si­bly be­cause in­tel­li­gence isn’t a co­her­ent con­cept, but there’s prob­a­bly some­thing more that I’m not get­ting. If it turned out that it was only say­ing “we’ll never get su­per­in­tel­li­gence”, and there weren’t any new sup­port­ing ar­gu­ments, I wouldn’t in­clude it in the newslet­ter, be­cause we’ve seen and heard that coun­ter­ar­gu­ment more than enough.

• rohinmshah 19 Oct 2019 18:02 UTC

  LW: 15 AF: 5

  AF

  on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  I en­joyed pages 185-190, on math­e­mat­i­cal guaran­tees, es­pe­cially be­cause I’ve been con­fused about what the “prov­ably benefi­cial” in CHAI’s mis­sion state­ment is meant to say. Some quotes:

  On the other hand, if you want to prove some­thing about the real world—for ex­am­ple, that AI sys­tems de­signed like so won’t kill you on pur­pose—your ax­ioms have to be true in the real world. If they aren’t true, you’ve proved some­thing about an imag­i­nary world.

  On the ap­pli­ca­bil­ity of the­o­rems to prac­tice:

  The trick is to know how far one can stray from the real world and still ob­tain use­ful re­sults. For ex­am­ple, if the rigid-beam as­sump­tion al­lows an en­g­ineer to calcu­late the forces in a struc­ture that in­cludes the beam, and those forces are small enough to bend a real steel beam by only a tiny amount, then the en­g­ineer can be rea­son­ably con­fi­dent that the anal­y­sis will trans­fer from the imag­i­nary world to the real world.

  as well as

  The pro­cess of re­mov­ing un­re­al­is­tic as­sump­tions con­tinues un­til the en­g­ineer is fairly con­fi­dent that the re­main­ing as­sump­tions are true enough in the real world. After that, the en­g­ineered sys­tem can be tested in the real world; but the test re­sults are just that. They do not prove that the same sys­tem will work in other cir­cum­stances or that other in­stances of the sys­tem will be­have the same way as the origi­nal.

  It then talks about as­sump­tion failure in cryp­tog­ra­phy due to side-chan­nel at­tacks.

  A some­what more con­crete ver­sion of what “prov­ably benefi­cial” might mean:

  Let’s look at the kind of the­o­rem we would like even­tu­ally to prove about ma­chines that are benefi­cial to hu­mans. One type might go some­thing like this:

  Sup­pose a ma­chine has com­po­nents $A$, $B$, $C$, con­nected to each other like so and to the en­vi­ron­ment like so, with in­ter­nal learn­ing al­gorithms $l_A$, $l_B$, $l_C$ that op­ti­mize in­ter­nal feed­back re­wards $r_A$, $r_B$, $r_C$ defined like so, and [a few more con­di­tions] . . . then, with very high prob­a­bil­ity, the ma­chine’s be­hav­ior will be very close in value (for hu­mans) to the best pos­si­ble be­hav­ior re­al­iz­able on any ma­chine with the same com­pu­ta­tional and phys­i­cal ca­pa­bil­ities.

  The main point here is that such a the­o­rem should hold re­gard­less of how smart the com­po­nents be­come—that is, the ves­sel never springs a leak and the ma­chine always re­mains benefi­cial to hu­mans.

  There are three other points worth mak­ing about this kind of the­o­rem. First, we can­not try to prove that the ma­chine pro­duces op­ti­mal (or even near-op­ti­mal) be­hav­ior on our be­half, be­cause that’s al­most cer­tainly com­pu­ta­tion­ally im­pos­si­ble. [...] Se­cond, we say “very high prob­a­bil­ity . . . very close” be­cause that’s typ­i­cally the best that can be done with ma­chines that learn. [...] Fi­nally, we are a long way from be­ing able to prove any such the­o­rem for re­ally in­tel­li­gent ma­chines op­er­at­ing in the real world!

  It then goes on to dis­cuss how such a the­o­rem is sub­ject to “side-chan­nel at­tacks” be­cause such the­o­rems typ­i­cally as­sume Carte­sian du­al­ity, which is not ac­tu­ally true (see Embed­ded Agency).

• rohinmshah 19 Oct 2019 17:46 UTC

  LW: 15 AF: 6

  AF

  on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  Quote from the book on the prob­lem of al­ign­ing black box mod­els:

  The task is, for­tu­nately, not the fol­low­ing: given a ma­chine that pos­sesses a high de­gree of in­tel­li­gence, work out how to con­trol it. If that were the task, we would be toast. A ma­chine viewed as a black box, a fait ac­com­pli, might as well have ar­rived from outer space. And our chances of con­trol­ling a su­per­in­tel­li­gent en­tity from outer space are roughly zero. Similar ar­gu­ments ap­ply to meth­ods of cre­at­ing AI sys­tems that guaran­tee we won’t un­der­stand how they work; these meth­ods in­clude whole-brain em­u­la­tion — cre­at­ing souped-up elec­tronic copies of hu­man brains — as well as meth­ods based on simu­lated evolu­tion of pro­grams. I won’t say more about these pro­pos­als be­cause they are so ob­vi­ously a bad idea.

  This is un­for­tu­nately the only para­graph that HC de­votes to the mat­ter.

• rohinmshah 19 Oct 2019 17:44 UTC

  LW: 2 AF: 1

  AF

  in reply to: habryka's comment on: [AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

  Glad to hear it! Yeah, I do ex­pect many peo­ple to dis­agree with many parts of this book. My guess is that it mostly boils down to a differ­ence in pre­dic­tions about how we build pow­er­ful AI sys­tems.

[AN #69] Stu­art Rus­sell’s new book on why we need to re­place the stan­dard model of AI

[AN #68]: The at­tain­able util­ity the­ory of impact

• rohinmshah 12 Oct 2019 23:11 UTC

  LW: 4 AF: 2

  AF

  on: What You See Isn’t Always What You Want

  Planned sum­mary:

  This post makes the point that for Marko­vian re­ward func­tions on ob­ser­va­tions, since any given ob­ser­va­tion can cor­re­spond to mul­ti­ple un­der­ly­ing states, we can­not know just by an­a­lyz­ing the re­ward func­tion whether it ac­tu­ally leads to good be­hav­ior: it also de­pends on the en­vi­ron­ment. For ex­am­ple, sup­pose we want an agent to col­lect all of the blue blocks in a room to­gether. We might sim­ply re­ward it for hav­ing blue in its ob­ser­va­tions: this might work great if the agent only has the abil­ity to pick up and move blocks, but won’t work well if the agent has a paint­brush and blue paint. This makes the re­ward de­signer’s job much more difficult. How­ever, the de­signer could use tech­niques that don’t re­quire a re­ward on in­di­vi­d­ual ob­ser­va­tions, such as re­wards that can de­pend on the agent’s in­ter­nal cog­ni­tion (as in iter­ated am­plifi­ca­tion), or re­wards that can de­pend on his­to­ries (as in Deep RL from Hu­man Prefer­ences).

  Planned opinion:

  I cer­tainly agree that we want to avoid re­ward func­tions defined on ob­ser­va­tions, and this is one rea­son why. It seems like a more gen­eral ver­sion of the wire­head­ing ar­gu­ment to me, and ap­plies even if you think that the AI won’t be able to wire­head, as long as it is ca­pa­ble enough to find other plans for get­ting high re­ward be­sides the one the de­signer in­tended.

• rohinmshah 12 Oct 2019 17:21 UTC

  4 points

  in reply to: DanielFilan's comment on: Thoughts on “Hu­man-Com­pat­i­ble”

  See also Re­ward Uncer­tainty.

• rohinmshah 12 Oct 2019 17:18 UTC

  LW: 6 AF: 2

  AF

  in reply to: rohinmshah's comment on: Thoughts on “Hu­man-Com­pat­i­ble”

  Clar­ifi­ca­tion: For me, the gen­eral worry is some­thing like “if I get quoted, I need to make sure that it’s not mis­lead­ing (which can hap­pen even if the per­son quot­ing me didn’t mean to be mis­lead­ing), and that takes time and effort and notic­ing all the places where I’m quoted, and it’s just eas­ier to not say things at all”.

  (Other peo­ple may have more wor­ries, like “If I say some­thing that could be in­ter­preted as be­ing crit­i­cal of the or­ga­ni­za­tion, and that be­comes suffi­ciently well-pub­li­cized, then I might get fired, so I’ll just never say any­thing like that.”)

• rohinmshah 11 Oct 2019 2:30 UTC

  LW: 12 AF: 5

  AF

  in reply to: Wei_Dai's comment on: Thoughts on “Hu­man-Com­pat­i­ble”

  I won­der why Pro­fes­sor Rus­sell doesn’t de­scribe his agenda in more tech­ni­cal de­tail, or en­gage much with the tech­ni­cal AI safety com­mu­nity, to the ex­tent that even grad stu­dents at CHAI ap­par­ently do not know much about his ap­proach.

  For the sake of ex­plain­ing this: for quite a while, he’s been en­gag­ing with aca­demics and poli­cy­mak­ers, and writ­ing a book; it’s not that he’s been do­ing re­search and not talk­ing to any­one about it.

  Fyi, when you quote peo­ple who work at an or­ga­ni­za­tion say­ing some­thing that has a nega­tive im­pli­ca­tion about that or­ga­ni­za­tion, you make it less likely that peo­ple will say things like that in the fu­ture. I’m not say­ing that you did any­thing wrong here; I just want to make sure that you know of this effect, and that it does make me in par­tic­u­lar more likely to be silent the next time you ask about CHAI rather than re­spond­ing.

• rohinmshah 11 Oct 2019 2:19 UTC

  LW: 4 AF: 3

  AF

  in reply to: Grue_Slinky's comment on: AI Align­ment Open Thread Oc­to­ber 2019

  Con­cerns about mesa-op­ti­miz­ers are mostly con­cerns that “ca­pa­bil­ities” will be ro­bust to dis­tri­bu­tional shift while “ob­jec­tives” will not be ro­bust.