rohinmshah

• rohinmshah 15 Jun 2019 1:17 UTC

  2 points

  AF

  in reply to: David Krueger's comment on: Let’s talk about “Con­ver­gent Ra­tion­al­ity”

  I guess my po­si­tion is that CRT is only true to the ex­tent that you build a goal-di­rected agent. (Tech­ni­cally, the in­ner op­ti­miz­ers ar­gu­ment is one way that CRT could be true even with­out build­ing an ex­plic­itly goal-di­rected agent, but it seems like you view CRT as broader and more likely than in­ner op­ti­miz­ers, and I’m not sure how.)

  Maybe an­other way to get at the un­der­ly­ing mi­s­un­der­stand­ing: do you see a differ­ence be­tween “con­ver­gent ra­tio­nal­ity” and “con­ver­gent goal-di­rect­ed­ness”? If so, what is it? From what you’ve writ­ten they sound equiv­a­lent to me.

• rohinmshah 14 Jun 2019 15:38 UTC

  3 points

  AF

  on: Let’s talk about “Con­ver­gent Ra­tion­al­ity”

  The main counter-ar­gu­ments arise from VNMUT, which can be in­ter­preted as say­ing “ra­tio­nal agents are more fit” (in an evolu­tion­ary sense).

  While I gen­er­ally agree with CRT as ap­plied to ad­vanced agents, the VNM the­o­rem is not the rea­son why, be­cause it is vac­u­ous. I agree with steve that the real ar­gu­ment for it is that hu­mans are more likely to build goal-di­rected agents be­cause that’s the only way we know how to get AI sys­tems that do what we want. But we to­tally could build non-goal-di­rected agents that CRT doesn’t ap­ply to, e.g. Google Maps.

• rohinmshah 12 Jun 2019 6:56 UTC

  2 points

  in reply to: G Gordon Worley III's comment on: Con­clu­sion to the se­quence on value learning

  It sounds to me like you’re re­quiring “su­per­in­tel­li­gent” to in­clude “has a goal” as part of the defi­ni­tion. If that’s part of the defi­ni­tion, then I would rephrase my point as “why do we have to build some­thing su­per­in­tel­li­gent? Let’s in­stead build some­thing that doesn’t have a goal but is still use­ful, like an AI sys­tem that fol­lows norms.”

  See also this com­ment, which an­swers a re­lated ques­tion.

• rohinmshah 11 Jun 2019 6:08 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: AGI will dras­ti­cally in­crease economies of scale

  In that case this model would only hold if gov­ern­ments:

  • Ac­tu­ally think through the long-term im­pli­ca­tions of AI

  • Think about this par­tic­u­lar argument

  • Have enough cer­tainty in this ar­gu­ment to ac­tu­ally act upon it

  Notably, there aren’t any feed­back loops for the thing-be­ing-com­peted-on, and so nat­u­ral-se­lec­tion style op­ti­miza­tion doesn’t hap­pen. This makes me much less likely to be­lieve in ar­gu­ments of the form “The thing-be­ing-com­peted-on will have a high value, be­cause there is com­pe­ti­tion”—the mechanism that usu­ally makes that true is nat­u­ral se­lec­tion or some equiv­a­lent.

• rohinmshah 11 Jun 2019 6:00 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: AGI will dras­ti­cally in­crease economies of scale

  Why? Each di­vi­sion can still have sep­a­rate profit-loss ac­count­ing, so you can de­cide to shut one down if it starts mak­ing losses, and the benefits of hav­ing that di­vi­sion to the rest of the com­pany doesn’t out­weigh the losses. The lat­ter may be some­what tricky to judge though. Per­haps that’s what you meant?

  That’s a good point. I was imag­in­ing that each di­vi­sion ends up be­com­ing a monopoly in its par­tic­u­lar area due to the benefits of within-firm co­or­di­na­tion, which means that even if the di­vi­sion is in­effi­cient there isn’t an al­ter­na­tive that the firm can go with. But that was an as­sump­tion, and I’m not sure it would ac­tu­ally hold.

• rohinmshah 11 Jun 2019 1:03 UTC

  5 points

  in reply to: Raemon's comment on: AGI will dras­ti­cally in­crease economies of scale

  Okay, I see, that makes sense and seems plau­si­ble, though I’d bet against it hap­pen­ing. But you’ve con­vinced me that I should qual­ify that sen­tence more.

• rohinmshah 11 Jun 2019 1:00 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: AGI will dras­ti­cally in­crease economies of scale

  If com­pa­nies had fully al­igned work­ers and man­agers, they could adopt what Robin Han­son calls the “di­vi­sions” model where each di­vi­sion works just like a sep­a­rate com­pany ex­cept that there is an over­all CEO that “looks for rare chances to gain value by co­or­di­nat­ing di­vi­sion ac­tivi­ties”

  Once you switch to the “di­vi­sions” model your di­vi­sions are no longer com­pet­ing with other firms, and all the di­vi­sions live or die as a group. So you’re giv­ing up the op­ti­miza­tion that you could get via ob­serv­ing which com­pa­nies suc­ceed /​ fail at di­vi­sion-level tasks. I’m not sure how big this effect is, though I’d guess it’s small.

  While search­ing for that post, I also came across Firm Ineffi­ciency which like Mo­ral Mazes (but much more con­cisely) lists many in­effi­cien­cies that seem all or mostly re­lated to value differ­ences.

  Yeah, I’m more con­vinced now that prin­ci­pal-agent is­sues are sig­nifi­cantly larger than other is­sues.

  I think it’s at least one of the main ar­gu­ments that Eric Drexler makes, since he wrote this in his abstract

  Yeah, I agree it’s an ar­gu­ment against that ar­gu­ment from Eric. I for­got about that Eric makes that point (mainly be­cause I have never been very con­vinced by it)

  Yeah I’m not very fa­mil­iar with this ei­ther, but my un­der­stand­ing is that such merg­ers are only ille­gal if the effect “may be sub­stan­tially to lessen com­pe­ti­tion” or “tend to cre­ate a monopoly”, which tech­ni­cally (it seems to me) isn’t the case when ex­ist­ing mo­nop­o­lies in differ­ent in­dus­tries merge.

  My guess would be that the spirit of the law would ap­ply, and that would be enough, but re­ally I’d want to ask a so­cial sci­en­tist or lawyer.

• rohinmshah 11 Jun 2019 0:38 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: AGI will dras­ti­cally in­crease economies of scale

  Does that seem right to you, or do you see things turn out a differ­ent way (in the long run)?

  I agree that di­rect mil­i­tary com­pe­ti­tion would cre­ate such a pres­sure.

  I’m not sure that ab­sent that there ac­tu­ally is com­pe­ti­tion be­tween coun­tries—what are they even com­pet­ing on? You’re rea­son­ing as though they com­pete on eco­nomic effi­ciency, but what causes coun­tries with lower eco­nomic effi­ciency to van­ish? Per­haps in coun­tries with lower eco­nomic effi­ciency, vot­ers tend to put in a new gov­ern­ment—but in that case it seems like re­ally the com­pe­ti­tion be­tween coun­tries is on “what pleases vot­ers”, which may not be ex­actly what we want but it prob­a­bly isn’t too risky if we have an AGI-fueled gov­ern­ment that’s in­tent-al­igned with “what pleases vot­ers”.

  (It’s pos­si­ble that you get poli­ti­ci­ans who look like they’re try­ing to please vot­ers but once they have enough power they then serve their own in­ter­ests, but this looks like “the gov­ern­ment gains power, and the peo­ple no longer have effec­tive con­trol over gov­ern­ment”.)

• rohinmshah 10 Jun 2019 17:47 UTC

  2 points

  in reply to: Raemon's comment on: AGI will dras­ti­cally in­crease economies of scale

  Maybe speci­fi­cally, which in­sti­tu­tions do you mean?

  Govern­ments, and speci­fi­cally an­titrust law.

  I think there are big differ­ences be­tween the cur­rent situ­a­tion and pre­vi­ous tech­nolo­gies: a) it is higher-stakes and b) even in­dus­try seems to be some­what pro-reg­u­la­tion.

  As­sum­ing I’m roughly cor­rect that there’s a lag, there’d be a sev­eral year win­dow where the in­sti­tu­tions that nor­mally reg­u­late com­pa­nies are go­ing to be too con­fused and di­s­ori­ented to do so.

  I’m try­ing to cache this out into a more con­crete failure story. Are you imag­in­ing that a com­pany de­vel­ops AGI, starts be­com­ing more and more pow­er­ful, and af­ter 10 years of be­ing con­fused and di­s­ori­ented the gov­ern­ment says “you’re too big, you need to be bro­ken up” and the com­pany says “no” and takes over the gov­ern­ment?

• rohinmshah 10 Jun 2019 17:36 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: AGI will dras­ti­cally in­crease economies of scale

  ETA: By “in­for­ma­tion asym­me­try” did you mean some­thing more like the fact that differ­ent copies or parts of an AGI can have ac­cess to differ­ent in­for­ma­tion and it can be costly (on a tech­ni­cal level) to prop­a­gate that in­for­ma­tion across the whole AGI? If so that seems like a much smaller cost than the kind of cost from “asym­met­ric in­for­ma­tion” that I’m talk­ing about. Also it seems like it would be good to use a differ­ent phrase to talk about what you mean, so peo­ple don’t con­fuse it with the con­cept of “asym­met­ric in­for­ma­tion” in eco­nomics.

  Yes, that’s right, though I can see how it’s con­fus­ing based on the eco­nomics liter­a­ture. Any sug­ges­tions for an al­ter­na­tive phrase? I was con­sid­er­ing “com­mu­ni­ca­tion costs”, but there could also be costs from the fact that differ­ent parts have differ­ent com­pe­ten­cies.

  It’s not clear to me that prin­ci­pal-agent costs are more im­por­tant than the ones I’m talk­ing about here. My ex­pe­rience of work­ing in large com­pa­nies is not that I was mis­al­igned with the com­pany, it was that the com­pany’s “plan” (to the ex­tent that one ex­isted) was ex­tremely large and com­plex and not some­thing I could eas­ily un­der­stand. It could be that this is ac­tu­ally the most effi­cient way to work even with in­tent-al­igned agents, since com­mu­ni­cat­ing the full plan could in­volve very large com­mu­ni­ca­tion costs.

  (I agree that the Mo­ral Mazes ar­gu­ments are pri­mar­ily about prin­ci­pal-agent prob­lems, but I don’t know how much to be­lieve Mo­ral Mazes.)

  This seems right to me, so I think con­tra Drexler, this is an­other rea­son to ex­pect a strong com­pet­i­tive pres­sure to move from CAIS to AGI.

  Seems rea­son­able, though I don’t think it is ar­gu­ing against the main ar­gu­ments in fa­vor of CAIS (which to me are that CAIS seems more tech­ni­cally fea­si­ble than AGI).

  With AGI-op­er­ated com­pa­nies, these prob­lems be­come smaller be­cause mo­nop­o­lies in differ­ent in­dus­tries can merge with­out be­ing limited by in­ter­nal co­or­di­na­tion costs and these merged com­pa­nies can in­ter­nally charge each other effi­cient prices.

  I don’t see how this sug­gests that our ex­ist­ing in­sti­tu­tions to pre­vent cen­tral­iza­tion of power will go away, since even now mo­nop­o­lies could merge, of­ten want to merge, but are pre­vented by law from do­ing so. (Though I’m not very con­fi­dent in this claim, I’m mostly par­rot­ing back things I’ve heard.)

  In the limit of a sin­gle AGI con­trol­ling the whole econ­omy, all such in­effi­cien­cies go away.

  Right, but that re­quires gov­ern­ment buy-in, which is ex­actly my model of risk in the opinion I wrote.

  On sec­ond thought, part of the rea­son for such in­sti­tu­tions to ex­ist must also be do­mes­tic poli­ti­cal pres­sure (from peo­ple who are afraid of too much con­cen­tra­tion of power), so at least that pres­sure would per­sist in coun­tries where such pres­sure ex­ists or has much force in the first place.

  Yeah, that’s my pri­mary model here. I’d be sur­prised but not shocked if com­pe­ti­tion be­tween coun­tries ex­plained most of the effect.

• rohinmshah 10 Jun 2019 17:09 UTC

  7 points

  in reply to: Raemon's comment on: AGI will dras­ti­cally in­crease economies of scale

  I do it some­times, es­pe­cially when I ex­pect it to be more con­tro­ver­sial, and also when I’m not do­ing it su­per last minute an hour be­fore the newslet­ter goes out. I’m a bit be­hind on ev­ery­thing but do want to move in this di­rec­tion in the fu­ture.

• rohinmshah 9 Jun 2019 16:48 UTC

  8 points

  AF

  on: AGI will dras­ti­cally in­crease economies of scale

  Planned sum­mary:

  Economies of scale would nor­mally mean that com­pa­nies would keep grow­ing larger and larger. With hu­man em­ploy­ees, the co­or­di­na­tion costs grow su­per­lin­early, which ends up limit­ing the size to which a com­pany can grow. How­ever, with the ad­vent of AGI, many of these co­or­di­na­tion costs will be re­moved. If we can al­ign AGIs to par­tic­u­lar hu­mans, then a cor­po­ra­tion run by AGIs al­igned to a sin­gle hu­man would at least avoid prin­ci­pal-agent costs. As a re­sult, the economies of scale would dom­i­nate, and com­pa­nies would grow much larger, lead­ing to more cen­tral­iza­tion.

  Planned opinion:

  This ar­gu­ment is quite com­pel­ling to me un­der the as­sump­tion of hu­man-level AGI sys­tems that can be in­tent-al­igned. Note though that while the de­vel­op­ment of AGI sys­tems re­moves prin­ci­pal-agent prob­lems, it doesn’t re­move is­sues that arise due to in­for­ma­tion asym­me­try.

  It does seem like this doesn’t hold with some­thing like CAIS, where each AI ser­vice is op­ti­mized for a par­tic­u­lar task, since there likely will be prin­ci­pal-agent prob­lems be­tween ser­vices.

  It seems like the ar­gu­ment should mainly make us more wor­ried about sta­ble au­thor­i­tar­ian regimes: the main effect based on this ar­gu­ment is a cen­tral­iza­tion of power in the hands of the AGI’s over­seers. This won’t hap­pen with com­pa­nies, be­cause we already have in­sti­tu­tions that pre­vent com­pa­nies from gain­ing too much power, and there doesn’t seem to be a strong rea­son to ex­pect that to stop. It could hap­pen with gov­ern­ment, but if long-term gov­ern­men­tal power still rests with the peo­ple via democ­racy, that seems okay. So the risky situ­a­tion seems to be when the gov­ern­ment gains power, and the peo­ple no longer have effec­tive con­trol over gov­ern­ment. (This would in­clude sce­nar­ios with e.g. a gov­ern­ment that has suffi­ciently good AI-fueled pro­pa­ganda that they always win elec­tions, re­gard­less of whether their gov­ern­ing is ac­tu­ally good.)

• rohinmshah 9 Jun 2019 4:57 UTC

  4 points

  AF

  in reply to: Vanessa Kosoy's comment on: [AN #57] Why we should fo­cus on ro­bust­ness in AI safety, and the analo­gous prob­lems in programming

  I think that most of the “in­ten­tional” catas­trophic ac­tions can be re­garded as “due to ig­no­rance” from an ap­pro­pri­ate perspective

  This makes sense to me at a high level, but I’m strug­gling to con­nect it to the math. It seems like del­ega­tive RL as de­scribed in the post I read wouldn’t solve re­ward hack­ing, be­cause it has a spe­cific re­ward func­tion that it is try­ing to max­i­mize, and it can’t “learn” a new re­ward func­tion that it should be op­ti­miz­ing. I sup­pose if the ad­vi­sor never ex­plores an area of state space then the agent will never go there, but it doesn’t feel like much progress if our safety guaran­tees re­quire the del­e­ga­tor to never ex­plore any­where that re­ward hack­ing could oc­cur.

  Ex­am­ple 1 is cor­rupt states, that I dis­cussed here. Th­ese are states in which the speci­fied re­ward func­tion doesn’t match the in­tended re­ward func­tion (and also pos­si­bly the ad­vi­sor be­comes un­re­li­able).

  So to be clear, this set­ting re­quires a differ­ent al­gorithm, de­tailed in the other post, right? (I haven’t read that post in de­tail.) Maybe that’s the an­swer to my ques­tion above; that in fact del­ega­tive RL doesn’t solve re­ward hack­ing, but this other post does.

• rohinmshah 9 Jun 2019 4:30 UTC

  4 points

  AF

  in reply to: Wei_Dai's comment on: [AN #57] Why we should fo­cus on ro­bust­ness in AI safety, and the analo­gous prob­lems in programming

  What’s PL? Pro­gram­ming lan­guages?

  Yes, sorry for the jar­gon.

  Do you know if there are any fur­ther de­tails about it some­where, aside from just the bare idea of “maybe we can learn speci­fi­ca­tions from eval­u­a­tive feed­back”?

  Not to my knowl­edge.

• rohinmshah 7 Jun 2019 17:47 UTC

  4 points

  AF

  in reply to: Wei_Dai's comment on: [AN #57] Why we should fo­cus on ro­bust­ness in AI safety, and the analo­gous prob­lems in programming

  I’m in­ter­ested to know what in­sights you gained from this anal­ogy, aside from what Push­meet talked about.

  I’m not sure there’s any par­tic­u­lar in­sights I can point to; it was more an anal­ogy that helped see how we had tack­led similar prob­lems be­fore. I don’t think it’s that use­ful for figur­ing out solu­tions—there’s a rea­son I haven’t done any pro­jects at the in­ter­sec­tion of PL and AI, de­spite my huge com­par­a­tive ad­van­tage at it.

  An ex­am­ple of the anal­ogy is in pro­gram syn­the­sis, in which you give some speci­fi­ca­tion of what a pro­gram should do, and then the com­puter figures out the pro­gram that meets that speci­fi­ca­tion. When you spec­ify a pro­gram via in­put-out­put ex­am­ples, you need to have “ca­pa­bil­ity con­trol” to make sure that it doesn’t come up with the pro­gram “if in­put == X: re­turn Y elif …”. It also of­ten “games” the speci­fi­ca­tion; one of my fa­vorite ex­am­ples is the func­tion for sort­ing an ar­ray—of­ten a new stu­dent will provide the speci­fi­ca­tion as $\forall i,j:i<j⟹A\left[i\right]<A\left[j\right]$, and then the out­putted pro­gram is “re­turn []” or “re­turn range(len(A))”.

  There are sup­posed to be reg­u­lar meet­ings be­tween FHI and Deep­Mind, but it seems like the lead­ers at Deep­Mind aren’t fa­mil­iar with or aren’t con­vinced of Nick Bostrom’s ideas about The Vuln­er­a­ble World Hy­poth­e­sis and Differ­en­tial Tech­nolog­i­cal Devel­op­ment.

  As con­text (not re­ally dis­agree­ing), afaik those meet­ings are be­tween Deep­Mind’s AGI safety team and FHI. Push­meet is not on that team and so prob­a­bly doesn’t at­tend those meet­ings.

  I’m cu­ri­ous to learn about speci­fi­ca­tion learn­ing but didn’t see where in the tran­script Push­meet talked about it. Can you give a poin­ter to what Push­meet said, or a blog post or pa­per?

  Here’s one sec­tion, I think there were oth­ers that also al­luded to it. Read­ing it again it’s maybe more ac­cu­rate from this sec­tion to talk about the ex­pres­sivity of our lan­guage of speci­fi­ca­tions, rather than speci­fi­ca­tion learn­ing:

  Robert Wiblin: Yeah. That leads into my next ques­tion, which was, is it go­ing to be pos­si­ble to for­mally ver­ify safety perfor­mance on the ML sys­tems that we want to use?
  
  Push­meet Kohli: I think a more per­ti­nent ques­tion is, would it be pos­si­ble to spec­ify what we want out of the sys­tem, be­cause at the end of the day you can only ver­ify what you can spec­ify. I think tech­ni­cally there is noth­ing, of course this is a very hard prob­lem, but fun­da­men­tally we have solved hard search prob­lems and challeng­ing op­ti­miza­tion prob­lems and so on. So it is some­thing that we can work to­wards, but a more crit­i­cal prob­lem is spec­i­fy­ing what do we want to ver­ify? What do we want to for­mally ver­ify? At the mo­ment we ver­ify, is my func­tion con­sis­tent with the in­put-out­put ex­am­ples, that I gave the ma­chine learn­ing sys­tem and that’s very easy. You can take all the in­puts in the train­ing set, you can com­pute the out­puts and then check whether the out­puts are the same or not. That’s a very sim­ple thing. No rocket sci­ence needed.
  
  Push­meet Kohli: Now, you can have a more so­phis­ti­cated speci­fi­ca­tion say­ing, well, if I per­turb the in­put in some way or trans­form the in­put and I ex­pect the out­put to not change or change in a spe­cific way, is it true? That’s a harder ques­tion and would be show­ing that we can try to make progress. But what other types of speci­fi­ca­tions or what other type of be­hav­ior or what kind of rich ques­tions might peo­ple want to ask in the fu­ture? That is a more challeng­ing prob­lem to think about.
  
  Robert Wiblin: In­ter­est­ing. So then rel­a­tive to other peo­ple you think it’s go­ing to be figur­ing out what we want to ver­ify that’s harder rather than the ver­ifi­ca­tion pro­cess it­self?
  
  Push­meet Kohli: Yeah, like how do you spec­ify what is the task? Like a task is not a data set..
  
  Robert Wiblin: How do you? Do you have any thoughts on that?
  
  Push­meet Kohli: Yes. I think this is some­thing that … It goes into like how this whole idea of, it’s a very philo­soph­i­cal thing, how do we spec­ify tasks? When we talk about tasks, we talk about in hu­man lan­guage. I can de­scribe a task to you and be­cause we share some no­tion of cer­tain con­cepts, I can tell you, well, we should try to de­tect whether a car passes by and what is a car, a car has some­thing which has four wheels and some­thing, and can drive it­self and so on. And a child with a scooter, which also has four wheels goes past and you say, “Oh that’s a car.” You say, “No, that’s not a car.” The car is slightly differ­ent, big­ger, ba­si­cally peo­ple can sit in­side it and so on. I’m de­scribing the task of de­tect­ing what is a car in these hu­man con­cepts that I be­lieve that you and I share a com­mon un­der­stand­ing of.
  
  Push­meet Kohli: That’s a key as­sump­tion that I’ve made. Will I be able to also com­mu­ni­cate with the ma­chine in those same con­cepts? Does the ma­chine un­der­stand those con­cepts? This is a key ques­tion that we have to try to think about. At the mo­ment we’re just say­ing, oh in­put, this is the out­put, in­put this out­put that. This is a very poor form of teach­ing. If you’re try­ing to teach an in­tel­li­gent sys­tem, just show­ing it ex­am­ples is a very poor form of teach­ing. There’s a much more richer, like when we are talk­ing about solv­ing a task, we are talk­ing in hu­man lan­guage and hu­man con­cepts.
  
  Robert Wiblin: It seems like you might think that it would be re­li­a­bil­ity en­hanc­ing to have bet­ter nat­u­ral lan­guage pro­cess­ing that, that’s go­ing to be dis­pro­por­tionately use­ful?
  
  Push­meet Kohli: Nat­u­ral pro­cess­ing would be use­ful, but the ground­ing prob­lem of does a ma­chine re­ally un­der­stand-
  
  Robert Wiblin: The con­cepts, or is it just pre­tend­ing, or is it just ap­ing it?
  
  Push­meet Kohli: Ex­actly.

  Speci­fi­ca­tion learn­ing is also ex­plic­itly called out in Towards Ro­bust and Ver­ified AI: Speci­fi­ca­tion Test­ing, Ro­bust Train­ing, and For­mal Ver­ifi­ca­tion (AN #52):

  Learn­ing speci­fi­ca­tions: Speci­fi­ca­tions that cap­ture “cor­rect” be­hav­ior in AI sys­tems are of­ten difficult to pre­cisely state. Build­ing sys­tems that can use par­tial hu­man speci­fi­ca­tions and learn fur­ther speci­fi­ca­tions from eval­u­a­tive feed­back would be re­quired as we build in­creas­ingly in­tel­li­gent agents ca­pa­ble of ex­hibit­ing com­plex be­hav­iors and act­ing in un­struc­tured en­vi­ron­ments.

  
  

• rohinmshah 6 Jun 2019 17:31 UTC

  5 points

  AF

  in reply to: DanielFilan's comment on: Con­di­tions for Mesa-Optimization

  You could imag­ine that, if you use gra­di­ent de­scent to reach a loss value of $L$, then amount of op­ti­miza­tion ap­plied in bits $=\log \frac{|\{\theta \in R^d:L(\theta )\le L\}|}{|R^d|}$ . (Yes, I know I shouldn’t be tak­ing sizes of con­tin­u­ous vec­tor spaces, but you know what I mean.)

[AN #57] Why we should fo­cus on ro­bust­ness in AI safety, and the analo­gous prob­lems in programming

• rohinmshah 5 Jun 2019 16:30 UTC

  3 points

  AF

  in reply to: ofer's comment on: Con­di­tions for Mesa-Optimization

  Yeah, agreed; I meant that claim to ap­ply to “re­al­is­tic” tasks (which I don’t yet know how to define).

• rohinmshah 5 Jun 2019 0:44 UTC

  2 points

  AF

  in reply to: vlad_m's comment on: Con­di­tions for Mesa-Optimization

  That makes sense, thanks.

• rohinmshah 5 Jun 2019 0:43 UTC

  2 points

  AF

  in reply to: vlad_m's comment on: Con­di­tions for Mesa-Optimization

  Just to make sure I un­der­stand, this ex­am­ple as­sumes that the base ob­jec­tive is “pre­dict hu­man be­hav­ior”, and doesn’t ap­ply to most base ob­jec­tives, right?