Vanessa Kosoy(Vanessa Kosoy)

• Vanessa Kosoy 27 Sep 2020 16:16 UTC

  LW: 4 AF: 2

  AF

  in reply to: Vladimir_Nesov’s comment on: Vanessa Kosoy’s Shortform

  Another thing that might hap­pen is a data bot­tle­neck.

  Maybe there will be a good enough dataset to pro­duce a side­load that simu­lates an “av­er­age” per­son, and that will be enough to au­to­mate many jobs, but for a simu­la­tion of a com­pe­tent AI re­searcher you would need a more spe­cial­ized dataset that will take more time to pro­duce (since there are a lot less com­pe­tent AI re­searchers than peo­ple in gen­eral).

  More­over, it might be that the sam­ple com­plex­ity grows with the du­ra­tion of co­her­ent thought that you re­quire. That’s be­cause, un­less you’re train­ing di­rectly on brain in­puts/​out­puts, non-re­al­iz­able (com­pu­ta­tion­ally com­plex) en­vi­ron­ment in­fluences con­tam­i­nate the data, and in or­der to con­verge you need to have enough data to av­er­age them out, which scales with the length of your “epi­sodes”. In­deed, all con­ver­gence re­sults for Bayesian al­gorithms we have in the non-re­al­iz­able set­ting re­quire er­god­ic­ity, and there­fore the time of con­ver­gence (= sam­ple com­plex­ity) scales with mix­ing time, which in our case is de­ter­mined by epi­sode length.

  In such a case, we might dis­cover that many tasks can be au­to­mated by side­loads with short co­her­ence time, but AI re­search might re­quire sub­stan­tially longer co­her­ence times. And, simu­lat­ing progress re­quires by de­sign go­ing off-dis­tri­bu­tion along cer­tain di­men­sions which might make things worse.

• Vanessa Kosoy 27 Sep 2020 13:09 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vladimir_Nesov’s comment on: Vanessa Kosoy’s Shortform

  The imi­ta­tion plateau can definitely be rather short. I also agree that com­pu­ta­tional over­hang is the ma­jor fac­tor here. How­ever, a failure to cap­ture some of the in­gre­di­ents can be a cause of low com­pu­ta­tional over­head, whereas a suc­cess to cap­ture all of the in­gre­di­ents is a cause of high com­pu­ta­tional over­hang, be­cause the com­pute nec­es­sary to reach su­per­in­tel­li­gence might be very differ­ent in those two cases. Us­ing side­loads to ac­cel­er­ate progress might still re­quire years, whereas an “in­trin­sic” AGI might lead to the clas­si­cal “foom” sce­nario.

  EDIT: Although, since train­ing is typ­i­cally much more com­pu­ta­tion­ally ex­pen­sive than de­ploy­ment, it is likely that the first hu­man-level imi­ta­tors will already be sig­nifi­cantly sped-up com­pared to hu­mans, im­ply­ing that ac­cel­er­at­ing progress will be rel­a­tively easy. It might still take some time from the first pro­to­type un­til such an ac­cel­er­ate-the-progress pro­ject, but prob­a­bly not much longer than de­ploy­ing lots of au­toma­tion.

• Vanessa Kosoy 27 Sep 2020 11:46 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vladimir_Nesov’s comment on: Vanessa Kosoy’s Shortform

  I don’t see any strong ar­gu­ment why this path will pro­duce su­per­in­tel­li­gence. You can have a stream of thought that can­not be ac­cel­er­ated with­out in­vest­ing a pro­por­tional amount of com­pute, while a com­pletely differ­ent al­gorithm would pro­duce a far su­pe­rior “stream of thought”. In par­tic­u­lar, such an ap­proach can­not differ­en­ti­ate be­tween fea­tures of the stream of thought that are im­por­tant (mean­ing that they ad­vance to­wards the goal) and fea­tures of the stream of though that are unim­por­tant (e.g. differ­ent ways to phrase the same idea). This forces you to solve a task that is po­ten­tially much more difficult than just achiev­ing the goal.

• Vanessa Kosoy 27 Sep 2020 10:30 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vladimir_Nesov’s comment on: Vanessa Kosoy’s Shortform

  That is similar to gain­ing up­loads (bor­row­ing ter­minol­ogy from Egan, we can call them “side­loads”), but it’s not ob­vi­ous am­plifi­ca­tion/​dis­til­la­tion will work. In the model based on re­al­iz­abil­ity, the dis­til­la­tion step can fail be­cause the sys­tem you’re dis­till­ing is too com­pu­ta­tion­ally com­plex (hence, too un­re­al­iz­able). You can deal with it by up­scal­ing the com­pute of the learn­ing al­gorithm, but that’s not bet­ter than plain speedup.

• Vanessa Kosoy 26 Sep 2020 15:28 UTC

  LW: 18 AF: 7

  AF

  on: Vanessa Kosoy’s Shortform

  An AI progress sce­nario which seems pos­si­ble and which I haven’t seen dis­cussed: an imi­ta­tion plateau.

  The key ob­ser­va­tion is, imi­ta­tion learn­ing al­gorithms^[1] might pro­duce close-to-hu­man-level in­tel­li­gence even if they are miss­ing im­por­tant in­gre­di­ents of gen­eral in­tel­li­gence that hu­mans have. That’s be­cause imi­ta­tion might be a qual­i­ta­tively eas­ier task than gen­eral RL. For ex­am­ple, given enough com­put­ing power, a hu­man mind be­comes re­al­iz­able from the per­spec­tive of the learn­ing al­gorithm, while the world-at-large is still far from re­al­iz­able. So, an al­gorithm that only performs well in the re­al­iz­able set­ting can learn to imi­tate a hu­man mind, and thereby in­di­rectly pro­duce rea­son­ing that works in non-re­al­iz­able set­tings as well. Of course, liter­ally em­u­lat­ing a hu­man brain is still com­pu­ta­tion­ally formidable, but there might be mid­dle sce­nar­ios where the learn­ing al­gorithm is able to pro­duce a good-enough-in-prac­tice imi­ta­tion of sys­tems that are not too com­plex.

  This opens the pos­si­bil­ity that close-to-hu­man-level AI will ar­rive while we’re still miss­ing key al­gorith­mic in­sights to pro­duce gen­eral in­tel­li­gence di­rectly. Such AI would not be eas­ily scal­able to su­per­hu­man. Nev­er­the­less, some su­per­hu­man perfor­mance might be pro­duced by sped-up simu­la­tion, re­duc­ing noise in hu­man be­hav­ior and con­trol­ling the ini­tial con­di­tions (e.g. simu­lat­ing a hu­man on a good day). As a re­sult, we will have some pe­riod of time dur­ing which AGI is already here, au­toma­tion is in full swing, but there’s lit­tle or no fur­ther es­ca­la­tion. At the end of this pe­riod, the miss­ing in­gre­di­ents will be as­sem­bled (maybe with the help of AI re­searchers) and su­per­hu­man AI (pos­si­bly a fast take­off) be­gins.

  It’s in­ter­est­ing to try and work out the con­se­quences of such a sce­nario, and the im­pli­ca­tions on AI strat­egy.

  ---

  1. Such as GPT-n ↩︎

• Vanessa Kosoy 26 Sep 2020 13:14 UTC

  15 points

  in reply to: Chris_Leong’s comment on: Honor­ing Petrov Day on LessWrong, in 2020

  Wait, what? I’m con­fused by this com­ment. Did you want to nuke the front­page? It seems in­con­sis­tent with what you wrote in the other com­ment.

• Vanessa Kosoy 26 Sep 2020 13:07 UTC

  34 points

  in reply to: Neel Nanda’s comment on: Honor­ing Petrov Day on LessWrong, in 2020

  Notably the greater­wrong front­page still works. Maybe it’s like a Mars colony.

  What links here?

  • Sur­viv­ing Petrov Day by Mati_Roy (26 Sep 2020 16:40 UTC; 26 points)
  • Mati_Roy's comment on Honor­ing Petrov Day on LessWrong, in 2020 by Ben Pace (26 Sep 2020 15:36 UTC; 1 point)

• Vanessa Kosoy 24 Sep 2020 17:21 UTC

  LW: 4 AF: 3

  AF

  on: [AN #118]: Risks, solu­tions, and pri­ori­ti­za­tion in a world with many AI systems

  Link to ARCHES is bro­ken, cur­rent URL is this.

• Vanessa Kosoy 23 Sep 2020 18:46 UTC

  LW: 2 AF: 1

  AF

  on: Clar­ify­ing “What failure looks like” (part 1)

  Re­gard­ing “CAIS sce­nario” vs “gen­eral AI” sce­nar­ios, I think that there are strong forces push­ing to­wards the lat­ter. For any ac­tor that’s in­ter­ested in eco­nomic, poli­ti­cal or mil­i­tary gain, there are large re­turns on long-term plan­ning ap­plied to open-ended real-world prob­lems. There­fore there are strong in­cen­tives to cre­ate sys­tems ca­pa­ble of that. As you cor­rectly no­tice, such sys­tems will even­tu­ally con­verge to ex­tremely mal­i­cious strate­gies. So, there is a tragedy of the com­mons push­ing to­wards the de­ploy­ment of many pow­er­ful and gen­eral AI sys­tems. In the short-term these sys­tems benefit the ac­tors de­ploy­ing them, in the long-term they de­stroy all hu­man value.

• Vanessa Kosoy 22 Sep 2020 16:16 UTC

  LW: 12 AF: 7

  AF

  in reply to: Davidmanheim’s comment on: Needed: AI in­fo­haz­ard policy

  I’m not talk­ing about guidelines for the wider AI com­mu­nity. I’m talk­ing about guidelines for my own re­search (and pre­sum­ably other al­ign­ment re­searchers would be in­ter­ested in the same). The wider AI com­mu­nity doesn’t share my as­sump­tions about AI risk. In par­tic­u­lar, I be­lieve that most of what they’re do­ing is ac­tively harm­ful. There­fore, I don’t ex­pect them to ac­cept these guidelines, and I’m also mostly un­in­ter­ested in their in­put. More­over, it’s not the broader pub­lic that wor­ries me, but pre­cisely the broader AI com­mu­nity. It is from them that I want to with­hold things.

  Creat­ing any sort of guidelines that the wider com­mu­nity would also ac­cept is a differ­ent sort of challenge al­to­gether. It’s also a job for other peo­ple. Per­son­ally, I have enough on my plate as it is, and poli­tics is not my com­par­a­tive ad­van­tage by any mar­gin.

• Vanessa Kosoy 22 Sep 2020 13:34 UTC

  LW: 2 AF: 1

  AF

  in reply to: steve2152’s comment on: Needed: AI in­fo­haz­ard policy

  Well, yes, the “policy” is meant to be en­tirely vol­un­tary, since we don’t have a way to en­force it. At most, there can be a mechanism of “soft en­force­ment” in the sense that, if you pledge to fol­low the “policy”, this some­what in­creases the trust in you, and pre­sum­ably peo­ple who are suffi­ciently trust­wor­thy will be privy to some un­pub­lished knowl­edge and de­ci­sion mak­ing. How­ever, even then, the rules will in­evitably be some­what given to in­ter­pre­ta­tion and per­sonal judge­ment.

• Vanessa Kosoy 22 Sep 2020 9:40 UTC

  1 point

  AF

  in reply to: Donald Hobson’s comment on: Needed: AI in­fo­haz­ard policy

  Hmm, so in this model we as­sume that (i) the re­search out­put of the rest of the world is known (ii) we are de­cid­ing about one re­sult only (iii) the thresh­olds are un­known. In this case you are right that we need to com­pare our al­ign­ment : ca­pa­bil­ity ra­tio to the rest of the world’s al­ign­ment : ca­pa­bil­ity ra­tio.

  Now as­sume that, in­stead of just one re­sult over­all, you pro­duce a sin­gle re­sult ev­ery year. Most of the re­sults in the se­quence have al­ign­ment : ca­pa­bil­ity ra­tio way above the rest of the world, but then there is a year in which the ra­tio is only barely above the rest of the world. In this case, you are bet­ter off not pub­lish­ing the ir­reg­u­lar re­sult, even though the naive ra­tio crite­rion says to pub­lish. We can rec­on­cile it with the pre­vi­ous model by in­clud­ing your own re­search in the refer­ence, but it cre­ates a some­what con­fus­ing self-refer­ence.

  Se­cond, we can switch to mod­el­ing the re­search out­put of the rest of the world as a ran­dom walk. In this case, if the av­er­age di­rec­tion of progress is point­ing to­wards failure, then mov­ing along this di­rec­tion is net nega­tive, since it re­duces the chance to get suc­cess by luck.

• Vanessa Kosoy 22 Sep 2020 7:01 UTC

  LW: 2 AF: 1

  AF

  in reply to: Diffractor’s comment on: Needed: AI in­fo­haz­ard policy

  Re­gard­ing mak­ing a policy ahead of time, I think we can have an evolv­ing model of what in­gre­di­ents are miss­ing to get trans­for­ma­tive AI, and some rule of thumb that says how dan­ger­ous your re­sult is, given how much progress it makes to­wards each in­gre­di­ent (rele­vant but clearly in­suffi­cient < might or might not be suffi­cient < plau­si­bly a full solu­tion), how con­crete/​ac­tion­able it is (ab­stract idea < im­prac­ti­cal method < prac­ti­cal method) and how origi­nal/​sur­pris­ing it is (syn­the­sis of ideas in the field < im­prove­ment on idea in the field < ap­pli­ca­tion of idea out­side the field < com­pletely out of the blue).

  One prob­lem is, the model it­self might be an in­fo­haz­ard. This con­sid­er­a­tion pushes to­wards mak­ing the guidelines se­cret in them­selves, but that would make it much harder to de­bate and dis­sem­i­nate them. Also, the new re­sult might have ma­jor im­pli­ca­tions for the model. So, yes, cer­tainly there is no re­place­ment for the in­side view, but I still feel that we can have guidelines that help fo­cus­ing on the right con­sid­er­a­tions.

• Vanessa Kosoy 22 Sep 2020 6:18 UTC

  LW: 4 AF: 2

  AF

  in reply to: Davidmanheim’s comment on: Needed: AI in­fo­haz­ard policy

  The first thing I would note is that stake­hold­ers need to be in­volved in mak­ing any guidelines, and that push­ing for guidelines from the out­side is un­helpful, if not harm­ful, since it pushes par­ti­ci­pants to be defen­sive about their work.

  Hmm, maybe I was un­clear. When I said that “we need to have a pub­lic de­bate on the topic in­side the com­mu­nity” I meant, the com­mu­nity of AI al­ign­ment re­searchers. So, not from the out­side.

  As to the links, thank you. They do seem like po­ten­tially valuable in­puts into the de­bate, al­though (from skim­ming) they don’t seem to reach the point of propos­ing con­crete guidelines and pro­ce­dures.

• Vanessa Kosoy 21 Sep 2020 15:50 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen’s comment on: In­tro­duc­tion To The In­fra-Bayesi­anism Sequence

  Well, I agree that both for­mal­isms use max­imin so there might be some way to merge them. It’s definitely some­thing to think about.

Needed: AI in­fo­haz­ard policy

• Vanessa Kosoy 21 Sep 2020 14:15 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform

  Some prob­lems to work on re­gard­ing goal-di­rected in­tel­li­gence. Con­jec­ture 5 is es­pe­cially im­por­tant for de­con­fus­ing ba­sic ques­tions in al­ign­ment, as it stands in op­po­si­tion to Stu­art Arm­strong’s the­sis about the im­pos­si­bil­ity to de­duce prefer­ences from be­hav­ior alone.

  1. Con­jec­ture. In­for­mally: It is un­likely to pro­duce in­tel­li­gence by chance. For­mally: Denote $\Pi$ the space of de­ter­minis­tic poli­cies, and con­sider some $\mu \in \Delta \Pi$. Sup­pose $\mu$ is equiv­a­lent to a stochas­tic policy ${\pi }^{\ast }$. Then, $E_{\pi \sim \mu }\left[g\right(\pi \left)\right]=O\left(C\right({\pi }^{\ast }\left)\right)$.

  2. Find an “in­tel­li­gence hi­er­ar­chy the­o­rem”. That is, find an in­creas­ing se­quence $\left\{g_n\right\}$ s.t. for ev­ery $n$, there is a policy with goal-di­rected in­tel­li­gence in $(g_n,g_{n+1})$ (no more and no less).

  3. What is the com­pu­ta­tional com­plex­ity of eval­u­at­ing $g$ given (i) or­a­cle ac­cess to the policy or (ii) de­scrip­tion of the policy as a pro­gram or au­toma­ton?

  4. What is the com­pu­ta­tional com­plex­ity of pro­duc­ing a policy with given $g$?

  5. Con­jec­ture. In­for­mally: In­tel­li­gent agents have well defined pri­ors and util­ity func­tions. For­mally: For ev­ery $(U,\zeta )$ with $C\left(U\right)<\infty$ and $D_{KL}\left({\zeta }_0||\zeta \right)<\infty$, and ev­ery $ϵ>0$, there ex­ists $g\in (0,\infty )$ s.t. for ev­ery policy $\pi$ with in­tel­li­gence at least $g$ w.r.t. $(U,\zeta )$, and ev­ery $(\tilde{U},\tilde{\zeta })$ s.t.$\pi$ has in­tel­li­gence at least $g$ w.r.t. them, any op­ti­mal poli­cies ${\pi }^{\ast },{\tilde{\pi }}^{\ast }$ for $(U,\zeta )$ and $(\tilde{U},\tilde{\zeta })$ re­spec­tively satisfy $E_{\zeta {\tilde{\pi }}^{\ast }}\left[U\right]\ge E_{\zeta {\pi }^{\ast }}\left[U\right]-ϵ$.

• Vanessa Kosoy 21 Sep 2020 11:52 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen’s comment on: In­tro­duc­tion To The In­fra-Bayesi­anism Sequence

  After read­ing some of your pa­per, I think that they are ac­tu­ally very differ­ent. IIUC, you are talk­ing about pes­simism as a method to avoid traps, but you as­sume re­al­iz­abil­ity. On the other hand, in­fra-Bayesi­anism is (to first ap­prox­i­ma­tion) or­thog­o­nal to deal­ing with traps, in­stead it al­lows deal­ing with non­re­al­iz­abil­ity.

• Vanessa Kosoy 20 Sep 2020 9:30 UTC

  6 points

  on: EA Re­la­tion­ship Status

  Another fac­tor that might be in play is, if you’re mar­ried with chil­dren then you have re­spon­si­bil­ities to­wards your fam­ily, and that is an in­cen­tive against spend­ing re­sources on al­tru­is­tic causes.

• Vanessa Kosoy 18 Sep 2020 17:55 UTC

  8 points

  in reply to: Zack_M_Davis’s comment on: Causal Real­ity vs So­cial Reality

  My re­search is go­ing very well, thank you :)

  I guess that putting up such a post would make things much more fair, at least. But, I’m not sure I will be will­ing to com­ment on it pub­li­cly, given the risk of an­other drain of time and en­ergy.