interstice

• interstice 28 Sep 2019 0:50 UTC

  16 points

  in reply to: lionhearted's comment on: Honor­ing Petrov Day on LessWrong, in 2019

  Can’t tell if jok­ing, but they prob­a­bly mean that they were “ac­tu­ally in the mafia” in the game, so not in the real-world mafia.

• interstice 18 Sep 2019 14:47 UTC

  3 points

  on: Ac­cel­er­at­ing the cap­i­tal­is­tic ten­den­cies of so­cial sys­tems to do good

  A bet­ter sys­tem won’t just mag­i­cally form it­self af­ter the ex­ist­ing sys­tem has been de­stroyed. In all like­li­hood what will form will be ei­ther a far more cor­rupt and oli­garchi­cal sys­tem, or no sys­tem at all. I think a bet­ter tar­get for in­ter­ven­tion would be at­tempt­ing to build su­pe­rior al­ter­na­tives so that some­thing is available when the ex­ist­ing sys­tems start to fail. In ed­u­ca­tion for ex­am­ple, Lambda School is pro­vid­ing a bet­ter way for many peo­ple to learn pro­gram­ming than col­lege.

  Note also that ex­ist­ing sys­tems of power are very big, so efforts to dam­age them prob­a­bly have low marginal im­pact. Build­ing ini­tially small new things can have much higher marginal im­pact. If the sys­tems are as a cor­rupt as you think they are, they should de­stroy them­selves on their own in any case.

• interstice 2 Sep 2019 16:29 UTC

  5 points

  on: Hal­i­fax Meetup—Board Games

  Up­date—We’re go­ing to have a meetup on Septem­ber 21st at Un­com­mon Grounds (1030 South Park Street). This is go­ing to be posted on SlateS­tarCodex as part of the wor­ld­wide meetup event.

• interstice 22 Aug 2019 19:47 UTC

  LW: 1 AF: 1

  AF

  in reply to: steve2152's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  No wor­ries, I also missed the ear­lier posts when I wrote mine. There’s lots of stuff on this web­site.

  I en­dorse your rephras­ing of ex­am­ple 1. I think my po­si­tion is that it’s just not that hard to cre­ate a “self-con­sis­tent prob­a­bil­ity dis­tri­bu­tion”. For ex­am­ple, say you trained an RNN to pre­dict se­quences, like in this post. De­spite be­ing very sim­ple, it already im­plic­itly rep­re­sents a prob­a­bil­ity dis­tri­bu­tion over se­quences. If you train it with back-prop­a­ga­tion on a con­fus­ing ar­ti­cle in­volv­ing pyrite, then its weights will be up­dated to try to model the ar­ti­cle bet­ter. How­ever, if “pyrite” it­self was easy to pre­dict, then the weights that lead to it out­putting “pyrite” will *not* be up­dated. The same thing holds for mod­ern Trans­former net­works, which pre­dict the next to­ken based only on what it has seen so far. (Here is a pa­per with a re­cent ex­am­ple us­ing GPT-2. Note the de­gen­er­acy of max­i­mum like­li­hood sam­pling, but how this be­comes less of a prob­lem when just sam­pling from the im­plied dis­tri­bu­tion)

  I agree that this sort of ma­nipu­la­tive pre­dic­tion could be a prob­lem in prin­ci­ple, but it does not seem to oc­cur in re­cent ML sys­tems. (Although, there are some things which are some­what like this; the ear­lier pa­per I linked and mode col­lapse do in­volve ne­glect­ing high-en­tropy com­po­nents of the dis­tri­bu­tion. How­ever, the most straight­for­ward gen­er­a­tion and train­ing schemes do not in­cen­tivize this)

  For ex­am­ple 2, the point about gra­di­ent de­scent is this: while it might be the case that out­putting “Help I’m stuck in a GPU Fac­tory000” would ul­ti­mately re­sult in a higher ac­cu­racy, the way the gra­di­ent is prop­a­gated would not en­courage the agent to be­have ma­nipu­la­tively. This is be­cause, *lo­cally*, “Help I’m stuck in a GPU Fac­tory” de­creases ac­cu­racy, so that be­hav­ior(or poli­cies lead­ing to it) will be dis-in­cen­tivized by gra­di­ent de­scent. It may be the case that this will re­sult in eas­ier pre­dic­tions later, but the struc­ture of the re­ward func­tion does not lead to any op­ti­miza­tion pres­sure to­wards such ma­nipu­la­tive strate­gies. Learn­ing tak­ing place over high-level ab­strac­tions doesn’t change any­thing, be­cause any high-level ab­strac­tions lead­ing to lo­cally bad be­hav­ior will like­wise be dis-in­cen­tivized by gra­di­ent descent

• interstice 20 Aug 2019 14:44 UTC

  LW: 5 AF: 3

  AF

  on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  Ex­am­ple 1 ba­si­cally seems to be the prob­lem of out­put di­ver­sity in gen­er­a­tive mod­els. This can be a prob­lem in gen­er­a­tive mod­els, but there are ways around it. e.g. in­stead of out­putting the high­est-prob­a­bil­ity in­di­vi­d­ual se­quence, which will cer­tainly look “ma­nipu­la­tive” as you say, sam­ple from the im­plied dis­tri­bu­tion over se­quences. Then the sen­tence in­volv­ing “pyrite” will be out­put with prob­a­bil­ity pro­por­tional to how likely the model thinks “pyrite” is on its own, dis­re­gard­ing sub­se­quent to­kens.

  For ex­am­ple 2, I wrote a similar post a few months ago (and in fact, this idea seems to have been pro­posed and for­got­ten a few times on LW). But for gra­di­ent de­scent-based learn­ing sys­tems, I don’t think the effect de­scribed will take place.

  The rea­son is that gra­di­ent-de­scent-based sys­tems are only up­dated to­wards what they ac­tu­ally ob­serve. Let’s say we’re train­ing a sys­tem to pre­dict EU laws. If it pre­dicts “The EU will pass potato laws...” but sees “The EU will pass corn laws...” the pa­ram­e­ters will be up­dated to make “corn” more likely to have been out­put than “potato”. There is no ex­plicit global op­ti­miza­tion for pre­dic­tion ac­cu­racy.

  As you train to con­ver­gence, the pre­dic­tions of the model will at­tempt to ap­proach a fixed point, a set of pre­dic­tions that im­ply them­selves. How­ever, due to the lo­cal na­ture of the up­date, this fixed-point will not be se­lected to be globally min­i­mal, it will just be the first min­ima the model falls into. (This is differ­ent from the prob­lems with “lo­cal min­ima” you may have heard about in or­di­nary neu­ral net­work train­ing—those go away in the in­finite-ca­pac­ity limit, whereas lo­cal min­ima among fixed-points do not) The fixed-point should look some­thing like “what I would pre­dict if I out­put [what I would pre­dict if I out­put [what I would pre­dict .. ]]]” where the ini­tial pre­dic­tion is some ran­dom gib­ber­ish. This might look pretty weird, but it’s not op­ti­miz­ing for global pre­dic­tion ac­cu­racy.

• interstice 1 Aug 2019 2:40 UTC

  3 points

  in reply to: NikoTM's comment on: Hal­i­fax Meetup—Board Games

  Hey! It did hap­pen. So far there are 3 of us, we’ve been meet­ing up pretty reg­u­larly. If you’re in­ter­ested I can let you know the next time we’re plan­ning to meet up.

• interstice 24 Jun 2019 2:33 UTC

  5 points

  on: No Safe AI and Creat­ing Optionality

  What this would mean is that we would have to re­cal­ibrate our no­tion of “safe”, as what­ever defi­ni­tion has been proved im­pos­si­ble does not match our in­tu­itive per­cep­tion. We con­sider lots of stuff we have around now to be rea­son­ably safe, al­though we don’t have a for­mal proof of safety for al­most any­thing.

• interstice 12 May 2019 16:06 UTC

  2 points

  on: “UDT2” and “against UD+ASSA”

  In the mad sci­en­tist ex­am­ple, why would your mea­sure for the die land­ing 0 be 0.91? I think Solomonoff In­duc­tion would as­sign prob­a­bil­ity 0.1 to that out­come, be­cause you need an ex­tra $lo{g}_2\left(90\right)$ bits to spec­ify which clone you are. Or is this just meant to illus­trate a prob­lem with ASSA, UD not in­cluded?

Hal­i­fax Meetup—Board Games

• interstice 25 Mar 2019 20:57 UTC

  LW: 1 AF: 1

  AF

  in reply to: capybaralet's comment on: Pre­dic­tors as Agents

  Yeah, if you train the al­gorithm by ran­dom sam­pling, the effect I de­scribed will take place. The same thing will hap­pen if you use an RL al­gorithm to up­date the pa­ram­e­ters in­stead of an un­su­per­vised learn­ing al­gorithm(though it seems willfully per­verse to do so—you’re throw­ing away a lot of the struc­ture of the prob­lem by do­ing this, so train­ing will be much slower)

  I also just found an old com­ment which makes the ex­act same ar­gu­ment I made here. (Though it now seems to me that ar­gu­ment is not nec­es­sar­ily cor­rect!)

• interstice 4 Mar 2019 23:33 UTC

  4 points

  in reply to: John_Maxwell's comment on: Two Small Ex­per­i­ments on GPT-2

  If you liter­ally ran (a pow­ered-up ver­sion of) GPT-2 on “A brilli­ant solu­tion to the AI al­ign­ment prob­lem is...” you would get the sort of thing an av­er­age in­ter­net user would think of as a brilli­ant solu­tion to the AI al­ign­ment prob­lem. Try­ing to do this more use­fully ba­si­cally leads to Paul’s agenda (which is about try­ing to do imi­ta­tion learn­ing of an im­plicit or­ga­ni­za­tion of hu­mans)

Pre­dic­tors as Agents

• interstice 3 Jan 2019 21:22 UTC

  LW: 3 AF: 2

  AF

  in reply to: jessicata's comment on: Pre­dic­tors as Agents

  Reflec­tive Or­a­cles are a bit of a weird case case be­cause their ‘loss’ is more like a ⁰⁄₁ loss than a log loss, so all of the min­ima are ex­actly the same(If we take a sam­ple of 100000 uni­verses to score them, the differ­ence is merely in­cred­ibly small in­stead of 0). I was be­ing a bit glib refer­enc­ing them in the ar­ti­cle; I had in mind some­thing more like a model pa­ram­e­ter­iz­ing a dis­tri­bu­tion over out­puts, whose only in­fluence on the world is via a ran­dom sam­ple from this dis­tri­bu­tion. I think that such mod­els should in gen­eral have fixed points for similar rea­sons, but am not sure. Re­gard­less, these mod­els will, I be­lieve, favour fixed points whose dis­tri­bu­tions are easy to com­pute(But not fixed points with low en­tropy, that is they will pun­ish log­i­cal un­cer­tainty but not in­trin­sic un­cer­tainy). I’m plan­ning to run some ex­per­i­ments with VAEs and post the re­sults later.

• interstice 3 Jan 2019 3:49 UTC

  3 points

  on: Gen­er­al­is­ing CNNs

  You might be in­ter­ested in Trans­former Net­works, which use a learned pat­tern of at­ten­tion to route data be­tween lay­ers. They’re pretty pop­u­lar and have been used in some im­pres­sive ap­pli­ca­tions like this very con­vinc­ing image-syn­the­sis GAN.

  re: whether this is a good re­search di­rec­tion. The fact that neu­ral net­works are highly com­press­ible is very in­ter­est­ing and I too sus­pect that ex­ploit­ing this fact could lead to more pow­er­ful mod­els. How­ever, if your goal is to in­crease the chance that AI has a pos­i­tive im­pact, then it seems like the rele­vant thing is how quickly our un­der­stand­ing of how to al­ign AI sys­tems pro­gresses, rel­a­tive to our un­der­stand­ing of how to build pow­er­ful AI sys­tems. As de­scribed, this idea sounds like it would be more use­ful for the lat­ter.

• interstice 1 Jan 2019 21:19 UTC

  LW: 1 AF: 1

  AF

  in reply to: jessicata's comment on: Pre­dic­tors as Agents

  Is there a rea­son you think a re­flec­tive or­a­cle (or equiv­a­lent) can’t just be se­lected “ar­bi­trar­ily”, and will likely be se­lected to max­i­mize some score?

  The gra­di­ent de­scent is not be­ing done over the re­flec­tive or­a­cles, it’s be­ing done over some gen­eral com­pu­ta­tional model like a neu­ral net. Any highly-perform­ing solu­tion will nec­es­sar­ily look like a fixed-point-find­ing com­pu­ta­tion of some kind, due to the self-refer­en­tial na­ture of the pre­dic­tions. Then, since this fixed-point-fin­der is *in­ter­nal* to the model, it will be op­ti­mized for log loss just like ev­ery­thing else in the model.

  That is, the global op­ti­miza­tion of the model is dis­tinct from what­ever in­ter­nal op­ti­miza­tion the fixed-point-fin­der uses to choose the re­flec­tive or­a­cle. The global op­ti­miza­tion will fa­vor in­ter­nal op­ti­miz­ers that pro­duce fixed-points with good score. So while fixed-point-fin­ders in gen­eral won’t op­ti­mize for any­thing in par­tic­u­lar, the one this model uses will.

• interstice 31 Dec 2018 22:53 UTC

  3 points

  on: An­nounce­ment: AI al­ign­ment prize round 3 win­ners and next round

  I sub­mit Pre­dic­tors as Agents.

• interstice 30 Sep 2018 15:18 UTC

  LW: 1 AF: 1

  AF

  in reply to: Charlie Steiner's comment on: Reflec­tive AIXI and Anthropics

  If we as­sume Sleep­ing Beauty has lots of in­for­ma­tion, we might ex­pect that the short­est match­ing pro­gram will look like a simu­la­tion of phys­i­cal law plus a “bridg­ing law” that, given this simu­la­tion, tells you what sym­bols get writ­ten to the tape

  I agree. I still think that the prob­a­bil­ities would be closer to ¹⁄₂, ¹⁄₄, ¹⁄₄. The bridg­ing law could look like this: search over the uni­verse for com­pact en­cod­ings of my mem­o­ries so far, then see what is writ­ten next onto this en­cod­ing. In this case, it would take no more bits to spec­ify wak­ing up on Tues­day, be­cause the mem­o­ries are iden­ti­cal, in the same for­mat, and just slightly later tem­po­rally.

  In a nat­u­ral­ized set­ting, it seems like the tricky part would be get­ting the AIXI on Mon­day to care what hap­pens af­ter it goes to sleep. It ‘knows’ that it’s go­ing to lose con­scious­ness(it can see that its cur­rent mem­ory en­cod­ing is go­ing to be over­writ­ten) so its next pre­dic­tion is un­de­ter­mined by its world-model. There is one pro­gram that will give it the re­ward of its suc­ces­sor then ter­mi­nates, as I de­scribed above, but it’s not clear why the AIXI would favour that hy­poth­e­sis. Maybe if it has been in situ­a­tions in­volv­ing mem­ory-wiping be­fore, or has ob­served other RO-AIXI’s in such situ­a­tions.

• interstice 29 Sep 2018 19:19 UTC

  1 point

  in reply to: Ilya Shpitser's comment on: Deep learn­ing—deeper flaws?

  “I can’t make bets on my be­liefs about the Escha­ton, be­cause they are about the Escha­ton.” -- Well, it makes sense. Be­sides, I did offer you a bet tak­ing into ac­count a) that the money may be worth less in my branch b) I don’t think DL + RL AGI is more likely than not, just plau­si­ble. If you’re more than 96% cer­tain there will be no such AI, 20:1 odds are a good deal.

  But any­ways, I would be fine with bet­ting on a nearer-term challenge. How about—in 5 years, a bipedal robot that can run on rough ter­rain, as in this video, us­ing a policy learned from scratch by DL + RL(pos­si­bly in­clud­ing a simu­lated en­vi­ron­ment dur­ing train­ing) 1:1 odds.

• interstice 28 Sep 2018 17:07 UTC

  1 point

  in reply to: Ilya Shpitser's comment on: Deep learn­ing—deeper flaws?

  Hmmm...but if I win the bet then the world may be de­stroyed, or our en­vi­ron­ment could change so much the money will be­come worth­less. Would you take 20:1 odds that there won’t be DL+RL-based HLAI in 25 years?

• interstice 28 Sep 2018 16:59 UTC

  LW: 1 AF: 1

  AF

  in reply to: Charlie Steiner's comment on: Reflec­tive AIXI and Anthropics

  I still don’t see how you’re get­ting those prob­a­bil­ities. Say it takes 1 bit to de­scribe the out­come of the coin toss, and as­sume it’s easy to find all the copies of your­self(ie your mem­o­ries) in differ­ent wor­lds. Then you need:

  1 bit to spec­ify if the coin landed heads or tails

  If the coin landed tails, you need 1 more bit to spec­ify if it’s Mon­day or Tues­day.

  So AIXI would give these sce­nar­ios P(HM)=0.50, P(TM)=0.25, P(TT)=0.25.