TurnTrout

• TurnTrout 13 Nov 2019 17:18 UTC

  6 points

  on: TurnTrout’s short­form feed

  Yes­ter­day, I put the finish­ing touches on my chef d’œu­vre, a se­ries of im­por­tant safety-rele­vant proofs I’ve been striv­ing for since early June. Strangely, I felt a great ex­haus­tion come over me. Th­ese proofs had been my ob­ses­sion for so long, and now—now, I’m done.

  I’ve had this feel­ing be­fore; three years ago, I stud­ied fer­vently for a Google in­ter­view. The literal mo­ment the in­ter­view con­cluded, a fever over­took me. I was sick for days. All the stress and ex­pec­ta­tion and readi­ness-to-fight which had been pent up, re­leased.

  I don’t know why this hap­pens. But right now, I’m still a lit­tle tired, even af­ter get­ting a good night’s sleep.

• TurnTrout 6 Nov 2019 16:32 UTC

  2 points

  in reply to: Wei_Dai's comment on: Ro­hin Shah on rea­sons for AI optimism

  Not on topic, but from the first ar­ti­cle:

  In re­al­ity, much of the suc­cess of a gov­ern­ment is due to the role of the par­tic­u­lar lead­ers, par­tic­u­lar peo­ple, and par­tic­u­lar places. If you have a mostly illiter­ate na­tion, di­vided 60%/​40% into two tribes, then ma­jori­tar­ian democ­racy is a re­ally, re­ally bad idea. But if you have a ho­mo­ge­neous, ed­u­cated, and savvy pop­u­lace, with a net­work of pri­vate in­sti­tu­tions, and a high-trust cul­ture, then many forms of gov­ern­ment will work quite well. Much of the pur­ported suc­cess of democ­racy is re­ally sur­vivor­ship bias. Coun­tries with the most hu­man cap­i­tal and strongest civic in­sti­tu­tions can sur­vive the chaos and dem­a­goguery that comes with reg­u­lar mass elec­tions. Lesser coun­tries suc­cumb to chaos, and then dic­ta­tor­ship.

  I don’t see why hav­ing a well-ed­u­cated pop­u­lace would pro­tect you against the nigh-in­evitable value drift of even well-in­ten­tioned lead­ers when they as­cend to power in a highly au­thor­i­tar­ian regime.

• TurnTrout 4 Nov 2019 16:28 UTC

  LW: 3 AF: 2

  AF

  in reply to: johnswentworth's comment on: But ex­actly how com­plex and frag­ile?

  There’s a dis­tinc­tion worth men­tion­ing be­tween the frag­ility of hu­man value in con­cept space, and the frag­ility in­duced by a hard max­i­mizer run­ning af­ter its proxy as fast as pos­si­ble.

  Like, we could have a dis­tance met­ric whereby hu­man value is dis­con­tin­u­ously sen­si­tive to nudges in con­cept space, while still be­ing OK prac­ti­cally (if we figure out eg mild op­ti­miza­tion). Like­wise, if we have a re­ally hard max­i­mizer pur­su­ing a mostly-ro­bust proxy of hu­man val­ues, and hu­man value is pretty ro­bust it­self, bad things might still hap­pen due to im­ple­men­ta­tion er­rors (the AI is in­cor­rigibly try­ing to ac­crue hu­man value for it­self, in­stead of helping us do it).

• TurnTrout 4 Nov 2019 1:29 UTC

  10 points

  on: TurnTrout’s short­form feed

  With re­spect to the in­te­gers, 2 is prime. But with re­spect to the Gaus­sian in­te­gers, it’s not: it has fac­tor­iza­tion $2=(1-i)(1+i)$. Here’s what’s hap­pen­ing.

  You can view com­plex mul­ti­pli­ca­tion as scal­ing and ro­tat­ing the com­plex plane. So, when we take our unit vec­tor 1 and mul­ti­ply by $(1+i)$, we’re scal­ing it by $|1+i|=\sqrt{2}$ and ro­tat­ing it coun­ter­clock­wise by ${45}^{\circ }$:

  

  This gets us to the pur­ple vec­tor. Now, we mul­ti­ply by $(1-i)$, scal­ing it up by $\sqrt{2}$ again (in green), and ro­tat­ing it clock­wise again by the same amount. You can even deal with the scal­ing and ro­ta­tions sep­a­rately (scale twice by $\sqrt{2}$, with zero net ro­ta­tion).

• TurnTrout 29 Oct 2019 22:34 UTC

  2 points

  in reply to: matthewp's comment on: Fetch The Coffee!

  I guess I’m not clear what the theta is for (maybe I missed some­thing, in which case I apol­o­gize). Is there one ini­tial ac­tion: how close it goes? And it’s trained to max­i­mize an eval­u­a­tion func­tion for its prox­im­ity, with just theta be­ing the pa­ram­e­ter?

  That as­sump­tion is do­ing a lot of work, it’s not clear what is packed into that, and it may not be suffi­cient to prove the ar­gu­ment.

  Well, my rea­son­ing isn’t pub­li­cly available yet, but this is in fact suffi­cient, and the as­sump­tion can be for­mal­ized. For any MDP, there is a dis­count rate $\gamma$, and for each re­ward func­tion there ex­ists an op­ti­mal policy ${\pi }^{\ast }$ for that dis­count rate. I’m claiming that given $\gamma$ suffi­ciently close to 1, op­ti­mal poli­cies likely end up gain­ing power as an in­stru­men­tally con­ver­gent sub­goal within that MDP.

  (All of this can be for­mally defined in the right way. If you want the proof, you’ll need to hold tight for a while)

• TurnTrout 29 Oct 2019 20:14 UTC

  2 points

  in reply to: matthewp's comment on: Fetch The Coffee!

  It’s still go­ing to act in­stru­men­tally con­ver­gently within the MDP it thinks it’s in. If you’re as­sum­ing it thinks it’s in a differ­ent MDP that can’t pos­si­bly model the real world, or if it is in the real world but has an empty ac­tion set, then you’re right—it won’t be­come an over­lord. But if we have a y-prox­im­ity max­i­mizer which can ac­tu­ally com­pute an op­ti­mal policy that’s far­sighted, over a state space that is “close enough” to rep­re­sent­ing the real world, then it does take over.

  The thing that’s fuzzy here is “agent act­ing in the real world”. In his new book, Rus­sell (as I un­der­stand it) ar­gues that an AGI trained to play Go could figure out it was just play­ing a game via sen­sory dis­crep­an­cies, and then start wire­head­ing on the “won a Go game” sig­nal. I don’t know if I buy that yet, but you’re cor­rect that there’s some kind of fuzzy bound­ary here. If we knew what ex­actly it took to get a “suffi­ciently good model”, we’d prob­a­bly be a lot closer to AGI.

  But Rus­sell’s origi­nal ar­gu­ment as­sumes the rele­vant fac­tors are within the model.

  If, in that MDP, there is an­other “hu­man” who has some prob­a­bil­ity, how­ever small, of switch­ing the agent off, and if the agent has available a but­ton that switches off that hu­man, the agent will nec­es­sar­ily press that but­ton as part of the op­ti­mal solu­tion for fetch­ing the coffee.

  I think this is a rea­son­able as­sump­tion, but we need to make it ex­plicit for clar­ity of dis­course. Given that as­sump­tion (and the as­sump­tion that an agent can com­pute a far­sighted op­ti­mal policy), in­stru­men­tal con­ver­gence fol­lows.

• TurnTrout 29 Oct 2019 17:51 UTC

  15 points

  on: The First Rung: In­sights from ‘Lin­ear Alge­bra Done Right’

  There are two cats where I live. Some­times, I watch them me­an­der around; it’s fas­ci­nat­ing to think how they go about their lives to­tally oblivi­ous to the true na­ture of the world around them. The above in­com­putabil­ity re­sult (ETA: the in­sol­u­bil­ity of the quin­tic) sur­prised me so much that I have be­gun to sus­pect that I too am a clue­less cat (un­til I learn com­plex anal­y­sis; you’ll ex­cuse me for hav­ing a few other text­books to study first).

  So I fi­nally got around to learn­ing Galois the­ory /​ group the­ory, and I now un­der­stand this in­sol­u­bil­ity re­sult (al­though my book stated some field the­o­retic re­sults with­out proof). It’s re­ally, re­ally cool, and I’m feel­ing a lot of things about it right now.

  But also… 2018-Trout was just so adorably ig­no­rant! Not only did I make the goof that daoza­ich pointed out, but that ver­sion of me also called the re­sult an “in­com­putabil­ity” re­sult!

  You know the feel­ing you get when you find a draw­ing you made your mom when you were five, and it’s full of adorable lit­tle spel­ling mis­takes? Yeah, that’s what I’m feel­ing right now.

• TurnTrout 29 Oct 2019 16:44 UTC

  2 points

  in reply to: matthewp's comment on: Fetch The Coffee!

  In which case I think it would be wise for some­one with Rus­sell’s views not to call the op­po­si­tion stupid. Or to as­sert that the po­si­tion is triv­ial. When in fact the ar­gu­ment might come down to fairly nu­anced points about nat­u­ral lan­guage un­der­stand­ing, com­pre­hen­sion, com­pe­tence, cor­rigi­bil­ity etc. As far as I can tell from limited read­ing, the ar­gu­ments around how tightly bun­dled these things may be are not wa­ter­tight.

  I agree from a gen­eral con­vinc­ing-peo­ple stand­point that call­ing dis­cus­sants stupid is a bad idea. How­ever, I think it is in­deed quite ob­vi­ous if framed prop­erly, and I don’t think the ar­gu­ment needs to come down to nu­anced points, as long as we agree on the agent de­sign we’re talk­ing about—the Roomba is not a far­sighted re­ward max­i­mizer, and is im­plied to be trained in a pretty weak fash­ion.

  Sup­pose an agent is in­cen­tivized to max­i­mize re­ward. That means it’s in­cen­tivized to be max­i­mally able to get re­ward. That means it will work to stay able to get as much re­ward as pos­si­ble. That means if we mess up, it’s work­ing against us.

  I think the main point of dis­agree­ment here is goal-di­rect­ed­ness, but if we as­sume RL as the thing that gets us to AGI, the in­stru­men­tal con­ver­gence case is open and shut.

• TurnTrout 26 Oct 2019 21:01 UTC

  5 points

  on: Fetch The Coffee!

  Here are my read­ings of the ar­gu­ments.

  Stu­art claims that if you give the sys­tem a fixed ob­jec­tive $R$, then it’s in­cen­tivized to en­sure it can achieve that goal. Nat­u­rally, it takes any ad­van­tage to stop us from stop­ping it from best achiev­ing the fixed ob­jec­tive.

  Yann’s re­sponse reads as “who would pos­si­bly be so dumb as to build a sys­tem which would be in­cen­tivized to stop us from stop­ping it?”.

  Now, I also think this re­sponse is weak. But let’s con­sider whether this is a rea­son­able re­sponse, even if it seemed rea­son­able to be con­fi­dent that it will be easy to see when sys­tems are flawed, and easy to build safe­guards; nei­ther is re­motely likely to be true, in my opinion.

  Sup­pose you have ac­cess to a com­puter ter­mi­nal. You have a math­e­mat­i­cal proof that if you type ran­dom char­ac­ters into the ter­mi­nal and then press En­ter, one of the most likely out­comes is that it ex­plodes and kills you. Now, a re­sponse analo­gous to Yann’s is: “who would be so dumb as to type ran­dom things into this com­puter? I won’t. Time to type my next pa­per.”.

  I think a wiser re­sponse would be to ask what about this com­puter kills you if you type in ran­dom stuff, and think very, very care­fully be­fore you type any­thing. If you have text you want to type, you should be check­ing it against your de­tailed gears-level model of the com­puter to as­sure your­self that it won’t trig­ger what­ever-causes-ex­plo­sions. It’s not enough to say that you won’t type ran­dom stuff.

  ETA: In par­tic­u­lar, if, af­ter learn­ing of this proof, you find your­self still ex­actly as en­thu­si­as­tic to type as you were be­fore—well, we’ve dis­cussed that failure mode be­fore.

• TurnTrout 26 Oct 2019 20:16 UTC

  LW: 3 AF: 2

  AF

  in reply to: Isnasene's comment on: All I know is Goodhart

  I think you’re us­ing Able #2 (which makes sense—it’s how the word is used col­lo­quially). I tend to use Able #1 (be­cause I read a lot about de­ter­minism when I was younger). I might be wrong about this though be­cause you made a similar dis­tinc­tion be­tween phys­i­cal ca­pa­bil­ity and an­ti­ci­pated pos­si­bil­ity like this in Gears of Im­pact:

  I am us­ing #2, but I’m aware that there’s a sep­a­rate #1 mean­ing (and thank you for dis­t­in­guish­ing be­tween them so clearly, here!).

• TurnTrout 26 Oct 2019 16:16 UTC

  LW: 3 AF: 2

  AF

  in reply to: Isnasene's comment on: All I know is Goodhart

  Well, they’re anti-cor­re­lated across differ­ent agents. But from the same agent’s per­spec­tive, they may still be able to max­i­mize their own red-see­ing, or even hu­man red-see­ing—they just won’t. (This will be in the next part of my se­quence on im­pact).

• TurnTrout 26 Oct 2019 0:42 UTC

  12 points

  in reply to: Stuart Anderson's comment on: What are some un­pop­u­lar (non-nor­ma­tive) opinions that you hold?

  The CDC has data on promis­cu­ity and pair bond­ing (https://​​www.cdc.gov/​​nchs/​​data/​​nhsr/​​nhsr036.pdf). I can find no data that in­di­cates neu­tral or pos­i­tive effects for promis­cu­ity.

  The study fo­cuses on STI rates. How would promis­cu­ity pos­si­bly im­prove these rates?

  It seems to me that when pressed on the com­pact claim you made above, you just made a lot more claims (as pjeby pointed out).

• TurnTrout 25 Oct 2019 16:55 UTC

  LW: 3 AF: 2

  AF

  in reply to: Isnasene's comment on: All I know is Goodhart

  Let me clar­ify the dis­tinc­tion I’m try­ing to point at:

  First, Good­hart’s law ap­plies to us when we’re op­ti­miz­ing a goal for our­selves, but we don’t know the ex­act goal. For ex­am­ple, if I’m try­ing to make my­self happy, I might find a proxy of danc­ing, even though danc­ing isn’t liter­ally the global op­ti­mum. This uses up time I could have used on the ac­tual best solu­tion. This can be bad, but it doesn’t seem that bad. I’m pretty cor­rigible to my­self.

  Se­cond, Good­hart’s law ap­plies to other agents who are in­structed to max­i­mize some proxy of what we want. This is bad. If it’s max­i­miz­ing the proxy, then it’s en­sur­ing it’s most able to max­i­mize the proxy, which means it’s in­cen­tivized to stop us from do­ing things (un­less the proxy speci­fi­cally in­cludes that—which safe­guard is also vuln­er­a­ble to mis­speci­fi­ca­tion; or is some­how oth­er­wise more in­tel­li­gently de­signed than the stan­dard re­ward-max­i­miza­tion model). The agent is pur­su­ing the proxy from its own per­spec­tive, not from ours.

  I think this en­tropy­ish thing is also why Stu­art’s makes his point that Good­hart ap­plies to hu­mans and not in gen­eral: It’s only be­cause of the unique state hu­mans are in (ex­ist­ing in a low en­tropy uni­verse, hav­ing an un­usu­ally large amount of power) that Good­hart tends to hit us af­fect us.

  Ac­tu­ally, I think I have I have a more pre­cise de­scrip­tion of the en­tropy­ish thing now. Good­hart’s Law isn’t driven by en­tropy; Good­hart’s Law is driven by try­ing to op­ti­mize a util­ity func­tion that already has an un­usu­ally high value rel­a­tive to what you’d ex­pect from your uni­verse. En­tropy just hap­pens to be a rea­son­able proxy for it some­times.

  I don’t think the in­tial value has much to do with what you la­bel the “AIS ver­sion” of Good­hart (nei­ther does the com­plex­ity of hu­man val­ues in par­tic­u­lar). Imag­ine we had a re­ward func­tion that gave one point of re­ward for each cone de­tect­ing red; re­ward is dis­pensed once per sec­ond. Imag­ine that the uni­verse is presently low-value; for what­ever rea­son, red stim­u­la­tion is hard to find. Good­hart’s law still ap­plies to agents we build to en­sure we can see red for­ever, but it doesn’t ap­ply to us di­rectly—we pre­sum­ably de­duce our true re­ward func­tion, and no longer rely on prox­ies to max­i­mize it.

  The rea­son it ap­plies to agents we build is that not only do you have to en­code the re­ward func­tion, but we have to point to peo­ple! This does not have a short de­scrip­tion length. With re­spect to hard max­i­miz­ers, a sin­gle mis­step means the agent is now show­ing it­self red, or some­thing.

  How prox­ies in­ter­act is worth con­sid­er­ing, but (IMO) it’s far from the main rea­son for Good­hart’s law be­ing re­ally, re­ally bad in the con­text of AI safety.

• TurnTrout 23 Oct 2019 15:32 UTC

  LW: 4 AF: 2

  AF

  in reply to: Isnasene's comment on: All I know is Goodhart

  The ex­pla­na­tion is a bit sim­pler than this. The agent has one goal, and we have other goals. It gains power to best com­plete its goal by tak­ing power away from us. There­fore, any uni­verse where we have an effec­tive max­i­mizer of some­thing mis­speci­fied is a uni­verse where we’re no longer able to get what we want. That’s why in­stru­men­tal con­ver­gence is so bad.

• TurnTrout 23 Oct 2019 15:27 UTC

  19 points

  in reply to: Stuart Anderson's comment on: What are some un­pop­u­lar (non-nor­ma­tive) opinions that you hold?

  I do not ac­cept this premise, and I’m sur­prised that it seems to you to be “al­most un­con­sciously ac­cepted by ev­ery­one” you’ve raised it with.

• TurnTrout 23 Oct 2019 3:12 UTC

  3 points

  in reply to: An1lam's comment on: An1lam’s Short Form Feed

  Although I haven’t used Anki for math, it seems to me like I want to build up con­cepts and com­pe­ten­cies, not re­mem­ber defi­ni­tions. Like, I couldn’t write down the defi­ni­tion of ab­solute con­ti­nu­ity, but if I got back in the zone and re­freshed my­self, I’d have all of my anal­y­sis skills in­tact.

  I sup­pose defi­ni­tions might be a use­ful scaf­fold­ing?

• TurnTrout 22 Oct 2019 19:35 UTC

  LW: 2 AF: 1

  AF

  in reply to: Ben Pace's comment on: De­duc­ing Impact

  Great re­sponses.

  What you’re in­fer­ring is im­pres­sively close to where the se­quence is lead­ing in some ways, but the fi­nal des­ti­na­tion is more in­di­rect and avoids the is­sues you rightly point out (with the ex­cep­tion of the “en­sur­ing the fu­ture is valuable” is­sue; I re­ally don’t think we can or should build eg low-im­pact yet am­bi­tious sin­gle­tons—more on that later).

• TurnTrout 22 Oct 2019 19:22 UTC

  4 points

  on: TurnTrout’s short­form feed

  I no­ticed I was con­fused and li­able to for­get my grasp on what the hell is so “nor­mal” about nor­mal sub­groups. You know what that means—col­or­ful pic­ture time!

  First, the clas­sic defi­ni­tion. A sub­group $H$ is nor­mal when, for all group el­e­ments $g$, $gH=Hg$ (this is triv­ially true for all sub­groups of abelian groups).

  

  

  ETA: I drew the bounds a bit in­cor­rectly; $g$ is most cer­tainly within the left coset ($ge=g$).

  No­tice that non­triv­ial cosets aren’t sub­groups, be­cause they don’t have the iden­tity $e$.

  This “nor­mal” thing mat­ters be­cause some­times we want to high­light reg­u­lar­i­ties in the group by tak­ing a quo­tient. Tak­ing an ex­am­ple from the ex­cel­lent Vi­sual Group The­ory, the in­te­gers $Z$ have a quo­tient group $Z/12$ con­sist­ing of the con­gru­ence classes $\overline{0},\dots ,\overline{11}$, each in­te­ger slot­ted into a class ac­cord­ing to its value mod 12. We’re tak­ing a quo­tient with the cyclic sub­group $⟨12⟩$.

  

  So, what can go wrong? Well, if the sub­group isn’t nor­mal, strange things can hap­pen when you try to take a quo­tient.

  

  Here’s what’s hap­pen­ing:

  

  Nor­mal­ity means that when you form the new Cayley di­a­gram, the ar­rows be­have prop­erly. You’re at the ori­gin, $e$. You travel to $Hg$ us­ing $g$. What we need for this di­a­gram to make sense is that if you fol­low any $h$ you please, ap­ply­ing $g^{-1}$ means you go back to $H$. In other words, $gh{g}^{-1}=h^{\prime }\in H$. In other words, $gh=h^{\prime }g$. In other other words (and us­ing a few prop­er­ties of groups), $gH=Hg$.

• TurnTrout 22 Oct 2019 15:37 UTC

  2 points

  in reply to: xrchz's comment on: Refram­ing Impact

  Ac­cessible in what way? I’m plan­ning to put up a full a text ver­sion at the end.

• TurnTrout 20 Oct 2019 20:52 UTC

  3 points

  in reply to: Ravi D'Elia's comment on: In­visi­ble Choices, Made by Default

  Start­ing with Duol­ingo might be OK, but it’s very bare-bones and un­like the real deal of hav­ing thoughts in a lan­guage and then ex­press­ing your­self. In fact, I recom­mend not us­ing it af­ter you have a rudi­men­tary grasp of the lan­guage.

  Try en­ter­ing full sen­tence-to-sen­tence cards in Anki. At first, you won’t be able to write good sen­tences on your own, so use a sen­tence-search ser­vice for that (I don’t re­mem­ber what they’re called, but at the least you can use the ex­am­ple wor­drefer­ence sen­tences). Over the course of a year, I in­put all the thoughts I’d had that day which I couldn’t trans­late. By learn­ing to pro­duce words in-con­text in­stead of mem­o­riz­ing the mul­ti­lin­gual dic­tio­nary, I be­gan to run out of ad­di­tions to the Anki deck. That ex­er­cise gave me near-fluency, which has per­sisted to this day.

  So, I think Anki is suited for nearly ev­ery­thing ex­cept oral skills. Plus, spaced rep­e­ti­tion saves time.

  Also, you might try an app like Tan­dem. If you’re sin­gle, try learn­ing to be in­ter­est­ing in the for­eign lan­guage—it’s nat­u­rally mo­ti­vat­ing, and both of you can chalk up any awk­ward­ness to mis­trans­la­tion. :)