Lanrian

• Lanrian 22 Oct 2019 19:15 UTC

  1 point

  in reply to: countedblessings's comment on: Sets and Functions

  As some­one with half an un­der­grads worth of math back­ground, I’ve found these posts use­ful to grasp the pur­pose and some of the ba­sics of cat­e­gory the­ory. It might be true that there’s ex­ist some ex­po­si­tion out there which would work bet­ter, but I haven’t found/​read that one, and I’m happy that this one ex­ists (among other things, it has the not-to-be-un­der­es­ti­mated virtue of be­ing un­effort­ful to read). Look­ing for­ward to the Yoneda and ad­junc­tion posts!

• Lanrian 22 Oct 2019 9:26 UTC

  LW: 1 AF: 1

  AF

  in reply to: John_Maxwell's comment on: The Dual­ist Pre­dict-O-Matic ($100 prize)

  Yes, that sounds more like re­in­force­ment learn­ing. It is not the de­sign I’m try­ing to point at in this post.

  Ok, cool, that ex­plains it. I guess the main differ­ences be­tween RL and on­line su­per­vised learn­ing is whether the model takes ac­tions that can af­fect their en­vi­ron­ment or only makes pre­dic­tions of fixed data; so it seems plau­si­ble that some­one train­ing the Pre­dict-O-Matic like that would think they’re do­ing su­per­vised learn­ing, while they’re ac­tu­ally closer to RL.

  That de­scrip­tion sounds a lot like SGD. I think you’ll need to be crisper for me to see what you’re get­ting at.

  No need, since we already found the point of dis­agree­ment. (But if you’re cu­ri­ous, the differ­ence is that sgd makes a change in the di­rec­tion of the gra­di­ent, and this one wouldn’t.)

• Lanrian 18 Oct 2019 10:58 UTC

  LW: 1 AF: 1

  AF

  in reply to: John_Maxwell's comment on: The Dual­ist Pre­dict-O-Matic ($100 prize)

  I think our dis­agree­ment comes from you imag­in­ing offline learn­ing, while I’m imag­in­ing on­line learn­ing. If we have a pre­defined set of (situ­a­tion, out­come) pairs, then the Pre­dict-O-Matic’s pre­dic­tions ob­vi­ously can’t af­fect the data that it’s eval­u­ated against (the out­come), so I agree that it’ll end up pretty du­al­is­tic. But if we put a Pre­dict-O-Matic in the real world, let it gen­er­ate pre­dic­tions, and then define the loss ac­cord­ing to what hap­pens af­ter­wards, a non-du­al­is­tic Pre­dict-O-Matic will be se­lected for over du­al­is­tic var­i­ants.

  If you still dis­agree with that, what do you think would hap­pen (in the limit of in­finite train­ing time) with an al­gorithm that just made a ran­dom change pro­por­tional to how wrong it was, at ev­ery train­ing step? Think­ing about SGD is a bit com­pli­cated, since it calcu­lates the gra­di­ent while as­sum­ing that the data stays con­stant, but if we use on­line train­ing on an al­gorithm that just tries things un­til some­thing works, I’m pretty con­fi­dent that it’d end up look­ing for fixed points.

• Lanrian 17 Oct 2019 9:24 UTC

  LW: 5 AF: 4

  AF

  on: The Dual­ist Pre­dict-O-Matic ($100 prize)

  If du­al­ism holds for Abram’s pre­dic­tion AI, the “Pre­dict-O-Matic”, its world model may hap­pen to in­clude this thing called the Pre­dict-O-Matic which seems to make ac­cu­rate pre­dic­tions—but it’s not spe­cial in any way and isn’t be­ing mod­eled any differ­ently than any­thing else in the world. Again, I think this is a pretty rea­son­able guess for the Pre­dict-O-Matic’s de­fault be­hav­ior. I sus­pect other be­hav­ior would re­quire spe­cial code which at­tempts to pin­point the Pre­dict-O-Matic in its own world model and give it spe­cial treat­ment (an “ego”).

  I don’t see why we should ex­pect this. We’re told that the Pre­dict-O-Matic is be­ing trained with some­thing like sgd, and sgd doesn’t re­ally care about whether the model it’s im­ple­ment­ing is du­al­ist or non-du­al­ist; it just tries to find a model that gen­er­ates a lot of re­ward. In par­tic­u­lar, this seems wrong to me:

  The Pre­dict-O-Matic doesn’t care about look­ing bad, and there’s noth­ing con­tra­dic­tory about it pre­dict­ing that it won’t make the very pre­dic­tion it makes, or some­thing like that.

  If the Pre­dict-O-Matic has a model that makes bad pre­dic­tion (i.e. looks bad), that model will be se­lected against. And if it ac­ci­den­tally stum­bled upon a model that could cor­rectly think about it’s own be­havi­our in a non-du­al­ist fash­ion, and find fixed points, that model would be se­lected for (since its pre­dic­tions come true). So at least in the limit of search and ex­plo­ra­tion, we should ex­pect sgd to end up with a model that finds fixed points, if we train it in a situ­a­tion where its pre­dic­tions af­fect the fu­ture.

  If we only train it on data where it can’t af­fect the data that it’s eval­u­ated against, and then freeze the model, I agree that it prob­a­bly won’t ex­hibit this kind of be­havi­our; is that the sce­nario that you’re think­ing about?

• Lanrian 9 Oct 2019 18:09 UTC

  LW: 9 AF: 3

  AF

  in reply to: Rob Bensinger's comment on: Mis­con­cep­tions about con­tin­u­ous takeoff

  Pos­si­bly you’d want to rule out (c) with your stipu­la­tion that the tests are “ro­bust”? But I’m not sure you can get tests that ro­bust.

  That sounds right. I was think­ing about an in­finitely ro­bust mis­al­ign­ment-or­a­cle to clar­ify my think­ing, but I agree that we’ll need to be very care­ful with any real-world-tests.

  If I imag­ine writ­ing code and us­ing the mis­al­ign­ment-or­a­cle on it, I think I mostly agree with Nate’s point. If we have the code and com­pute to train a su­per­hu­man ver­sion of GPT-2, and the or­a­cle tells us that any agent com­ing out from that train­ing pro­cess is likely to be mis­al­igned, we haven’t learned much new, and it’s not clear how to de­sign a safe agent from there.

  I imag­ine a mis­al­ign­ment-or­a­cle to be more use­ful if we use it dur­ing the train­ing pro­cess, though. Con­cretely, it seems like a mis­al­ign­ment-or­a­cle would be ex­tremely use­ful to achieve in­ner al­ign­ment in IDA: as soon as the AI be­comes mis­al­igned, we can ei­ther rewind the train­ing pro­cess and figure out what we did wrong, or di­rectly use the or­a­cle as a train­ing sig­nal that severely pun­ish any step that makes the agent mis­al­igned. Cou­pled with the abil­ity to iter­ate on de­signs, since we won’t ac­ci­den­tally blow up the world on the way, I’d guess that some­thing like this is more likely to work than . This idea is ex­tremely sen­si­tive to (c), though.

• Lanrian 9 Oct 2019 11:02 UTC

  LW: 3 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: Mis­con­cep­tions about con­tin­u­ous takeoff

  We might reach a state of knowl­edge when it is easy to cre­ate AIs that (i) mis­al­igned (ii) su­per­hu­man and (iii) non-sin­gu­lar (i.e. a sin­gle such AI is not stronger than the sum to­tal of hu­man­ity and al­igned AIs) but hard/​im­pos­si­ble to cre­ate al­igned su­per­hu­man AIs.

  My in­tu­ition is that it’d prob­a­bly be pretty easy to cre­ate an al­igned su­per­hu­man AI if we knew how to cre­ate non-sin­gu­lar, mis-al­igned su­per­hu­man AIs, and had cheap, ro­bust meth­ods to tell if a par­tic­u­lar AI was mis­al­igned. How­ever, it seems pretty plau­si­ble that we’ll end up in a state where we know how to cre­ate non-sin­gu­lar, su­per­hu­man AIs; strongly sus­pect that most/​all of them are mis-al­igned; but don’t have great meth­ods to tell whether any par­tic­u­lar AI is al­igned or mis-al­igned. Does that sound right to you?

• Lanrian 9 Oct 2019 9:06 UTC

  LW: 16 AF: 6

  AF

  on: Mis­con­cep­tions about con­tin­u­ous takeoff

  Se­cond, we could more-or-less deal with sys­tems which defect as they arise. For in­stance, dur­ing de­ploy­ment we could no­tice that some sys­tems are op­ti­miz­ing some­thing differ­ent than what we in­tended dur­ing train­ing, and there­fore we shut them down.

  Each in­di­vi­d­ual sys­tem won’t by them­selves carry more power than the sum of pro­jects be­fore it. In­stead, AIs will only be slightly bet­ter than the ones that came be­fore it, in­clud­ing any AIs we are us­ing to mon­i­tor the newer ones.

  If the sum of pro­jects from be­fore carry more power than the in­di­vi­d­ual sys­tem, such that it can’t win by defec­tion, there’s no rea­son for it to defect. It might just join the ranks of “pro­jects from be­fore”, and sub­tly try to al­ter fu­ture sys­tems to be similarly defec­tive, wait­ing for a fu­ture op­por­tu­nity to strike. If the way we build these things sys­tem­at­i­cally ren­ders them mis­al­igned, we’ll sooner or later end up with a ma­jor­ity of them be­ing mis­al­igned, at which point we can’t triv­ially use them to shut down defec­tors.

  (I agree that con­tin­u­ous take­off does give us more warn­ing, be­cause some sys­tems will pre­sum­ably defect early, es­pe­cially weaker ones. And IDA is kind of similar to this strat­egy, and could plau­si­bly work. I just wanted to point out that a naive im­ple­men­ta­tion of this doesn’t solve the prob­lem of treach­er­ous turns.)

• Lanrian 7 Oct 2019 13:03 UTC

  5 points

  on: What do the baby eaters tell us about ethics?

  In the se­quence in­tro­duc­tion Eliezer says it makes points about “nat­u­ral­is­tic metaethics” but I won­der what points are these speci­fi­cally, since af­ter read­ing the SEP page on moral nat­u­ral­ism https://​​plato.stan­ford.edu/​​en­tries/​​nat­u­ral­ism-moral/​​ I can’t re­ally figure out what the mind-in­de­pen­dent moral facts are in the story.

  I wouldn’t nec­es­sar­ily ex­pect Eliezer’s us­age to be con­sis­tent with Stan­ford’s en­try. LW in gen­eral and Eliezer in par­tic­u­lar are not great at us­ing words from aca­demic philos­o­phy in the same way that philoso­phers do (see e.g. “util­i­tar­i­anism”).

• Lanrian 6 Oct 2019 10:28 UTC

  6 points

  in reply to: MalcolmOcean's comment on: Against “Sys­tem 1” and “Sys­tem 2″ (sub­agent se­quence)

  The book is say­ing that the left hemi­sphere an­swers in­cor­rectly, in both cases! As I said, this is sur­pris­ing.

  That’s not just sur­pris­ing, that’s ab­surd. I can ab­solutely be­lieve the claim that the left hemi­sphere always takes what’s writ­ten for granted, and solves the syl­l­o­gism for­mally. But the claim here is that the left hemi­sphere pays care­ful at­ten­tion to the ques­tions, solves them cor­rectly, and then re­verses the an­swer. Why would it do that? No mechanism is pro­posed.

  I looked at the one pa­per that’s men­tioned in the quote (Deglin and Kins­bourne), and they never ask the sub­jects whether the syl­l­o­gisms are ‘struc­turally cor­rect’; they only ask about the truth. And their main con­clu­sion is that the left hemi­sphere always solves syl­l­o­gisms for­mally, not that it’s always wrong.

  If you’ve heard the bizarre sto­ries about pa­tients con­fab­u­lat­ing af­ter strokes (eg “my limb isn’t par­a­lyzed, I just don’t want to move it) this is al­most unilat­er­ally as­so­ci­ated with dam­age to the right hemi­sphere.

  In­ter­est­ing, I didn’t know this only hap­pened with the left hemi­sphere in­tact.

• Lanrian 4 Oct 2019 10:47 UTC

  7 points

  in reply to: MalcolmOcean's comment on: Against “Sys­tem 1” and “Sys­tem 2″ (sub­agent se­quence)

  But in a differ­ent situ­a­tion, where one is asked the differ­ent ques­tion “Is this syl­l­o­gism struc­turally cor­rect?”, even when the con­clu­sion flies in the face of one’s ex­pe­rience, it is the right hemi­sphere which gets the an­swer cor­rect, and the left hemi­sphere which is dis­tracted by the fa­mil­iar­ity of what it already thinks it knows, and gets the an­swer wrong.

  Wait what? Surely the left/​right hemi­spheres are ac­ci­den­tally re­versed here? Or is the book say­ing that the left hemi­sphere always an­swers in­cor­rectly, no mat­ter what ques­tion you ask?

• Lanrian 25 Sep 2019 18:39 UTC

  1 point

  in reply to: cousin_it's comment on: How can I re­frame my study mo­ti­va­tion?

  Similarly to TurnTrout’s point about the se­quences, learn­ing logic/​com­pu­ta­tion/​ML is cer­tainly rele­vant to and use­ful for AI safety, but there are things in Su­per­in­tel­li­gence which no com­puter sci­ence text­book will tell you. It’s cer­tainly valuable to pick the most use­ful re­sources within what­ever field you’re try­ing to study, but pick­ing your field based on which one has the best text­books seems mis­guided.

  Also, text­books typ­i­cally re­quire a great deal more effort than pop­u­lar sci­ence books, so if OP is strug­gling with mo­ti­va­tion for the lat­ter, text­books are likely to make things worse.

• Lanrian 24 Aug 2019 21:25 UTC

  LW: 1 AF: 1

  AF

  in reply to: Donald Hobson's comment on: Log­i­cal Op­ti­miz­ers

  Ah, sorry, I mis­read the ter­minol­ogy. I agree.

• Lanrian 23 Aug 2019 19:35 UTC

  LW: 6 AF: 4

  AF

  in reply to: Adele Lopez's comment on: Soft take­off can still lead to de­ci­sive strate­gic advantage

  Hm, my prior is that speed of learn­ing how stolen code works would scale along with gen­eral in­no­va­tion speed, though I haven’t thought about it a lot. On the one hand, learn­ing the ba­sics of how the code works would scale well with more au­to­mated test­ing, and a lot of fine­tun­ing could pre­sum­ably be au­to­mated with­out in­ti­mate knowl­edge. On the other hand, we might be in a paradigm where AI tech al­lows us to gen­er­ate lots of ar­chi­tec­tures to test, any­way, and the bot­tle­neck is for en­g­ineers to de­velop an in­tu­ition for them, which seems like the thing that you’re point­ing at.

• Lanrian 23 Aug 2019 18:47 UTC

  LW: 3 AF: 2

  AF

  in reply to: Vanessa Kosoy's comment on: For­mal­is­ing de­ci­sion the­ory is hard

  In INP, any re­in­force­ment learn­ing (RL) al­gorithm will con­verge to one-box­ing, sim­ply be­cause one-box­ing gives it the money. This is de­spite RL naively look­ing like CDT.

  Yup, like Cas­par, I think that model-free RL learns the EDT policy in most/​all situ­a­tions. I’m not sure what you mean with it look­ing like CDT.

  In New­comb’s para­dox CDT suc­ceeds but EDT fails. Let’s con­sider an ex­am­ple where EDT suc­ceeds and CDT fails: the XOR black­mail.

  Isn’t it the other way around? The one-boxer gets more money, but gives in to black­mail, and there­fore gets black­mailed in the first place.

• Lanrian 23 Aug 2019 18:29 UTC

  LW: 15 AF: 6

  AF

  on: Soft take­off can still lead to de­ci­sive strate­gic advantage

  Great post!

  One thing I no­ticed is that claim 1 speak about na­tion­states while most of the AI-bits speak about com­pa­nies/​pro­jects. I don’t think this is a huge prob­lem, but it seems worth look­ing into.

  It seems true that it’ll be nec­es­sary to lo­cal­ize the se­cret bits into sin­gle pro­jects, in or­der to keep things se­cret. It also seems true that such pro­jects could keep a lead on the or­der of months/​years.

  How­ever, note that this does no longer cor­re­spond to hav­ing a coun­try that’s 30 years ahead of the rest of the world. In­stead, it cor­re­sponds to hav­ing a coun­try with a sin­gle com­pany 30 years ahead of the world. The equiv­a­lent anal­ogy is: could a com­pany trans­ported 30 years back in time gain a de­ci­sive strate­gic ad­van­tage for it­self /​ what­ever coun­try it landed in?

  A few ar­gu­ments:

  • A sin­gle com­pany might have been able to bring back a sin­gle mil­i­tary tech­nol­ogy, which may or may not have been suffi­cient to turn the world, alone. How­ever, I think one can ar­gue that AI is more mul­ti­pur­pose than most tech­nolo­gies.

  • If the com­pany wanted to co­op­er­ate with its coun­try, there would be an im­ple­men­ta­tion lag af­ter the tech­nol­ogy was shared. In old times, this would per­haps cor­re­spond to build­ing the new ships/​planes. To­day, it might in­volve tak­ing AI ar­chi­tec­tures and train­ing them for par­tic­u­lar pur­poses, which could be more or less easy de­pend­ing on the gen­er­al­ity of the tech. (Maybe also scal­ing up hard­ware?) Dur­ing this time, it would be eas­ier for other pro­jects and coun­tries to steal the tech­nol­ogy (though of course, they would have im­ple­men­ta­tion lags of their own).

  • In the his­tor­i­cal case, one might worry that a mod­ern air­plane com­pany couldn’t pro­duce much use­ful things 30 years back in time, be­cause they re­lied on new ma­te­ri­als and prod­ucts from other com­pa­nies. Trans­lated to when AI-com­pa­nies de­vel­ops along with the world, this would high­light that the AI-com­pany could de­velop a 30-year-lead-equiv­a­lent in AI-soft­ware, but that might not cor­re­spond to a 30-year-lead-equiv­a­lent in AI-tech­nol­ogy, in­so­far as progress is largely driven by im­prove­ments to hard­ware or other pub­lic in­puts to the pro­cess. (Un­less the se­cret AI-pro­ject is also de­vel­op­ing hard­ware.) I don’t think this is very prob­le­matic: hard­ware progress seems to be slow­ing down, while soft­ware is speed­ing up (?), so if ev­ery­thing went faster things would prob­a­bly be more soft­ware driven?

  • Per­haps one could also ar­gue that a 3-year lead would trans­late to an even greater lead, be­cause of re­cur­sive self-im­prove­ment, in which case the com­pany would have an even greater lead over the rest of the world.

  Over­all, these points don’t seem too im­por­tant, and I think your claims still go through.

• Lanrian 23 Aug 2019 17:35 UTC

  LW: 4 AF: 2

  AF

  on: Log­i­cal Op­ti­miz­ers

  One prob­lem that could cause the search­ing pro­cess to be un­safe is if the prior con­tained a rel­a­tively large mea­sure of ma­lign agents. This could hap­pen if you used the uni­ver­sal prior, per Paul’s ar­gu­ment. Such agents could max­i­mize across the propo­si­tions you test them on, but do some­thing else once they think they’re de­ployed.

For­mal­is­ing de­ci­sion the­ory is hard

• Lanrian 10 Aug 2019 6:11 UTC

  5 points

  in reply to: Matthew Barnett's comment on: Is there a stan­dard dis­cus­sion of veg­e­tar­i­anism/​ve­g­anism?

  There’s still a stigma to be­ing ve­gan, so peo­ple are less likely to want to be friends with you, and your net­work­ing skills will suffer.

  Note that the op­po­site can also be true, es­pe­cially if your plan to im­prove the world in­volves en­gag­ing with the an­i­mal rights com­mu­nity, or other peo­ple who care about an­i­mals.

• Lanrian 9 Aug 2019 4:58 UTC

  11 points

  in reply to: Fluttershy's comment on: Can we re­ally pre­vent all warm­ing for less than 10B$ with the mostly side-effect free geo­eng­ineer­ing tech­nique of Marine Cloud Bright­en­ing?

  I pre­fer to re­serve “liter­ally ly­ing” for when peo­ple in­ten­tion­ally say things that are demon­stra­bly false. It’s use­ful to have words for that kind of thing. As long as things are plau­si­bly defen­si­ble, it seems bet­ter to say that he made “mis­lead­ing state­ments”, or some­thing like that.

  Ac­tu­ally, I’m not even sure that this was a par­tic­u­larly egre­gious er­ror. Given that they never say they’re go­ing to rank things af­ter the ex­plicit cost-effec­tive­ness es­ti­mates, not do­ing that seems quite rea­son­able to me. See for ex­am­ple givewell’s why we can’t take ex­pected value es­ti­mates liter­ally. All the ar­gu­ments in that ar­ti­cle should be even stronger when it’s differ­ent peo­ple mak­ing es­ti­mates across differ­ent ar­eas. If you think that peo­ple should “make a guess” even when they don’t have time to do more re­search, that’s a method­olog­i­cal dis­agree­ment with a non-ob­vi­ous an­swer.

  I still think it’s plau­si­ble that some of the economists were act­ing in bad faith (it’s cer­tainly bad that they don’t even give qual­iti­tive jus­tifi­ca­tions for some of their rank­ings). But when their ac­tions are plau­si­bly defen­si­ble in any par­tic­u­lar in­stance, you need sev­eral differ­ent pieces of ev­i­dence to be con­fi­dent of that (like where they get their fund­ing from, if they’re mak­ing sys­tem­atic er­rors in the same di­rec­tion, etc). If some­one are say­ing things that I would clas­sify as “literal lies”, that’s sig­nifi­cantly stronger ev­i­dence that they’re act­ing in bad faith, which means you can skip over some of that ev­i­dence-gath­er­ing. I thought that you were claiming that Lom­borg had made such a state­ment, and the fact that he hadn’t makes a large differ­ence from my epistem­i­cal point of view, even if you have heard suffi­ciently much un­re­lated ev­i­dence to be­live that he’s sys­tem­at­i­cally act­ing in bad faith.

• Lanrian 8 Aug 2019 18:36 UTC

  6 points

  in reply to: Fluttershy's comment on: Can we re­ally pre­vent all warm­ing for less than 10B$ with the mostly side-effect free geo­eng­ineer­ing tech­nique of Marine Cloud Bright­en­ing?

  He also liter­ally lies in his cost-benefit anal­y­sis (by more than 10x).

  What’s the literal lie, here? The link seems to say that a group led by Lom­borg made mis­lead­ing state­ments about how they made their pri­ori­ti­sa­tions, but I can’t see any out­right false­hoods.