ricraz

• ricraz 7 Jun 2019 14:49 UTC

  6 points

  AF

  on: Risks from Learned Op­ti­miza­tion: Introduction

  We will say that a sys­tem is an op­ti­mizer if it is in­ter­nally search­ing through a search space (con­sist­ing of pos­si­ble out­puts, poli­cies, plans, strate­gies, or similar) look­ing for those el­e­ments that score high ac­cord­ing to some ob­jec­tive func­tion that is ex­plic­itly rep­re­sented within the sys­tem.

  I ap­pre­ci­ate the difficulty of ac­tu­ally defin­ing op­ti­miz­ers, and so don’t want to quib­ble with this defi­ni­tion, but am in­ter­ested in whether you think hu­mans are a cen­tral ex­am­ple of op­ti­miz­ers un­der this defi­ni­tion, and if so whether you think that most mesa-op­ti­miz­ers will “ex­plic­itly rep­re­sent” their ob­jec­tive func­tions to a similar de­gree that hu­mans do.

• ricraz 3 Jun 2019 1:19 UTC

  3 points

  in reply to: Samuel Hapák's comment on: On alien science

  Agreed that this points in the right di­rec­tion. I think there’s more to it than that though. Con­sider for ex­am­ple a three-body prob­lem un­der New­to­nian me­chan­ics. Then there’s a sense in which spec­i­fy­ing the ini­tial masses and ve­loc­i­ties of the bod­ies, along with New­ton’s laws of mo­tion, is the best way to com­press the in­for­ma­tion about these chaotic tra­jec­to­ries.

  But there’s still an open ques­tion here, which is why are three-body sys­tems chaotic? Two-body sys­tems aren’t. What makes the differ­ence? Find­ing an ex­pla­na­tion prob­a­bly doesn’t al­low you to com­press any data any more, but it still seems im­por­tant and in­ter­est­ing.

  (This seems re­lated to a po­ten­tial mod­ifi­ca­tion of your data com­pres­sion stan­dard: that good ex­pla­na­tions com­press data in a way that min­imises not just stor­age space, but also the com­pu­ta­tion re­quired to un­pack the data. I’m a lit­tle con­fused about this though.)

On alien science

• ricraz 28 May 2019 16:32 UTC

  4 points

  in reply to: Raemon's comment on: Book re­view: The Sleep­walk­ers by Arthur Koestler

  Thanks for the kind words. I agree that re­fac­tor­ing would be use­ful, but don’t have the time now. I have added some head­ings though.

A shift in ar­gu­ments for AI risk

• ricraz 24 May 2019 14:42 UTC

  4 points

  on: “Other peo­ple are wrong” vs “I am right”

  A rele­vant book recom­men­da­tion: The Enigma of Rea­son ar­gues that think­ing of high-level hu­man rea­son­ing as a tool for at­tack­ing other peo­ple’s be­liefs and defend­ing our own (re­gard­less of their ac­tual ve­rac­ity) helps ex­plain a lot of weird asym­me­tries in cog­ni­tive bi­ases we’re sus­cep­ti­ble to, in­clud­ing this one.

• ricraz 22 May 2019 15:25 UTC

  6 points

  AF

  on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  I’d like to push back on the as­sump­tion that AIs will have ex­plicit util­ity func­tions. Even if you think that suffi­ciently ad­vanced AIs will be­have in a util­ity-max­imis­ing way, their util­ity func­tions may be en­coded in a way that’s difficult to for­mal­ise (e.g. some­where within a neu­ral net­work).

  It may also be the case that co­or­di­na­tion is much harder for AIs than for hu­mans. For ex­am­ple, hu­mans are con­strained by hav­ing bod­ies, which makes it eas­ier to pun­ish defec­tion—hid­ing from the gov­ern­ment is tricky! Our bod­ies also make anonymity much harder. Whereas if you’re a piece of code which can copy it­self any­where in the world, reneg­ing on agree­ments may be­come rel­a­tively easy. Another rea­sion why AI co­op­er­a­tion might be harder is sim­ply that AIs will be ca­pa­ble of a much wider range of goals and cog­ni­tive pro­cesses than hu­mans, and so they may be less pre­dictable to each other and/​or have less com­mon ground with each other.

• ricraz 22 May 2019 15:19 UTC

  6 points

  AF

  on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  This pa­per by Critch is rele­vant: it ar­gues that agents with differ­ent be­liefs will bet their fu­ture share of a merged util­ity func­tion, such that it skews to­wards who­ever’s pre­dic­tions are more cor­rect.

• ricraz 21 May 2019 23:54 UTC

  10 points

  in reply to: JacobKopczynski's comment on: What are the open prob­lems in Hu­man Ra­tion­al­ity?

  Which poli­cies in par­tic­u­lar?

• ricraz 21 May 2019 11:27 UTC

  2 points

  in reply to: Thrasymachus's comment on: What are the open prob­lems in Hu­man Ra­tion­al­ity?

  This point seems ab­solutely cru­cial; and I re­ally ap­pre­ci­ate the cited ev­i­dence.

[Question] Would an op­tion to pub­lish to AF users only be a use­ful fea­ture?

• ricraz 17 May 2019 10:56 UTC

  12 points

  in reply to: shminux's comment on: Which sci­en­tific dis­cov­ery was most ahead of its time?

  Ac­tu­ally, gen­eral rel­a­tivity seems to have been dis­cov­ered by Hilbert at al­most ex­actly the same time that Ein­stein did.

  https://​​en.wikipe­dia.org/​​wiki/​​Rel­a­tivity_pri­or­ity_dis­pute#Gen­eral_rel­a­tivity_2

• ricraz 16 May 2019 15:12 UTC

  2 points

  in reply to: ryan_b's comment on: Which sci­en­tific dis­cov­ery was most ahead of its time?

  Biggest jump for­ward.

• ricraz 16 May 2019 15:10 UTC

  6 points

  on: The Vuln­er­a­ble World Hy­poth­e­sis (by Bostrom)

  Does any­one know how this pa­per re­lates to Paul Chris­ti­ano’s blog post ti­tled Han­dling de­struc­tive tech­nol­ogy, which seems to pre­empt some of the key ideas? It’s not di­rectly ac­knowl­edged in the pa­per.

[Question] Which sci­en­tific dis­cov­ery was most ahead of its time?

• ricraz 15 May 2019 13:31 UTC

  8 points

  on: Eight Books To Read

  In­ter­est­ing list. How would you com­pare read­ing the best mod­ern sum­maries and analy­ses of the older texts, ver­sus read­ing them in the origi­nal?

  Quigley’s ca­reer demon­strates an ex­cel­lent piece of so­ciolog­i­cal method­ol­ogy… He builds a the­ory that em­pha­sizes the im­por­tance of elites, and sub­se­quently goes and talks to mem­bers of the elite to test and ap­ply the the­ory.

  I’m not sure if this is meant to be ironic, but that method­ol­ogy seems like an ex­cel­lent way to in­tro­duce con­fir­ma­tion bias. I guess it’s ex­cel­lent com­pared to not go­ing and talk­ing to any­one at all?

• ricraz 1 May 2019 1:31 UTC

  2 points

  in reply to: Donald Hobson's comment on: When is ra­tio­nal­ity use­ful?

  Depends what type of re­search. If you’re do­ing ex­per­i­men­tal cell biol­ogy, it’s less likely that your re­search will be ru­ined by ab­stract philo­soph­i­cal as­sump­tions which can’t be over­come by look­ing at the data.

• ricraz 1 May 2019 1:27 UTC

  2 points

  in reply to: norswap's comment on: When is ra­tio­nal­ity use­ful?

  So when is ra­tio­nal­ity rele­vant? Always! It’s liter­ally the sci­ence of how to make your life bet­ter /​ achiev­ing your val­ues.

  Some­times sci­ence isn’t helpful or use­ful. The sci­ence of how mu­sic works may be to­tally ir­rele­vant to ac­tual mu­si­ci­ans.

  If you think of in­stru­men­tal ra­tio­nal­ity of the sci­ence of how to win, then nec­es­sar­ily it en­tails con­sid­er­ing things like how to setup your en­vi­ron­ment, un­think­ing habits, how to “hack” into your psy­che/​emo­tions.

  It’s an em­piri­cal ques­tion when and whether these things are very use­ful; my post gives cases in which they are, and in which they aren’t.

• ricraz 26 Apr 2019 21:46 UTC

  3 points

  in reply to: Dagon's comment on: When is ra­tio­nal­ity use­ful?

  Some effort spent in de­ter­min­ing which things are good, and in which things lead to more op­por­tu­nity for good is go­ing to be re­warded (statis­ti­cally) with bet­ter out­comes.

  All else equal, do you think a ra­tio­nal­ist math­e­mat­i­cian will be­come more suc­cess­ful in their field than a non-ra­tio­nal­ist math­e­mat­i­cian? My guess is that if they spent the (fairly sig­nifi­cant) time taken to learn and do ra­tio­nal­ist things on just learn­ing more maths, they’d do bet­ter.

  (Here I’m ig­nor­ing the pos­si­bil­ity that learn­ing ra­tio­nal­ity makes them de­cide to leave the field).

  I’ll also wave at your wave at the re­cur­sion prob­lem: “when is ra­tio­nal­ity use­ful” is a fun­da­men­tally ra­tio­nal­ist ques­tion both in the sense of be­ing philo­soph­i­cal, and in the sense that an­swer­ing it is prob­a­bly not very use­ful for ac­tu­ally im­prov­ing your work in most fields.

• ricraz 26 Apr 2019 21:42 UTC

  3 points

  in reply to: Viliam's comment on: When is ra­tio­nal­ity use­ful?

  When I talk about do­ing use­ful work, I mean some­thing much more sub­stan­tial than what you out­line above. Ob­vi­ously 15 min­utes ev­ery day think­ing about your prob­lems is helpful, but the peo­ple at the lead­ing edges of most fields spend all day think­ing about their prob­lems.

  Per­haps do­ing this rit­ual makes you think about the prob­lem in a more meta way. If so, there’s an em­piri­cal ques­tion about how much be­ing meta can spark clever solu­tions. Here I have an in­tu­ition that it can, but when I look at any par­tic­u­lar sub­field that in­tu­ition be­comes much weaker. How much could a lead­ing math­e­mat­i­cian gain by be­ing more meta, for ex­am­ple?