So8res

• So8res 15 Nov 2019 18:22 UTC

  LW: 40 AF: 12

  AF

  on: What I’ll be do­ing at MIRI

  I have dis­cussed with MIRI their de­ci­sion to make their re­search non-dis­closed-by-de­fault and we agreed that my re­search agenda is a rea­son­able ex­cep­tion.

  Small note: my view of MIRI’s nondis­closed-by-de­fault policy is that if all re­searchers in­volved with a re­search pro­gram think it should ob­vi­ously be pub­lic then it should ob­vi­ously be pub­lic, and that doesn’t re­quire a bunch of bu­reau­cracy. I think this while si­mul­ta­neously pre­dict­ing that when re­searchers have a part of them­selves that feels un­cer­tain or un­easy about whether their re­search should be pub­lic, they will find that there are large benefits to in­sti­tut­ing a nondis­closed-by-de­fault policy. But the policy is there to en­able re­searchers, not to an­noy them and make them jump through hoops.
  
  (Caveat: within ML, it’s still rare for risk-based nondis­clo­sure to be treated as a real op­tion, and many so­cial in­cen­tives fa­vor pub­lish­ing-by-de­fault. I want to be very clear that within the con­text of those in­cen­tives, I ex­pect many peo­ple to jump to “this seems ob­vi­ously safe to me” when the ev­i­dence doesn’t war­rant it. I think it’s im­por­tant to fa­cil­i­tate an en­vi­ron­ment where it’s not just OK-on-pa­per but also so­cially-he­do­nic to de­cide against pub­lish­ing, and I think that these de­ci­sions of­ten war­rant se­ri­ous thought. The aim of MIRI’s dis­clo­sure policy is to re­move un­due pres­sures to make pub­li­ca­tion de­ci­sions pre­ma­turely, not to over­ride re­searchers’ con­sid­ered con­clu­sions.)

  What links here?

  • MIRI’s 2019 Fundraiser by Malo (3 Dec 2019 1:16 UTC; 60 points)
  • MIRI’s 2019 Fundraiser by malo (EA Forum; 7 Dec 2019 0:30 UTC; 19 points)

Prob­a­bil­ity in­ter­pre­ta­tions: Examples

Cor­re­spon­dence vi­su­al­iza­tions for differ­ent in­ter­pre­ta­tions of “prob­a­bil­ity”

In­ter­pre­ta­tions of “prob­a­bil­ity”

• So8res 29 Jan 2017 16:01 UTC

  LW: 1 AF: 1

  AF

  in reply to: jessicata's comment on: On mo­ti­va­tions for MIRI’s highly re­li­able agent de­sign research

  The sec­ond state­ment seems pretty plau­si­ble (when we con­sider hu­man-ac­cessible AGI de­signs, at least), but I’m not su­per con­fi­dent of it, and I’m not rest­ing my ar­gu­ment on it.

  The weaker state­ment you provide doesn’t seem like it’s ad­dress­ing my con­cern. I ex­pect there are ways to get highly ca­pa­ble rea­son­ing (suffi­cient for, e.g., gain­ing de­ci­sive strate­gic ad­van­tage) with­out un­der­stand­ing low-K “good rea­son­ing”; the con­cern is that said sys­tems are much more difficult to al­ign.

• So8res 29 Jan 2017 1:53 UTC

  LW: 5 AF: 4

  AF

  on: On mo­ti­va­tions for MIRI’s highly re­li­able agent de­sign research

  As I noted when we chat­ted about this in per­son, my in­tu­ition is less “there is some small core of good con­se­quen­tial­ist rea­son­ing (it has “low Kol­mogorov com­plex­ity” in some sense), and this small core will be quite im­por­tant for AI ca­pa­bil­ities” and more “good con­se­quen­tial­ist rea­son­ing is low-K and those who un­der­stand it will be bet­ter equipped to de­sign AGI sys­tems where the rele­vant con­se­quen­tial­ist rea­son­ing hap­pens in trans­par­ent boxes rather than black boxes.”

  In­deed, if I thought one had to un­der­stand good con­se­quen­tial­ist rea­son­ing in or­der to de­sign a highly ca­pa­ble AI sys­tem, I’d be less wor­ried by a de­cent mar­gin.

• So8res 13 Jan 2017 2:11 UTC

  0 points

  AF

  in reply to: Wei_Dai's comment on: My cur­rent take on the Paul-MIRI dis­agree­ment on al­ignabil­ity of messy AI

  Weigh­ing in late here, I’ll briefly note that my cur­rent stance on the difficulty of philo­soph­i­cal is­sues is (in col­lo­quial terms) “for the love of all that is good, please don’t at­tempt to im­ple­ment CEV with your first tran­shu­man in­tel­li­gence”. My strat­egy at this point is very much “build the min­i­mum AI sys­tem that is ca­pa­ble of sta­bi­liz­ing the over­all strate­gic situ­a­tion, and then buy a whole lot of time, and then use that time to figure out what to do with the fu­ture.” I might be more op­ti­mistic than you about how easy it will turn out to be to find a rea­son­able method for ex­trap­o­lat­ing hu­man vo­li­tion, but I sus­pect that that’s a moot point ei­ther way, be­cause re­gard­less, thou shalt not at­tempt to im­ple­ment CEV with hu­man­ity’s very first tran­shu­man in­tel­li­gence.

  Also, +1 to the over­all point of “also pur­sue other ap­proaches”.

• So8res 3 Nov 2016 22:24 UTC

  0 points

  in reply to: blogospheroid's comment on: MIRI’s 2016 Fundraiser

  Thanks :-)

• So8res 1 Nov 2016 23:30 UTC

  0 points

  in reply to: Gyrodiot's comment on: MIRI’s 2016 Fundraiser

  Thanks!

• So8res 19 Oct 2016 23:21 UTC

  1 point

  in reply to: PhilGoetz's comment on: MIRI’s 2016 Fundraiser

  Fixed, thanks.

• So8res 4 Oct 2016 20:41 UTC

  2 points

  in reply to: Plasmon's comment on: MIRI’s 2016 Fundraiser

  Huh, thanks for the heads up. If you use an ad-blocker, try paus­ing that and re­fresh­ing. Mean­while, I’ll have some­one look into it.

• So8res 26 Sep 2016 18:39 UTC

  3 points

  in reply to: Furcas's comment on: MIRI’s 2016 Fundraiser

  Thanks!

MIRI’s 2016 Fundraiser

• So8res 2 May 2016 19:25 UTC

  4 points

  on: Dou­ble Cor­rigi­bil­ity: bet­ter Corrigibility

  FYI, this is not what the word “cor­rigi­bil­ity” means in this con­text. (Or, at least, it’s not how we at MIRI have been us­ing it, and it’s not how Stu­art Rus­sell has been us­ing it, and it’s not a us­age that I, as one of the peo­ple who origi­nally brought that word into the AI al­ign­ment space, en­dorse.) We use the phrase “util­ity in­differ­ence” to re­fer to what you’re call­ing “cor­rigi­bil­ity”, and we use the word “cor­rigi­bil­ity” for the broad vague prob­lem that “util­ity in­differ­ence” was but one at­tempt to solve.

  By anal­ogy, imag­ine peo­ple grop­ing around in the dark at­tempt­ing to de­velop prob­a­bil­ity the­ory. They might call the whole topic the topic of “man­ag­ing un­cer­tainty,” and they might call spe­cific at­tempts things like “fuzzy logic” or “multi-val­ued logic” be­fore even­tu­ally set­tling on some­thing that seems to work pretty well (which hap­pened to be an at­tempt called “prob­a­bil­ity the­ory.”) We’re cur­rently re­serv­ing the “cor­rigi­bilty” word for the ana­log of “man­ag­ing un­cer­tainty”; that is, we use the “cor­rigi­bil­ity” la­bel to re­fer to the highly gen­eral prob­lem of de­vel­op­ing AI al­gorithms that cause a sys­tem to (in an in­tu­itive sense) rea­son with­out in­cen­tives to de­ceive/​ma­nipu­late, and to rea­son (vaguely) as if it’s still un­der con­struc­tion and po­ten­tially dan­ger­ous :-)

• So8res 31 Dec 2015 16:28 UTC

  5 points

  in reply to: V_V's comment on: Safety en­g­ineer­ing, tar­get se­lec­tion, and al­ign­ment theory

  By your anal­ogy, one of the main crit­i­cism of do­ing MIRI-style AGI safety re­search now is that it’s like 10th cen­tury Chi­nese philoso­phers do­ing Saturn V safety re­search based on what they knew about fire ar­rows.

  This is a fairly com­mon crit­i­cism, yeah. The point of the post is that MIRI-style AI al­ign­ment re­search is less like this and more like Chi­nese math­e­mat­i­ci­ans re­search­ing calcu­lus and grav­ity, which is still difficult, but much eas­ier than at­tempt­ing to do safety en­g­ineer­ing on the Saturn V far in ad­vance :-)

Safety en­g­ineer­ing, tar­get se­lec­tion, and al­ign­ment theory

• So8res 18 Dec 2015 17:37 UTC

  13 points

  in reply to: Lumifer's comment on: Why CFAR? The view from 2015

  Yes, pre­cisely. (Trans­parency illu­sion strikes again! I had con­sid­ered it ob­vi­ous that the de­fault out­come was “a few peo­ple are nudged slightly more to­wards be­com­ing AI al­ign­ment re­searchers some­day”, and that the out­come of “ac­tu­ally cause at least one very tal­ented per­son to be­come AI al­ign­ment re­searcher who oth­er­wise would not have, over the course of three weeks” was clearly in “re­sound­ing suc­cess” ter­ri­tory, whereas “turn half the at­ten­dees into AI al­ign­ment re­searchers” is in I’ll-eat-my-hat ter­ri­tory.)

  What links here?

  • Academian's comment on Why CFAR? The view from 2015 by PeteMichaud (20 Dec 2015 23:05 UTC; 5 points)

• So8res 18 Dec 2015 2:44 UTC

  11 points

  in reply to: Qiaochu_Yuan's comment on: Why CFAR? The view from 2015

  I don’t claim that it de­vel­oped skill and tal­ent in all par­ti­ci­pants, nor even in the me­dian par­ti­ci­pant. I do stand by my claim that it ap­pears to have had dras­tic good effects on a few peo­ple though, and that it led di­rectly to MIRI hires, at least one of which would not have hap­pened oth­er­wise :-)

  What links here?

  • 2016 AI Risk Liter­a­ture Re­view and Char­ity Comparison by Larks (EA Forum; 13 Dec 2016 4:36 UTC; 53 points)

• So8res 15 Dec 2015 16:26 UTC

  2 points

  in reply to: Dr_Manhattan's comment on: MIRI’s 2015 Win­ter Fundraiser!

  Thanks! :-p It’s con­ve­nient to have the 2015 fundraisers end be­fore 2015 ends, but we may well change the way fundraisers work next year.

• So8res 15 Dec 2015 4:45 UTC

  2 points

  in reply to: BrandonReinhart's comment on: MIRI’s 2015 Win­ter Fundraiser!

  Thanks! Our lan­guages and frame­works definitely have been im­prov­ing greatly over the last year or so, and I’m ex­cited to see where we go now that we’ve pul­led a siz­able team to­gether.