Larks

• Larks 17 May 2019 3:04 UTC

  5 points

  on: Fi­nan­cial en­g­ineer­ing for fund­ing drug research

  Is this very differ­ent from found­ing a phar­ma­ceu­ti­cal com­pany?

• Larks 2 May 2019 20:51 UTC

  10 points

  in reply to: michaelcohen's comment on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  Critch wrote a re­lated pa­per:

  Ex­ist­ing multi-ob­jec­tive re­in­force­ment learn­ing (MORL) al­gorithms do not ac­count for ob­jec­tives that arise from play­ers with differ­ing be­liefs.Con­cretely, con­sider two play­ers with differ­ent be­liefs and util­ity func­tions who may co­op­er­ate to build a ma­chine that takes ac­tions on their be­half. A rep­re­sen­ta­tion is needed for how much the ma­chine’s policy will pri­ori­tize each player’s in­ter­ests over time. As­sum­ing the play­ers have reached com­mon knowl­edge of their situ­a­tion, this pa­per de­rives a re­cur­sion that any Pareto op­ti­mal policy must satisfy. Two qual­i­ta­tive ob­ser­va­tions can be made from the re­cur­sion: the ma­chine must (1) use each player’s own be­liefs in eval­u­at­ing how well an ac­tion will serve that player’s util­ity func­tion, and (2) shift the rel­a­tive pri­or­ity it as­signs to each player’s ex­pected util­ities over time, by a fac­tor pro­por­tional to how well that player’s be­liefs pre­dict the ma­chine’s in­puts. Ob­ser­va­tion (2) rep­re­sents a sub­stan­tial di­ver­gence from naive lin­ear util­ity ag­gre­ga­tion (as in Harsanyi’s util­i­tar­ian the­o­rem, and ex­ist­ing MORL al­gorithms), which is shown here to be in­ad­e­quate for Pareto op­ti­mal se­quen­tial de­ci­sion-mak­ing on be­half of play­ers with differ­ent be­liefs.

  Toward ne­go­tiable re­in­force­ment learn­ing: shift­ing pri­ori­ties in Pareto op­ti­mal se­quen­tial de­ci­sion-making

• Larks 2 May 2019 20:44 UTC

  2 points

  in reply to: cousin_it's comment on: Strate­gic im­pli­ca­tions of AIs’ abil­ity to co­or­di­nate at low cost, for ex­am­ple by merging

  War only hap­pens if two agents don’t have com­mon knowl­edge about who would win (oth­er­wise they’d agree to skip the costs of war).

  They might also have poorly al­igned in­cen­tives, like a war be­tween two coun­tries that al­lows both gov­ern­ments to gain power and pres­tige, at the cost of de­struc­tion that is borne by the or­di­nary peo­ple of both coun­tries. But this sort of prin­ci­ple-agent prob­lem also seems like some­thing AIs should be bet­ter at deal­ing with.

• Larks 16 Apr 2019 16:06 UTC

  12 points

  on: Liter­a­ture Re­view: Distributed Teams

  In light of this:

  Build over-com­mu­ni­ca­tion into the pro­cess.

  In par­tic­u­lar, don’t let silence carry in­for­ma­tion. Silence can be in­ter­preted a mil­lion differ­ent ways (Cram­ton 2001).

  Thanks for writ­ing this! I found it very in­ter­est­ing, and I like the style. I par­tic­u­larly hadn’t prop­erly ap­pre­ci­ated how semi-dis­tributed was worth than ei­ther ex­treme. It’s dis­ap­point­ing to hear, but seem­ingly ob­vi­ous in ret­ro­spect and good to know.

• Larks 5 Jan 2019 18:45 UTC

  2 points

  in reply to: Dr_Manhattan's comment on: 2018 AI Align­ment Liter­a­ture Re­view and Char­ity Comparison

  Thanks for shar­ing, seems like a rea­son­able take to me.

• Larks 5 Jan 2019 18:43 UTC

  2 points

  in reply to: AdamGleave's comment on: 2018 AI Align­ment Liter­a­ture Re­view and Char­ity Comparison

  I definitely be­ing near AI hubs is helpful, and I’d be in­ter­ested in sup­port­ing any cred­ible new groups that started in other hubs.

  Thanks for that ex­tra info on CHAI staff. In gen­eral my ob­jec­tions to the bay area are partly about the EA/​LW cul­ture there, and partly about the broader cul­ture. I did end up donat­ing to CHAI de­spite this!

• Larks 5 Jan 2019 18:22 UTC

  4 points

  in reply to: Benito's comment on: 2018 AI Align­ment Liter­a­ture Re­view and Char­ity Comparison

  Thanks! I have fixed most of the ty­pos.

• Larks 19 Dec 2018 0:37 UTC

  3 points

  in reply to: Wei_Dai's comment on: In Defense of Finance

  Espe­cially as in re­cent years in­vestor prefer­ences for lower risk, low beta, low vol, higher yield stocks (like util­ities and sta­ples) has been well doc­u­mented as strong.

  One re­lated fact is that (at least some of the banks) feel like they haven’t got­ten credit for the risk re­duc­tions they have done. Given their lower lev­er­age now, they feel like they should have higher mul­ti­ples, whereas ac­tu­ally their rel­a­tive mul­ti­ples have com­pressed vs the mar­ket vs pre-crisis. Of course, this may be be­cause their risk was un­der-es­ti­mated pre-crisis, so their rel­a­tive mul­ti­ple was too high.

• Larks 19 Dec 2018 0:33 UTC

  18 points

  in reply to: Benquo's comment on: In Defense of Finance

  At least in pub­lic equity mar­kets the size pre­mium is very weak. My team looked at it and de­cided not to use it as a fac­tor to guide our in­vest­ing. It’s closely cor­re­lated to an illiquidity pre­mia, which I do be­lieve in.

2018 AI Align­ment Liter­a­ture Re­view and Char­ity Comparison

• Larks 15 Dec 2018 19:04 UTC

  2 points

  on: Get geno­typed for free ( If your IQ is high enough)

  Up­date: Jeff says they only ever sent him part of his data.

• Larks 17 Nov 2018 14:48 UTC

  2 points

  in reply to: Benito's comment on: The Vuln­er­a­ble World Hy­poth­e­sis (by Bostrom)

  Nick’s space pa­pers are largely about how to har­vest large amounts of util­ity from the galaxy, not about how to in­crease hu­man­ity’s ro­bust­ness. And yes, there are some Xrisks (in­clud­ing the one I am fo­cused on) that space colonies do not help with, but the reader may not be con­vinced of these, so it is surely worth men­tion­ing that some risks would be guarded against with in­ter­stel­lar di­ver­sifi­ca­tion. If noth­ing else you should prob­a­bly ar­gue that space coloniza­tion is not an ad­e­quate solu­tion for these rea­sons.

• Larks 17 Nov 2018 3:02 UTC

  27 points

  in reply to: jessicata's comment on: Some cruxes on im­pact­ful al­ter­na­tives to AI policy work

  A study by the in­vest­ment-re­search firm Strate­gas which was cited in The Economist and the Wash­ing­ton Post com­pared the 50 firms that spent the most on lob­by­ing rel­a­tive to their as­sets, and com­pared their fi­nan­cial perfor­mance against that of the S&P 500 in the stock mar­ket; the study con­cluded that spend­ing on lob­by­ing was a “spec­tac­u­lar in­vest­ment” yield­ing “blis­ter­ing” re­turns com­pa­rable to a high-fly­ing hedge fund, even de­spite the fi­nan­cial down­turn of the past few years.

  I think I read this re­search while I was a Strate­gas client; if I’m re­mem­ber­ing it cor­rectly it was ex­tremely poorly done. Short back test (just a few years), gar­den of fork­ing paths, etc. Most sell-side re­search is not epistem­i­cally rigourous and Strate­gas is not one of the bet­ter firms. I would not put much weight on this re­search.

  There is wide­spread agree­ment that a key in­gre­di­ent in effec­tive lob­by­ing is money. This view is shared by play­ers in the lob­by­ing in­dus­try.

  Well of course lob­by­ists would say they’re worth the money!

• Larks 22 Mar 2018 23:33 UTC

  8 points

  on: Values de­ter­mined by “stop­ping” properties

  This, by the way, ex­plains my in­tu­itive dis­like for some types of moral re­al­ism. If there are true ob­jec­tive moral facts that hu­mans can ac­cess, then what­ever pro­cess counts as “ac­cess­ing them” be­comes… a lo­cal stop­ping con­di­tion for defin­ing value.

  I’m not sure I un­der­stand what you’re get­ting at here. Yes, they are both lo­cal stop­ping con­di­tions, but there seems to be a clear dis-anal­ogy. The other lo­cal stop­ping con­di­tions seem to be bad not be­cause they are stop­ping con­di­tions, but be­cause most con­tem­po­rary peo­ple don’t want to end up as Lo­tus-Eaters, or as mind­less out­sourcers. We would op­pose such a de­vel­op­ment even if it wasn’t sta­ble! For ex­am­ple, a fu­ture where we os­cillate be­tween lo­tus-eaters and mind­less out­sourcers seems about as bad as ei­ther in­di­vi­d­ual sce­nario. So it’s not re­ally the sta­bil­ity we ob­ject to.

  But in that case, it’s not clear why we should be op­posed to the moral re­al­ism. After all, many peo­ple would like to go there, even if we are presently a long way away.

• Larks 6 Mar 2018 4:01 UTC

  0 points

  in reply to: gwern's comment on: Open Thread, Fe­bru­ary 1-14, 2012

  the one aca­demic do­ing good work in the area is Sheffer, who is run­ning a lon­gi­tu­di­nal sur­vey which may or may not have enough statis­ti­cal power to rule out par­tic­u­larly dra­matic var­i­ances in out­comes. (Sheffer men­tions the se­lec­tion bias prob­lem but seems to have the at­ti­tude that it’s not a prob­lem for her work.)

  Was there any fol­low-up here?

• Larks 8 Jan 2018 2:13 UTC

  0 points

  on: Ex­pe­riences in ap­ply­ing “The Biode­ter­minist’s Guide to Par­ent­ing”

  The bioter­minist’s guide is now 5 years old. Does any­one know of an up­dated ver­sion?

• Larks 30 Dec 2017 20:21 UTC

  8 points

  in reply to: Raemon's comment on: 2018 AI Safety Liter­a­ture Re­view and Char­ity Com­par­i­son

  Thanks, I’m hon­oured! I’ve sent you a pri­vate mes­sage.

2017 AI Safety Liter­a­ture Re­view and Char­ity Com­par­i­son

2018 AI Safety Liter­a­ture Re­view and Char­ity Comparison

• Larks 9 Dec 2017 16:43 UTC

  5 points

  on: More Dakka

  In our house we started a tra­di­tion of hold­ing hands and tak­ing turns say­ing some­thing we’re grate­ful be­fore din­ner each night. We then soft-evan­gal­ise this by hav­ing guests over and in­clud­ing them—most no­tably to hun­dreds of peo­ple at our wed­ding.