Unnamed

• Unnamed 18 Jun 2019 4:56 UTC

  20 points

  on: Why the tails come apart

  One an­gle for think­ing about why the tails come apart (which seems worth high­light­ing even more than it was high­lighted in the OP) is that the farther out you go in the tail on some vari­able, the smaller the set of peo­ple you’re deal­ing with.

  Which is bet­ter, the best bas­ket­ball team that you can put to­gether from peo­ple born in Penn­syl­va­nia or the best bas­ket­ball team that you can put to­gether from peo­ple born in Delaware? Prob­a­bly the Penn­syl­va­nia team, since there are about 13x as many peo­ple in that state so you get to draw from a larger pool. If there were no other rele­vant differ­ences be­tween the states then you’d ex­pect 13 of the best 14 play­ers to be Penn­syl­va­ni­ans, and prob­a­bly the two neigh­bor­ing states are similar enough so that Delaware can’t over­come that pop­u­la­tion gap.

  Now, imag­ine you’re pick­ing the best 10 bas­ket­ball play­ers from the 1,000 tallest bas­ket­ball-aged Amer­i­cans (20-34 year-olds), and you’re putting to­gether an­other group con­sist­ing of the best 10 bas­ket­ball play­ers from the next 100,000 tallest bas­ket­ball-aged Amer­i­cans. Which is a bet­ter group of bas­ket­ball play­ers? In this case it’s not ob­vi­ous—get­ting to pick from a pool of 100x as many peo­ple is an ob­vi­ous ad­van­tage, but that height ad­van­tage could mat­ter a lot too. That’s the tails com­ing apart—the very tallest don’t nec­es­sar­ily give you the very best bas­ket­ball play­ers, be­cause “the very tallest” is a much smaller set than the “also re­ally tall but not quite as tall”.

  (I ran some num­bers and es­ti­mate that the two teams are pretty similar in bas­ket­ball abil­ity. Which is a re­mark­able sign of how im­por­tant height is for bas­ket­ball—one pool has about a 4 inch height ad­van­tage on av­er­age, the other pool has 100x as many peo­ple, and those fac­tors roughly bal­ance out. If you want the ex­am­ple to more defini­tively show the tails com­ing apart, you can ex­pand the larger pool by an­other fac­tor of 30x and then they’ll clearly be bet­ter.)

  Similarly, who has higher arm strength: the one per­son in our sam­ple who has the high­est grip strength, or the most arm-strong per­son out of the next ten peo­ple who rank 2-11 in grip strength? Grip strength is closely re­lated to arm strength, but you get to pick the best from a 10x larger pool if you give up a lit­tle bit of grip strength. In the graph in the OP, the per­son who was 6th (or maybe 5th) in grip strength had the high­est arm strength, so get­ting to pick from a pool of 10 was more im­por­tant. (The av­er­age arm strength of the peo­ple ranked 2-11 in grip strength was lower than the arm strength of the #1 grip­per, but we get to pick out the strongest arm of the ten rather than av­er­ag­ing them.)

  So: the tails come apart be­cause most of the peo­ple aren’t way out on the tail. And you usu­ally won’t find the very best per­son at some­thing if you’re look­ing in a tiny pool, even if that’s a pretty well se­lected pool.

  Thrasy­machus’s in­tu­itive ex­pla­na­tion cov­ered this—hav­ing a smaller pool to pick from hurts be­cause there are other vari­ables that mat­ter, and the smaller the pool the less you get to se­lect for peo­ple who do well on those other vari­ables. But his ex­pla­na­tion high­lighted the “other vari­ables mat­ter” part of this more than the pool size part of it, and both of these points of em­pha­sis seem helpful for get­ting an in­tu­itive grasp of the statis­tics in these types of situ­a­tions, so I figured I’d add this com­ment.

• Unnamed 9 Jun 2019 0:43 UTC

  8 points

  on: The Schel­ling Choice is “Rab­bit”, not “Stag”

  And I said, in a move de­signed to be some­what so­cially pun­ish­ing: “I don’t re­ally trust the con­ver­sa­tion to go any­where use­ful.” And then I took out my lap­top and mostly stopped pay­ing at­ten­tion.

  This ‘so­cial pun­ish­ment’ move seems prob­le­matic, in a way that isn’t high­lighted in the rest of the post.

  One is­sue: What are you pun­ish­ing them for? It seems like the pun­ish­ment is in­tended to en­force the norm that you wanted the group to have, which is a differ­ent kind of move than en­forc­ing a norm that is already es­tab­lished. En­forc­ing ex­ist­ing norms is gen­er­ally proso­cial, but it’s more prob­le­matic if each per­son is try­ing to en­force the norms that he per­son­ally wishes the group to have.

  A sec­ond thing worth high­light­ing is that this at­tempt at norm en­force­ment looked a lot like a norm vi­o­la­tion (of norms against dis­en­gag­ing from a meet­ing). Some­times “pun­ish­ing oth­ers for vi­o­lat­ing norms” is a spe­cial case where it’s ap­pro­pri­ate to do some­thing which would oth­er­wise be a norm vi­o­la­tion, but that’s of­ten a costly/​risky way of do­ing things (es­pe­cially when the norm you’re en­forc­ing isn’t clearly es­tab­lished and so your ac­tions are less leg­ible).

• Unnamed 7 Jun 2019 21:42 UTC

  29 points

  on: Asym­met­ric Weapons Aren’t Always on Your Side

  When Scott used the term “asym­met­ric weapons”, I un­der­stood him to mean truth-asym­met­ric weapons or weapons that fa­vor what’s good & true. He was try­ing to set that par­tic­u­lar di­men­sion of asym­me­try apart from the var­i­ous other ways in which a weapon might be more use­ful in some hands than in oth­ers.

  I think it’s an im­por­tant con­cept, and I wish we had bet­ter ter­minol­ogy for it.

• Unnamed 28 May 2019 0:41 UTC

  9 points

  on: Was CFAR always in­tended to be a dis­tinct or­ga­ni­za­tion from MIRI?

  Yes. Or at least, be­com­ing a dis­tinct or­ga­ni­za­tion was already the plan when I got there in early 2012: get a group of peo­ple to­gether to cre­ate a ra­tio­nal­ity or­ga­ni­za­tion, ini­tially rely on MIRI for in­sti­tu­tional sup­port, be­come an in­de­pen­dent or­ga­ni­za­tion some months later once all the pieces are in place to do so.

• Unnamed 2 May 2019 5:45 UTC

  9 points

  on: Nash equil­ibriums can be ar­bi­trar­ily bad

  Unilat­eral pre­com­mit­ment lets peo­ple win at “Al­most Free Lunches”.

  One way to model pre­com­mit­ment is as a se­quen­tial game: first player 1 chooses a num­ber, then player 1 has the op­tion of ei­ther show­ing that num­ber to player 2 or keep­ing it hid­den, then player 2 chooses a num­ber. Op­ti­mal play is for player 1 to pick £1,000,000 and show that num­ber, and then for player 2 to choose £999,999.99.

  An in­ter­est­ing fea­ture of this is that player 1′s pre­com­mit­ment helped player 2 even more than it helped player 1. Player 1 is “tak­ing one for the team”, but still win­ning big. This dis­t­in­guishes it from games like chicken, where pre­com­mit­ment is a threat that al­lows the pre­com­mit­ter to win the larger share. Though this means that if ei­ther player can pre­com­mit (rather than one be­ing pre-as­signed to go first as player 1) then they’d both pre­fer to have the other one be the pre­com­mit­ter.

  This benefit of pre­com­mit­ment does not ex­tend to the two op­tion ver­sion (n2 vs. n1). In that ver­sion, player 2 is in­cen­tivized to say “n1” re­gard­less of what player 1 com­mits to, so unilat­eral pre­com­mit­ment doesn’t help them avoid the Nash Equil­ibrium. As in the pris­oner’s dilemma.

• Unnamed 30 Mar 2019 0:39 UTC

  19 points

  on: How much fund­ing and re­searchers were in AI, and AI Safety, in 2018?

  Some num­bers re­lated to c (how many ca­pa­bil­ities re­searchers):

  In 2018 about 8,500 peo­ple at­tended NeurIPS and about 4,000 peo­ple at­tended ICML. There are about 2,000 re­searchers who work at Google AI, and in De­cem­ber 2017 there were re­ports that about 700 to­tal peo­ple work at Deep­Mind in­clud­ing about 400 with a PhD.

  Turn­ing this into a sin­gle es­ti­mate for “num­ber of re­searchers” is tricky for the sorts of rea­sons that cathe­rio gives. Ca­pa­bil­ities re­searchers is a fuzzy cat­e­gory and it’s not clear to what ex­tent peo­ple who are work­ing on ad­vanc­ing the state of the art in gen­eral AI ca­pa­bil­ities should in­clude peo­ple who are pri­mar­ily work­ing on ap­pli­ca­tions us­ing the cur­rent art and peo­ple who are pri­mar­ily work­ing on ad­vanc­ing the state of the art in nar­rower sub­fields. Also ob­vi­ously only some frac­tion of the rele­vant re­searchers at­tended those con­fer­ences or work at those com­pa­nies.

  I’ll sug­gest 10,000 peo­ple as a rough or­der-of-mag­ni­tude es­ti­mate. I’d be sur­prised if the num­ber that came out of a more care­ful es­ti­ma­tion pro­cess wasn’t within a fac­tor of ten of that.

• Unnamed 20 Feb 2019 6:30 UTC

  11 points

  in reply to: Unnamed's comment on: Blackmail

  After dis­cussing this offline, I think the main ar­gu­ment that I laid out does not hold up well in the case of black­mail (though it works bet­ter for many other kinds of threats). They key bit is here:

  if Bob re­fuses and Alice car­ries out her threat then it is nega­tive sum (Bob loses a lot and Alice loses some­thing too)

  This only looks at the effects on Alice and on Bob, as a sim­plifi­ca­tion. But with black­mail “car­ry­ing out the threat” means tel­ling other peo­ple in­for­ma­tion about Bob, and that is of­ten use­ful for those other peo­ple. If Alice tells Casey some­thing bad about Bob, that will of­ten be bad for Bob but good for Casey. So it’s not ob­vi­ously nega­tive sum for the whole world.

• Unnamed 19 Feb 2019 23:06 UTC

  23 points

  on: Blackmail

  There’s a pretty sim­ple eco­nomic ar­gu­ment for why black­mail is bad: it in­volves a nega­tive-sum threat rather than a pos­i­tive-sum deal. I was sur­prised to not see this ar­gu­ment in the econ­blog­gers’ dis­cus­sion; good to see it come up here. To lay it out suc­cinctly and sep­a­rate from other ar­gu­ments:

  Or­di­nar­ily, when two peo­ple make a deal we can con­clude that it’s win-win be­cause both of them chose to make the deal rather than just not in­ter­act­ing with each other. By de­fault Alice would just act on her own prefer­ences and com­pletely ig­nore Bob’s prefer­ences, and the mir­ror image for Bob, but some­times they find a deal where they each give up some­thing in re­turn for the other per­son do­ing some­thing that they value even more. With some sim­plify­ing as­sump­tions, the worst case sce­nario is that they don’t reach a deal and they both break even (com­pared to if they hadn’t in­ter­acted), and if they do reach a deal then they both wind up bet­ter off.

  With a threat, Alice has an al­ter­na­tive course of ac­tion available which is some­what worse for Alice than her de­fault ac­tion but much worse for Bob, and Alice tells Bob that she will do the al­ter­na­tive ac­tion un­less Bob does some­thing for Alice. With some sim­plify­ing as­sump­tions, if Bob agrees to give in then their in­ter­ac­tion is zero-sum (Alice gets a trans­fer from Bob), if Bob re­fuses and Alice car­ries out her threat then it is nega­tive sum (Bob loses a lot and Alice loses some­thing too), and if Bob re­fuses and Alice backs down then it’s zero sum (both take the de­fault ac­tion).

  Or­di­nary deals add value to the world and threats sub­tract value from the world, and black­mail is a type of threat.

  If we re­move some sim­plify­ing as­sump­tions (e.g. no trans­ac­tion costs, one-shot in­ter­ac­tion) then things get more com­pli­cated, but mostly in ways that make or­di­nary deals bet­ter and threats worse. In the long run deals bring peo­ple to­gether as they seek more in­ter­ac­tions which could lead to win-win deals, deals en­courage peo­ple to in­vest in abil­ities that will make them more use­ful to other peo­ple so that they’ll have more/​bet­ter op­por­tu­ni­ties to make deals, and the benefits of deals must out­weigh the trans­ac­tion costs & risks in­volved at least in ex­pec­ta­tion (oth­er­wise peo­ple would just opt out of try­ing to make those deals). Whereas threats push peo­ple apart as they seek to avoid nega­tive sum in­ter­ac­tions, threats en­courage peo­ple to in­vest in abil­ities that make them more able to harm other peo­ple, and trans­ac­tion costs in­crease the bad­ness of threats (turn­ing zero sum in­ter­ac­tions into nega­tive sum) but don’t pre­vent those in­ter­ac­tions un­less they drive the threat­maker’s re­turns down far enough.

• Unnamed 19 Feb 2019 21:57 UTC

  32 points

  on: Epistemic Tenure

  I think that there’s a spec­trum be­tween treat­ing some­one as a good source of con­clu­sions and treat­ing them as a good source of hy­pothe­ses.

  I can have thoughts like “Carol looked closely into the topic and came away con­vinced that Y is true, so for now I’m go­ing to act as if Y is prob­a­bly true” if I take Carol to be a good source of con­clu­sions.

  Whereas if I took Alice to be a good source of hy­pothe­ses but not a good source of con­clu­sions, then I would in­stead have thoughts like “Alice in­sists that Z is true, so Z seems like some­thing that’s worth think­ing about more.”

  Giv­ing some­one epistemic tenure as a source of con­clu­sions seems much more costly than giv­ing them epistemic tenure as a source of hy­pothe­ses.

• Unnamed 18 Feb 2019 5:09 UTC

  4 points

  in reply to: Benito's comment on: Avoid­ing Jar­gon Confusion

  Huh? I am suffi­ciently sur­prised/​con­fused by this ex­am­ple to want a cita­tion.

  Edit: The sur­prise/​con­fu­sion was in refer­ence to the pre-edit ver­sion of the above com­ment, and does not ap­ply to the cur­rent edi­tion.

• Unnamed 16 Feb 2019 22:35 UTC

  3 points

  on: Heuris­tics for de­cen­tral­ised co­or­di­na­tion?

  I think we should take more care to sep­a­rate the ques­tion of of whether AI de­vel­op­ments will be de­cen­tral­ized with the ques­tion of whether de­cen­tral­iza­tion is safer. It is not ob­vi­ous to me whether a de­cen­tral­ized, econ­omy-wide path to ad­vanced AIs will be safer or riskier than a con­cen­trated path within a sin­gle or­ga­ni­za­tion. It seems like the open­ing sen­tence of this ques­tion is car­ry­ing the as­sump­tion that de­cen­tral­ized is safer (“Robin Han­son has ar­gued that those who be­lieve AI Risk to be a pri­mary con­cern for hu­man­ity, are suffer­ing from a bias to­ward think­ing that con­cen­tra­tion of power is always more effi­cient than a de­cen­tral­ised sys­tem”).

• Unnamed 6 Feb 2019 9:02 UTC

  4 points

  in reply to: avturchin's comment on: Great­est Lower Bound for AGI

  I think you mean ⁵⁰⁄₆₂ = 0.81?

• Unnamed 13 Oct 2018 6:16 UTC

  2 points

  on: The Valley of Bad Theory

  Some­times the­ory can open up pos­si­bil­ities rather than clos­ing them off. In these cases, once you have a the­ory that claims that X is im­por­tant, then you can ex­plore differ­ent val­ues of X and do lo­cal hill-climb­ing. But be­fore that it is difficult to ex­plore by vary­ing X, ei­ther be­cause there are too many di­men­sions or be­cause there is some sub­tlety in rec­og­niz­ing that X is a di­men­sion and be­ing able to vary its level.

  This de­pends on be­ing able to have and use a the­ory with­out be­liev­ing it.

• Unnamed 3 Oct 2018 5:19 UTC

  6 points

  on: Does This Fal­lacy Have A Name?

  This sounds most similar to what LWers call gen­er­al­iz­ing from one ex­am­ple or the typ­i­cal mind fal­lacy and to what psy­chol­o­gists call the false-con­sen­sus effect or ego­cen­tric bias.

• Unnamed 14 Sep 2018 20:24 UTC

  15 points

  in reply to: ioannes_shade's comment on: No stan­dard met­ric for CFAR work­shops?

  Here are rel­a­tively brief re­sponses on these 3 par­tic­u­lar points; I’ve made a sep­a­rate com­ment which lays out my think­ing on met­rics like the Big 5 which pro­vides some con­text for these re­sponses.

  We have con­tinued to col­lect mea­sures like the ones in the 2015 lon­gi­tu­di­nal study. We are mainly an­a­lyz­ing them in large batches, rather than work­shop to work­shop, be­cause the sam­ple size isn’t big enough to dis­t­in­guish sig­nal from noise for sin­gle work­shops. One of the pro­jects that I’m cur­rently work­ing on is an anal­y­sis of a cou­ple years of these data.

  The 2017 im­pact re­port was not in­tended as a com­pre­hen­sive ac­count of all of CFAR’s met­rics, it was just fo­cused on CFAR’s EA im­pact. So it looked at the data that were most di­rectly re­lated to CFAR alums’ im­pact on the world, and “on av­er­age alums have some in­crease in con­scien­tious­ness” seemed less rele­vant than the in­for­ma­tion that we did in­clude. The first few para­graphs of the re­port say more about this.

  I’m cu­ri­ous why you’re es­pe­cially in­ter­ested in Raven’s Pro­gres­sive Ma­tri­ces. I haven’t looked closely at the liter­a­ture on it, but my im­pres­sion is that it’s one of many met­rics which are loosely re­lated to the thing that we mean by “ra­tio­nal­ity.” It has the method­olog­i­cal ad­van­tage of be­ing a perfor­mance score rather than self-re­port (though this is par­tially offset by the pos­si­bil­ity of prac­tice effects and effort effects). The big dis­ad­van­tage is the one that Kaj pointed to: it seems to track rel­a­tively sta­ble as­pects of a per­son’s think­ing skills, and might not change much even if a per­son made large im­prove­ments. For in­stance, I could imag­ine a per­son de­vel­op­ing MacGyver-level prob­lem-solv­ing abil­ity while hav­ing lit­tle or no change in their Raven’s score.

• Unnamed 14 Sep 2018 20:20 UTC

  22 points

  on: No stan­dard met­ric for CFAR work­shops?

  Here’s a sketch of my think­ing about the use­ful­ness of met­rics like the Big 5 for what CFAR is try­ing to do.

  It would be con­ve­nient if there was a defini­tive mea­sure of a per­son’s ra­tio­nal­ity which closely matched what we mean by the term and was highly sen­si­tive to changes. But as far as I can tell there isn’t one, and there isn’t likely to be one any­time soon. So we rely on a mix of in­di­ca­tors, in­clud­ing some that are more like sys­tem­atic met­rics, some that are more like in­di­vi­d­u­als’ sub­jec­tive im­pres­sions, and some that are in be­tween.

  I think of the es­tab­lished psy­chol­ogy met­rics (Big 5, life satis­fac­tion, gen­eral self-effi­cacy, etc.) as pri­mar­ily pro­vid­ing a san­ity check on whether the work­shop is do­ing some­thing, along with a very very rough pic­ture of some of what it is do­ing. They are quan­ti­ta­tive mea­sures that don’t rely on staff mem­bers’ sub­jec­tive im­pres­sions of par­ti­ci­pants, they have been val­i­dated (at least to some ex­tent) in ex­ist­ing psy­chol­ogy re­search, and they seem at least loosely re­lated to the effects that CFAR hopes to have. And, com­pared to other ways of eval­u­at­ing CFAR’s im­pact on in­di­vi­d­u­als, they’re rel­a­tively easy for an out­sider to make sense of.

  A ma­jor limi­ta­tion of these es­tab­lished psy­chol­ogy met­rics is that they haven’t been that helpful as feed­back loops. One of the main pur­poses of a met­ric is to provide in­put into CFAR’s day-to-day and work­shop-to-work­shop efforts to de­velop bet­ter tech­niques and re­fine the work­shop. That is hard to do with met­rics like the ones in the lon­gi­tu­di­nal study, be­cause of a com­bi­na­tion of a few fac­tors:

  1. The re­sults aren’t available un­til sev­eral months af­ter the work­shop, which would make for very slow feed­back loops and iter­a­tion.

  2. The re­sults are too noisy to tell if changes from one work­shop to the next are just ran­dom vari­a­tion. It takes sev­eral work­shops worth of data to get a clear sig­nal on most of the met­rics.

  3. Th­ese met­rics are only loosely re­lated to what we care about. If a change to the work­shop leads to larger in­creases in con­scien­tious­ness that does not nec­es­sar­ily mean that we want to do it, and when a cur­ricu­lum de­vel­oper is work­ing on a class they are gen­er­ally not that in­ter­ested in these par­tic­u­lar met­rics.

  4. Th­ese met­rics are rel­a­tively gen­eral/​coarse in­di­ca­tors of the effect of the work­shop as a whole, not tied to par­tic­u­lar in­puts. So (for ex­am­ple) if we make some changes to the TAPs class and want to see if the new ver­sion of the class works bet­ter or worse, there isn’t a met­ric that iso­lates the effects of the TAPs class from the rest of the work­shop.

• Unnamed 6 Sep 2018 18:59 UTC

  50 points

  on: No stan­dard met­ric for CFAR work­shops?

  (This is Dan from CFAR)

  CFAR’s 2015 Lon­gi­tu­di­nal Study mea­sured the Big 5 and some other stan­dard psy­chol­ogy met­rics. It did find changes in­clud­ing de­creased neu­roti­cism and in­creased con­scien­tious­ness.

• Unnamed 5 Sep 2018 20:28 UTC

  16 points

  in reply to: Bucky's comment on: Birth or­der effect found in No­bel Lau­re­ates in Physics

  Seems in­ter­est­ing to get data on:

  Some group that isn’t heav­ily se­lected for in­tel­li­gence /​ in­tel­lec­tual cu­ri­os­ity: skate­board­ers, protestors, pro­fes­sional hockey play­ers...

  Some non-STEM group that is se­lected for suc­cess based on men­tal skills: liter­a­ture lau­re­ates, gov­er­nors, …

  Not sure which groups it would be easy to get data on.

  There is also the op­tion of look­ing into ex­ist­ing re­search on birth or­der to see what groups other peo­ple have already looked at.

• Unnamed 4 Sep 2018 23:45 UTC

  28 points

  on: nos­talge­braist—bayes: a kinda-sorta masterpost

  Seems worth not­ing that nos­talge­braist pub­lished this post in June 2017, which was (for ex­am­ple) be­fore Eliezer’s post on toolbox think­ing.

• Unnamed 4 Sep 2018 22:41 UTC

  27 points

  on: Birth or­der effect found in No­bel Lau­re­ates in Physics

  Now that we have data on LWers/​SSCers, math­e­mat­i­ci­ans, and physi­cists, if any­one wants to put more work into this I’d like to see them look some­place differ­ent. We don’t want to fall into the Wa­son 2-4-6 trap of only look­ing for birth or­der effects among smart STEM folks. We want data that can dis­t­in­guish Scott’s in­tel­li­gence /​ in­tel­lec­tual cu­ri­os­ity hy­poth­e­sis from other pos­si­bil­ities like some non-big-5 per­son­al­ity differ­ence or a gen­eral first­borns more likely phe­nomenon.