Unnamed

• Unnamed 30 Mar 2019 0:39 UTC

  19 points

  on: How much fund­ing and re­searchers were in AI, and AI Safety, in 2018?

  Some num­bers re­lated to c (how many ca­pa­bil­ities re­searchers):

  In 2018 about 8,500 peo­ple at­tended NeurIPS and about 4,000 peo­ple at­tended ICML. There are about 2,000 re­searchers who work at Google AI, and in De­cem­ber 2017 there were re­ports that about 700 to­tal peo­ple work at Deep­Mind in­clud­ing about 400 with a PhD.

  Turn­ing this into a sin­gle es­ti­mate for “num­ber of re­searchers” is tricky for the sorts of rea­sons that cathe­rio gives. Ca­pa­bil­ities re­searchers is a fuzzy cat­e­gory and it’s not clear to what ex­tent peo­ple who are work­ing on ad­vanc­ing the state of the art in gen­eral AI ca­pa­bil­ities should in­clude peo­ple who are pri­mar­ily work­ing on ap­pli­ca­tions us­ing the cur­rent art and peo­ple who are pri­mar­ily work­ing on ad­vanc­ing the state of the art in nar­rower sub­fields. Also ob­vi­ously only some frac­tion of the rele­vant re­searchers at­tended those con­fer­ences or work at those com­pa­nies.

  I’ll sug­gest 10,000 peo­ple as a rough or­der-of-mag­ni­tude es­ti­mate. I’d be sur­prised if the num­ber that came out of a more care­ful es­ti­ma­tion pro­cess wasn’t within a fac­tor of ten of that.

• Unnamed 20 Feb 2019 6:30 UTC

  11 points

  in reply to: Unnamed's comment on: Blackmail

  After dis­cussing this offline, I think the main ar­gu­ment that I laid out does not hold up well in the case of black­mail (though it works bet­ter for many other kinds of threats). They key bit is here:

  if Bob re­fuses and Alice car­ries out her threat then it is nega­tive sum (Bob loses a lot and Alice loses some­thing too)

  This only looks at the effects on Alice and on Bob, as a sim­plifi­ca­tion. But with black­mail “car­ry­ing out the threat” means tel­ling other peo­ple in­for­ma­tion about Bob, and that is of­ten use­ful for those other peo­ple. If Alice tells Casey some­thing bad about Bob, that will of­ten be bad for Bob but good for Casey. So it’s not ob­vi­ously nega­tive sum for the whole world.

• Unnamed 19 Feb 2019 23:06 UTC

  23 points

  on: Blackmail

  There’s a pretty sim­ple eco­nomic ar­gu­ment for why black­mail is bad: it in­volves a nega­tive-sum threat rather than a pos­i­tive-sum deal. I was sur­prised to not see this ar­gu­ment in the econ­blog­gers’ dis­cus­sion; good to see it come up here. To lay it out suc­cinctly and sep­a­rate from other ar­gu­ments:

  Or­di­nar­ily, when two peo­ple make a deal we can con­clude that it’s win-win be­cause both of them chose to make the deal rather than just not in­ter­act­ing with each other. By de­fault Alice would just act on her own prefer­ences and com­pletely ig­nore Bob’s prefer­ences, and the mir­ror image for Bob, but some­times they find a deal where they each give up some­thing in re­turn for the other per­son do­ing some­thing that they value even more. With some sim­plify­ing as­sump­tions, the worst case sce­nario is that they don’t reach a deal and they both break even (com­pared to if they hadn’t in­ter­acted), and if they do reach a deal then they both wind up bet­ter off.

  With a threat, Alice has an al­ter­na­tive course of ac­tion available which is some­what worse for Alice than her de­fault ac­tion but much worse for Bob, and Alice tells Bob that she will do the al­ter­na­tive ac­tion un­less Bob does some­thing for Alice. With some sim­plify­ing as­sump­tions, if Bob agrees to give in then their in­ter­ac­tion is zero-sum (Alice gets a trans­fer from Bob), if Bob re­fuses and Alice car­ries out her threat then it is nega­tive sum (Bob loses a lot and Alice loses some­thing too), and if Bob re­fuses and Alice backs down then it’s zero sum (both take the de­fault ac­tion).

  Or­di­nary deals add value to the world and threats sub­tract value from the world, and black­mail is a type of threat.

  If we re­move some sim­plify­ing as­sump­tions (e.g. no trans­ac­tion costs, one-shot in­ter­ac­tion) then things get more com­pli­cated, but mostly in ways that make or­di­nary deals bet­ter and threats worse. In the long run deals bring peo­ple to­gether as they seek more in­ter­ac­tions which could lead to win-win deals, deals en­courage peo­ple to in­vest in abil­ities that will make them more use­ful to other peo­ple so that they’ll have more/​bet­ter op­por­tu­ni­ties to make deals, and the benefits of deals must out­weigh the trans­ac­tion costs & risks in­volved at least in ex­pec­ta­tion (oth­er­wise peo­ple would just opt out of try­ing to make those deals). Whereas threats push peo­ple apart as they seek to avoid nega­tive sum in­ter­ac­tions, threats en­courage peo­ple to in­vest in abil­ities that make them more able to harm other peo­ple, and trans­ac­tion costs in­crease the bad­ness of threats (turn­ing zero sum in­ter­ac­tions into nega­tive sum) but don’t pre­vent those in­ter­ac­tions un­less they drive the threat­maker’s re­turns down far enough.

• Unnamed 19 Feb 2019 21:57 UTC

  32 points

  on: Epistemic Tenure

  I think that there’s a spec­trum be­tween treat­ing some­one as a good source of con­clu­sions and treat­ing them as a good source of hy­pothe­ses.

  I can have thoughts like “Carol looked closely into the topic and came away con­vinced that Y is true, so for now I’m go­ing to act as if Y is prob­a­bly true” if I take Carol to be a good source of con­clu­sions.

  Whereas if I took Alice to be a good source of hy­pothe­ses but not a good source of con­clu­sions, then I would in­stead have thoughts like “Alice in­sists that Z is true, so Z seems like some­thing that’s worth think­ing about more.”

  Giv­ing some­one epistemic tenure as a source of con­clu­sions seems much more costly than giv­ing them epistemic tenure as a source of hy­pothe­ses.

• Unnamed 18 Feb 2019 5:09 UTC

  4 points

  in reply to: Benito's comment on: Avoid­ing Jar­gon Confusion

  Huh? I am suffi­ciently sur­prised/​con­fused by this ex­am­ple to want a cita­tion.

  Edit: The sur­prise/​con­fu­sion was in refer­ence to the pre-edit ver­sion of the above com­ment, and does not ap­ply to the cur­rent edi­tion.

• Unnamed 16 Feb 2019 22:35 UTC

  3 points

  on: Heuris­tics for de­cen­tral­ised co­or­di­na­tion?

  I think we should take more care to sep­a­rate the ques­tion of of whether AI de­vel­op­ments will be de­cen­tral­ized with the ques­tion of whether de­cen­tral­iza­tion is safer. It is not ob­vi­ous to me whether a de­cen­tral­ized, econ­omy-wide path to ad­vanced AIs will be safer or riskier than a con­cen­trated path within a sin­gle or­ga­ni­za­tion. It seems like the open­ing sen­tence of this ques­tion is car­ry­ing the as­sump­tion that de­cen­tral­ized is safer (“Robin Han­son has ar­gued that those who be­lieve AI Risk to be a pri­mary con­cern for hu­man­ity, are suffer­ing from a bias to­ward think­ing that con­cen­tra­tion of power is always more effi­cient than a de­cen­tral­ised sys­tem”).

• Unnamed 6 Feb 2019 9:02 UTC

  4 points

  in reply to: avturchin's comment on: Great­est Lower Bound for AGI

  I think you mean ⁵⁰⁄₆₂ = 0.81?

• Unnamed 13 Oct 2018 6:16 UTC

  2 points

  on: The Valley of Bad Theory

  Some­times the­ory can open up pos­si­bil­ities rather than clos­ing them off. In these cases, once you have a the­ory that claims that X is im­por­tant, then you can ex­plore differ­ent val­ues of X and do lo­cal hill-climb­ing. But be­fore that it is difficult to ex­plore by vary­ing X, ei­ther be­cause there are too many di­men­sions or be­cause there is some sub­tlety in rec­og­niz­ing that X is a di­men­sion and be­ing able to vary its level.

  This de­pends on be­ing able to have and use a the­ory with­out be­liev­ing it.

• Unnamed 3 Oct 2018 5:19 UTC

  6 points

  on: Does This Fal­lacy Have A Name?

  This sounds most similar to what LWers call gen­er­al­iz­ing from one ex­am­ple or the typ­i­cal mind fal­lacy and to what psy­chol­o­gists call the false-con­sen­sus effect or ego­cen­tric bias.

• Unnamed 14 Sep 2018 20:24 UTC

  15 points

  in reply to: ioannes_shade's comment on: No stan­dard met­ric for CFAR work­shops?

  Here are rel­a­tively brief re­sponses on these 3 par­tic­u­lar points; I’ve made a sep­a­rate com­ment which lays out my think­ing on met­rics like the Big 5 which pro­vides some con­text for these re­sponses.

  We have con­tinued to col­lect mea­sures like the ones in the 2015 lon­gi­tu­di­nal study. We are mainly an­a­lyz­ing them in large batches, rather than work­shop to work­shop, be­cause the sam­ple size isn’t big enough to dis­t­in­guish sig­nal from noise for sin­gle work­shops. One of the pro­jects that I’m cur­rently work­ing on is an anal­y­sis of a cou­ple years of these data.

  The 2017 im­pact re­port was not in­tended as a com­pre­hen­sive ac­count of all of CFAR’s met­rics, it was just fo­cused on CFAR’s EA im­pact. So it looked at the data that were most di­rectly re­lated to CFAR alums’ im­pact on the world, and “on av­er­age alums have some in­crease in con­scien­tious­ness” seemed less rele­vant than the in­for­ma­tion that we did in­clude. The first few para­graphs of the re­port say more about this.

  I’m cu­ri­ous why you’re es­pe­cially in­ter­ested in Raven’s Pro­gres­sive Ma­tri­ces. I haven’t looked closely at the liter­a­ture on it, but my im­pres­sion is that it’s one of many met­rics which are loosely re­lated to the thing that we mean by “ra­tio­nal­ity.” It has the method­olog­i­cal ad­van­tage of be­ing a perfor­mance score rather than self-re­port (though this is par­tially offset by the pos­si­bil­ity of prac­tice effects and effort effects). The big dis­ad­van­tage is the one that Kaj pointed to: it seems to track rel­a­tively sta­ble as­pects of a per­son’s think­ing skills, and might not change much even if a per­son made large im­prove­ments. For in­stance, I could imag­ine a per­son de­vel­op­ing MacGyver-level prob­lem-solv­ing abil­ity while hav­ing lit­tle or no change in their Raven’s score.

• Unnamed 14 Sep 2018 20:20 UTC

  22 points

  on: No stan­dard met­ric for CFAR work­shops?

  Here’s a sketch of my think­ing about the use­ful­ness of met­rics like the Big 5 for what CFAR is try­ing to do.

  It would be con­ve­nient if there was a defini­tive mea­sure of a per­son’s ra­tio­nal­ity which closely matched what we mean by the term and was highly sen­si­tive to changes. But as far as I can tell there isn’t one, and there isn’t likely to be one any­time soon. So we rely on a mix of in­di­ca­tors, in­clud­ing some that are more like sys­tem­atic met­rics, some that are more like in­di­vi­d­u­als’ sub­jec­tive im­pres­sions, and some that are in be­tween.

  I think of the es­tab­lished psy­chol­ogy met­rics (Big 5, life satis­fac­tion, gen­eral self-effi­cacy, etc.) as pri­mar­ily pro­vid­ing a san­ity check on whether the work­shop is do­ing some­thing, along with a very very rough pic­ture of some of what it is do­ing. They are quan­ti­ta­tive mea­sures that don’t rely on staff mem­bers’ sub­jec­tive im­pres­sions of par­ti­ci­pants, they have been val­i­dated (at least to some ex­tent) in ex­ist­ing psy­chol­ogy re­search, and they seem at least loosely re­lated to the effects that CFAR hopes to have. And, com­pared to other ways of eval­u­at­ing CFAR’s im­pact on in­di­vi­d­u­als, they’re rel­a­tively easy for an out­sider to make sense of.

  A ma­jor limi­ta­tion of these es­tab­lished psy­chol­ogy met­rics is that they haven’t been that helpful as feed­back loops. One of the main pur­poses of a met­ric is to provide in­put into CFAR’s day-to-day and work­shop-to-work­shop efforts to de­velop bet­ter tech­niques and re­fine the work­shop. That is hard to do with met­rics like the ones in the lon­gi­tu­di­nal study, be­cause of a com­bi­na­tion of a few fac­tors:

  1. The re­sults aren’t available un­til sev­eral months af­ter the work­shop, which would make for very slow feed­back loops and iter­a­tion.

  2. The re­sults are too noisy to tell if changes from one work­shop to the next are just ran­dom vari­a­tion. It takes sev­eral work­shops worth of data to get a clear sig­nal on most of the met­rics.

  3. Th­ese met­rics are only loosely re­lated to what we care about. If a change to the work­shop leads to larger in­creases in con­scien­tious­ness that does not nec­es­sar­ily mean that we want to do it, and when a cur­ricu­lum de­vel­oper is work­ing on a class they are gen­er­ally not that in­ter­ested in these par­tic­u­lar met­rics.

  4. Th­ese met­rics are rel­a­tively gen­eral/​coarse in­di­ca­tors of the effect of the work­shop as a whole, not tied to par­tic­u­lar in­puts. So (for ex­am­ple) if we make some changes to the TAPs class and want to see if the new ver­sion of the class works bet­ter or worse, there isn’t a met­ric that iso­lates the effects of the TAPs class from the rest of the work­shop.

• Unnamed 6 Sep 2018 18:59 UTC

  50 points

  on: No stan­dard met­ric for CFAR work­shops?

  (This is Dan from CFAR)

  CFAR’s 2015 Lon­gi­tu­di­nal Study mea­sured the Big 5 and some other stan­dard psy­chol­ogy met­rics. It did find changes in­clud­ing de­creased neu­roti­cism and in­creased con­scien­tious­ness.

• Unnamed 5 Sep 2018 20:28 UTC

  16 points

  in reply to: Bucky's comment on: Birth or­der effect found in No­bel Lau­re­ates in Physics

  Seems in­ter­est­ing to get data on:

  Some group that isn’t heav­ily se­lected for in­tel­li­gence /​ in­tel­lec­tual cu­ri­os­ity: skate­board­ers, protestors, pro­fes­sional hockey play­ers...

  Some non-STEM group that is se­lected for suc­cess based on men­tal skills: liter­a­ture lau­re­ates, gov­er­nors, …

  Not sure which groups it would be easy to get data on.

  There is also the op­tion of look­ing into ex­ist­ing re­search on birth or­der to see what groups other peo­ple have already looked at.

• Unnamed 4 Sep 2018 23:45 UTC

  28 points

  on: nos­talge­braist—bayes: a kinda-sorta masterpost

  Seems worth not­ing that nos­talge­braist pub­lished this post in June 2017, which was (for ex­am­ple) be­fore Eliezer’s post on toolbox think­ing.

• Unnamed 4 Sep 2018 22:41 UTC

  27 points

  on: Birth or­der effect found in No­bel Lau­re­ates in Physics

  Now that we have data on LWers/​SSCers, math­e­mat­i­ci­ans, and physi­cists, if any­one wants to put more work into this I’d like to see them look some­place differ­ent. We don’t want to fall into the Wa­son 2-4-6 trap of only look­ing for birth or­der effects among smart STEM folks. We want data that can dis­t­in­guish Scott’s in­tel­li­gence /​ in­tel­lec­tual cu­ri­os­ity hy­poth­e­sis from other pos­si­bil­ities like some non-big-5 per­son­al­ity differ­ence or a gen­eral first­borns more likely phe­nomenon.

• Unnamed 21 Aug 2018 7:14 UTC

  6 points

  in reply to: gwern's comment on: His­tor­i­cal math­e­mat­i­ci­ans ex­hibit a birth or­der effect too

  For each math­e­mat­i­cian, ac­tual first­born­ness was coded as 0 or 1, and ex­pected first­born­ness as 1/​n (where n is the num­ber of chil­dren that their par­ents had). Then we just did a paired t-test, which is equiv­a­lent to sub­tract­ing ac­tual minus ex­pected for each data point and then do­ing a one sam­ple t-test against a mean of 0. You can see this all in Eli’s spread­sheet here; the data are also all there for you to try other statis­ti­cal tests if you want to.

• Unnamed 14 Aug 2018 7:53 UTC

  2 points

  in reply to: Said Achmiz's comment on: The ever ex­pand­ing moral circle

  You could think of CEV ap­plied to a sin­gle uni­tary agent as a spe­cial case where achiev­ing co­her­ence is triv­ial. It’s an edge case where the prob­lem be­comes eas­ier, rather than an edge case where the con­cepts threaten to break.

  Although this ter­minol­ogy makes it harder to talk about sev­eral agents who each sep­a­rately have their own ex­trap­o­lated vo­li­tion (as you were try­ing to do in your origi­nal com­ment in this thread). Though re­plac­ing it with Per­sonal Ex­trap­o­lated Vo­li­tion only helps a lit­tle, if we also want to talk about sev­eral sep­a­rately groups who each have their own within-group ex­trap­o­lated vo­li­tion (which is co­her­ent within each group but not be­tween groups).

• Unnamed 13 Aug 2018 5:40 UTC

  9 points

  on: Log­a­r­ithms and To­tal Utilitarianism

  Look­ing at the math of di­vid­ing a fixed pool of re­sources among a non-fixed num­ber of peo­ple, a fea­ture of log(r) that mat­ters a lot is that log(0)<0. The first unit of re­sources that you give to a per­son is es­sen­tially wasted, be­cause it just gets them up to 0 util­ity (which is no bet­ter than just hav­ing 1 fewer per­son around).

  That fa­vors hav­ing fewer peo­ple, so that you don’t have to keep wast­ing that first unit of re­source on each per­son. If the util­ity func­tion for a per­son in terms of their re­sources was f(r)=r-1 you would similarly find that it is best not to have too many peo­ple (in that case hav­ing ex­actly 1 per­son would work best).

  Whereas if it was f(r)=sqrt(r) then it would be best to have as many peo­ple as pos­si­ble, be­cause you’re start­ing from 0 util­ity at 0 re­sources and sqrt is steep­est right near 0. Do­ing the calcu­la­tion… if you have R units of re­sources di­vided equally among N peo­ple, the to­tal util­ity is sqrt(RN). log(1+r) is similar to sqrt—it in­creases as N in­creases—but it is bounded if R is fixed and just ap­proaches that bound (if we use nat­u­ral log, that bound is just R).

  To sum up: diminish­ing marginal util­ity fa­vors hav­ing more peo­ple each with fewer re­sources (in ad­di­tion to fa­vor­ing equal dis­tri­bu­tion of re­sources), f(0)<0 fa­vors hav­ing fewer peo­ple each with more re­sources (to avoid “wast­ing” the bit of re­sources that get a per­son up to 0 util­ity), and func­tions with both fea­tures like log(r) fa­vor some in­ter­me­di­ate solu­tion with a mod­er­ate pop­u­la­tion size.

• Unnamed 13 Aug 2018 3:26 UTC

  4 points

  in reply to: shminux's comment on: Log­a­r­ithms and To­tal Utilitarianism

  To­tal util­i­tar­i­anism does im­ply the re­pug­nant con­clu­sion, very straight­for­wardly.

  For ex­am­ple, imag­ine that world A has 1000000000000000000 peo­ple each with 10000000 util­ity and world Z has 10000000000000000000000000000000000000000 peo­ple each with 0.0000000001 util­ity. Which is bet­ter?

  To­tal util­i­tar­i­anism says that you just mul­ti­ply. World A has 10^18 peo­ple x 10^7 util­ity per per­son = 10^25 to­tal util­ity. World Z has 10^40 peo­ple x 10^-10 util­ity per per­son = 10^30 to­tal util­ity. World Z is way bet­ter.

  This seems re­pug­nant; in­tu­itively world Z is much worse than world A.

  Parfit went through clev­erer steps be­cause he wanted his ar­gu­ment to ap­ply more gen­er­ally, not just to to­tal util­i­tar­i­anism. Even much weaker as­sump­tions can get to this re­pug­nant-seem­ing con­clu­sion that a world like Z is bet­ter than a world like A.

  The point is that lots of peo­ple are con­fused about ax­iol­ogy. When they try to give opinions about pop­u­la­tion ethics, judg­ing in var­i­ous sce­nar­ios whether one hy­po­thet­i­cal world is bet­ter than an­other, they’ll wind up mak­ing judg­ments that are in­con­sis­tent with each other.

• Unnamed 13 Aug 2018 3:13 UTC

  2 points

  in reply to: shminux's comment on: Log­a­r­ithms and To­tal Utilitarianism

  The para­graph that I was quot­ing from was just about diminish­ing marginal util­ity and equal­ity/​re­dis­tri­bu­tion, not about the re­pug­nant con­clu­sion in par­tic­u­lar.