Rob Bensinger

• Rob Bensinger 21 Sep 2019 3:44 UTC

  7 points

  on: Ques­tion about a past donor to MIRI.

  (I work at MIRI.)
  
  News that SIAI re­ceived funds from Ep­stein ac­tu­ally came as a sur­prise to us. (We found out about this a few days be­fore OP’s ques­tion went up.) Ep­stein had pre­vi­ously ap­proached us in 2016 look­ing for or­ga­ni­za­tions to donate to, and we de­cided against pur­su­ing the op­tion; we didn’t re­al­ize there was any pre­vi­ous in­ter­ac­tion be­tween MIRI/​SIAI and Ep­stein or his foun­da­tions.
  
  The 2009 dona­tion was brought to our at­ten­tion when some­one sent us a Mi­ami Her­ald ar­ti­cle that in­cluded SIAI in a spread­sheet of or­ga­ni­za­tions that re­ceived money from one of Ep­stein’s foun­da­tions. We couldn’t ini­tially find ev­i­dence of the dona­tion in our records, so we had to go dig­ging a bit; it was ap­par­ently seed money for OpenCog while they were get­ting up and run­ning rather than money for “real” SIAI stuff, hence cur­rent staff be­ing out of the loop.

• Rob Bensinger 16 Sep 2019 16:16 UTC

  4 points

  in reply to: calebo's comment on: Real­ism and Rationality

  Sayre-McCord in SEP’s “Mo­ral Real­ism” ar­ti­cle:

  Mo­ral re­al­ists are those who think that [...] moral claims do pur­port to re­port facts and are true if they get the facts right. More­over, they hold, at least some moral claims ac­tu­ally are true. [...]

  As a re­sult, those who re­ject moral re­al­ism are use­fully di­vided into (i) those who think moral claims do not pur­port to re­port facts in light of which they are true or false (noncog­ni­tivists) and (ii) those who think that moral claims do carry this pur­port but deny that any moral claims are ac­tu­ally true (er­ror the­o­rists).

  Joyce in SEP’s “Mo­ral Anti-Real­ism” ar­ti­cle:

  Tra­di­tion­ally, to hold a re­al­ist po­si­tion with re­spect to X is to hold that X ex­ists in a mind-in­de­pen­dent man­ner (in the rele­vant sense of “mind-in­de­pen­dence”). On this view, moral anti-re­al­ism is the de­nial of the the­sis that moral prop­er­ties—or facts, ob­jects, re­la­tions, events, etc. (what­ever cat­e­gories one is will­ing to coun­te­nance)—ex­ist mind-in­de­pen­dently. This could in­volve ei­ther (1) the de­nial that moral prop­er­ties ex­ist at all, or (2) the ac­cep­tance that they do ex­ist but that ex­is­tence is (in the rele­vant sense) mind-de­pen­dent. Bar­ring var­i­ous com­pli­ca­tions to be dis­cussed be­low, there are broadly two ways of en­dors­ing (1): moral noncog­ni­tivism and moral er­ror the­ory. Pro­po­nents of (2) may be var­i­ously thought of as moral non-ojec­tivists, or ideal­ists, or con­struc­tivists.

  So, ev­ery­one defines “non-re­al­ism” so as to in­clude er­ror the­ory and non-cog­ni­tivism; some peo­ple define it so as to also in­clude all or most views on which moral prop­er­ties are in some sense “sub­jec­tive.”

  Th­ese am­bi­gui­ties seem like good rea­sons to just avoid the term “re­al­ism” and talk about more spe­cific po­si­tions, though I guess it works to think about a slid­ing scale where sub­stan­tive re­al­ism is at one ex­treme, er­ror the­ory and non-cog­ni­tivism are at the other ex­treme, and re­main­ing views are some­where in the mid­dle.

• Rob Bensinger 16 Sep 2019 16:00 UTC

  8 points

  in reply to: Rob Bensinger's comment on: Real­ism and Rationality

  Ex­am­ple: Eliezer’s Ex­trap­o­lated Vo­li­tion is easy to round off to “con­struc­tivism”, By Which It May Be Judged to “sub­stan­tive re­al­ism”, and Orthog­o­nal­ity Th­e­sis and The Gift We Give To To­mor­row to “sub­jec­tivism”. I’m guess­ing it’s not a co­in­ci­dence that those are also the most pop­u­lar an­swers in the poll above, and that no one of them has ma­jor­ity sup­port.

  (Though I don’t think I could have made a strong pre­dic­tion like this a pri­ori. If non-cog­ni­tivism or er­ror the­ory had done bet­ter, some­one could have said “well, of course!”, cit­ing LessWrong’s in­ter­est in sig­nal­ing or their gen­eral re­duc­tion­ist/​elimi­na­tivist/​anti-su­per­nat­u­ral­ist ten­den­cies.)

• Rob Bensinger 16 Sep 2019 15:33 UTC

  13 points

  on: Real­ism and Rationality

  The most pop­u­lar meta-eth­i­cal views on LessWrong seem to be rel­a­tively re­al­ist ones, with views like non-cog­ni­tivism and er­ror the­ory get­ting sig­nifi­cantly less sup­port. From the 2016 LessWrong di­as­pora sur­vey (ex­clud­ing peo­ple who didn’t pick one of the op­tions):

  • 772 re­spon­dents (39.5%) voted for “Con­struc­tivism: Some moral state­ments are true, and the truth of a moral state­ment is de­ter­mined by whether an agent would ac­cept it if they were un­der­go­ing a pro­cess of ra­tio­nal de­liber­a­tion. ‘Mur­der is wrong’ can mean some­thing like ‘So­cietal agree­ment to the rule “do
    not mur­der” is in­stru­men­tally ra­tio­nal’.”

  • 550 re­spon­dents (28.2%) voted for “Sub­jec­tivism: Some moral state­ments are true, but not uni­ver­sally, and the truth of a moral state­ment is de­ter­mined by non-uni­ver­sal opinions or pre­scrip­tions, and there is no nonat­ti­tu­di­nal de­ter­mi­nant of right­ness and wrong­ness. ‘Mur­der is wrong’ means some­thing like
    ’My cul­ture has judged mur­der to be wrong’ or ‘I’ve judged mur­der to be wrong’.”

  • 346 re­spon­dents (17.7%) voted for “Sub­stan­tive re­al­ism: Some moral state­ments are true, and the truth of a moral state­ment is de­ter­mined by mind-in­de­pen­dent moral prop­er­ties. ‘Mur­der is wrong’ means that mur­der has an ob­jec­tive mind-in­de­pen­dent prop­erty of wrong­ness that we dis­cover by em­piri­cal in­ves­ti­ga­tion, in­tu­ition, or some other method.”

  • 186 re­spon­dents (9.5%) voted for “Non-cog­ni­tivism: Mo­ral state­ments don’t ex­press propo­si­tions and can nei­ther be true nor false. ‘Mur­der is wrong’ means some­thing like ‘Boo mur­der!’.”

  • 99 re­spon­dents (5.1%) voted for “Er­ror the­ory: Mo­ral state­ments have a truth-value, but at­tempt to de­scribe fea­tures of the world that don’t ex­ist. ‘Mur­der is wrong’ and ‘Mur­der is right’ are both false state­ments be­cause moral right­ness and wrong­ness aren’t fea­tures that ex­ist.”

  I sus­pect that a lot of ra­tio­nal­ists would be happy to en­dorse any of the above five views in differ­ent con­texts or on differ­ent fram­ings, and would say that real-world moral judg­ment is com­pli­cated and doesn’t cleanly fit into ex­actly one of these cat­e­gories. E.g., I think Luke Muehlhauser’s Plu­ral­is­tic Mo­ral Re­duc­tion­ism is just cor­rect.

• Rob Bensinger 8 Sep 2019 20:18 UTC

  2 points

  in reply to: Rob Bensinger's comment on: Rob B’s Short­form Feed

  Old dis­cus­sion of this on LW: https://​​www.less­wrong.com/​​s/​​fqh9TLuo­qux­p­ducDb/​​p/​​syn­sRtBKDeAFuo7e3

• Rob Bensinger 8 Sep 2019 2:58 UTC

  23 points

  on: Rob B’s Short­form Feed

  Face­book com­ment I wrote in Fe­bru­ary, in re­sponse to the ques­tion ‘Why might hav­ing beauty in the world mat­ter?’:

  I as­sume you’re ask­ing about why it might be bet­ter for beau­tiful ob­jects in the world to ex­ist (even if no one ex­pe­riences them), and not ask­ing about why it might be bet­ter for ex­pe­riences of beauty to ex­ist.

  [… S]ome rea­sons I think this:

  1. If it cost me liter­ally noth­ing, I feel like I’d rather there ex­ist a planet that’s beau­tiful, or­nate, and com­plex than one that’s dull and sim­ple—even if the planet can never be seen or vis­ited by any­one, and has no other im­pact on any­one’s life. This feels like a weak prefer­ence, but it helps get a foot in the door for beauty.

  (The ob­vi­ous coun­ter­ar­gu­ment here is that my brain might be bad at simu­lat­ing the sce­nario where there’s liter­ally zero chance I’ll ever in­ter­act with a thing; or I may be oth­er­wise con­fused about my val­ues.)

  2. Another weak foot-in-the-door ar­gu­ment: Peo­ple seem to value beauty, and some peo­ple claim to value it ter­mi­nally. Since hu­man value is com­pli­cated and messy and idiosyn­cratic (com­pare per­son-spe­cific ASMR trig­gers or nos­tal­gia trig­gers or culi­nary prefer­ences) and ter­mi­nal and in­stru­men­tal val­ues are eas­ily al­tered and in­ter­changed in our brain, our prior should be that at least some peo­ple re­ally do have weird prefer­ences like that at least some of the time.

  (And if it’s just a few other peo­ple who value beauty, and not me, I should still value it for the sake of al­tru­ism and co­op­er­a­tive­ness.)

  3. If moral­ity isn’t “spe­cial”—if it’s just one of many facets of hu­man val­ues, and isn’t a par­tic­u­larly nat­u­ral-kind-ish facet—then it’s like­lier that a full un­der­stand­ing of hu­man value would lead us to treat aes­thetic and moral prefer­ences as more co­ex­ten­sive, in­ter­con­nected, and fuzzy. If I can value some­one else’s hap­piness in­her­ently, with­out need­ing to ex­pe­rience or know about it my­self, it then be­comes harder to say why I can’t value non-con­scious states in­her­ently; and “beauty” is an ob­vi­ous can­di­date. My prefer­ences aren’t all about my own ex­pe­riences, and they aren’t sim­ple, so it’s not clear why aes­thetic prefer­ences should be an ex­cep­tion to this rule.

  4. Similarly, if phe­nom­e­nal con­scious­ness is fuzzy or fake, then it be­comes less likely that our prefer­ences range only and ex­actly over sub­jec­tive ex­pe­riences (or their clos­est non-fake coun­ter­parts). Which re­moves the main rea­son to think un­ex­pe­rienced beauty doesn’t mat­ter to peo­ple.

  Com­bin­ing the lat­ter two points, and the liter­a­ture on emo­tions like dis­gust and pu­rity which have both moral and non-moral as­pects, it seems plau­si­ble that the ex­trap­o­lated ver­sions of prefer­ences like “I don’t like it when other sen­tient be­ings suffer” could turn out to have aes­thetic as­pects or in­ter­pre­ta­tions like “I find it ugly for brain re­gions to have suffer­ing-ish con­figu­ra­tions”.

  Even if con­scious­ness is fully a real thing, it seems as though a suffi­ciently deep re­duc­tive un­der­stand­ing of con­scious­ness should lead us to un­der­stand and eval­u­ate con­scious­ness similarly whether we’re think­ing about it in in­ten­tional/​psy­chol­o­giz­ing terms or just think­ing about the phys­i­cal struc­ture of the cor­re­spond­ing brain state. We shouldn’t be more out­raged by a world-state un­der one de­scrip­tion than un­der an equiv­a­lent de­scrip­tion, ideally.

  But then it seems less ob­vi­ous that the brain states we care about should ex­actly cor­re­spond to the ones that are con­scious, with no other brain states mat­ter­ing; and aes­thetic emo­tions are one of the main ways we re­late to things we’re treat­ing as phys­i­cal sys­tems.

  As a con­crete ex­am­ple, maybe our ideal selves would find it in­her­ently dis­gust­ing for a brain state that sort of al­most looks con­scious to go through the mo­tions of be­ing tor­tured, even when we aren’t the least bit con­fused or un­cer­tain about whether it’s re­ally con­scious, just be­cause our ter­mi­nal val­ues are as­so­ci­a­tive and sym­bolic. I use this ex­am­ple be­cause it’s an es­pe­cially easy one to un­der­stand from a moral­ity- and con­scious­ness-cen­tered per­spec­tive, but I ex­pect our ideal prefer­ences about phys­i­cal states to end up be­ing very weird and com­pli­cated, and not to end up be­ing all that much like our moral in­tu­itions to­day.

  Ad­den­dum: As always, this kind of thing is ridicu­lously spec­u­la­tive and not the kind of thing to put one’s weight down on or try to “lock in” for civ­i­liza­tion. But it can be use­ful to keep the range of op­tions in view, so we have them in mind when we figure out how to test them later.

• Rob Bensinger 30 Aug 2019 23:35 UTC

  7 points

  in reply to: Liam Donovan's comment on: [Link] Book Re­view: Refram­ing Su­per­in­tel­li­gence (SSC)

  It might also be worth com­par­ing CAIS and “tool AI” to Paul Chris­ti­ano’s IDA and the desider­ata MIRI tends to talk about (task-di­rected AGI [1,2,3], mild op­ti­miza­tion, limited AGI).

  At a high level, I tend to think of Chris­ti­ano and Drexler as both ap­proach­ing al­ign­ment from very much the right an­gle, in that they’re (a) try­ing to break apart the vague idea of “AGI rea­son­ing” into smaller parts, and (b) shoot­ing for a sys­tem that won’t op­ti­mize harder (or more do­main-gen­er­ally) than we need for a given task. From con­ver­sa­tions with Nate, one way I’d sum­ma­rize MIRI-cluster dis­agree­ments with Chris­ti­ano and Drexler’s pro­pos­als is that MIRI peo­ple don’t tend to think these pro­pos­als de­com­pose cog­ni­tive work enough. Without a lot more de­com­po­si­tion/​un­der­stand­ing, ei­ther the sys­tem as a whole won’t be ca­pa­ble enough, or it will be ca­pa­ble by virtue of atomic parts that are smart enough to be dan­ger­ous, where safety is a mat­ter of how well we can open those black boxes.

  In my ex­pe­rience peo­ple use “tool AI” to mean a bunch of differ­ent things, in­clud­ing things MIRI con­sid­ers very im­por­tant and use­ful (like “only works on a limited task, rather than putting any cog­ni­tive work into more gen­eral top­ics or try­ing to open-end­edly op­ti­mize the fu­ture”) as well as ideas that don’t seem rele­vant or that ob­scure where the hard parts of the prob­lem prob­a­bly are.

• Rob Bensinger 24 Aug 2019 12:13 UTC

  2 points

  in reply to: cousin_it's comment on: how should a sec­ond ver­sion of “ra­tio­nal­ity: A to Z” look like?

  For Face­book, I use FBPu­rity to block my news feed. Then if there are par­tic­u­lar in­di­vi­d­u­als I es­pe­cially want to fol­low, I add them to a Face­book List.

• Rob Bensinger 17 Aug 2019 22:44 UTC

  9 points

  in reply to: Unreal's comment on: Par­tial sum­mary of de­bate with Ben­quo and Jes­si­cata [pt 1]

  For ‘things that aren’t an ac­ci­dent but aren’t nec­es­sar­ily con­scious or en­dorsed’, an­other op­tion might be to use lan­guage like ‘de­ci­sion’, ‘ac­tion’, ‘choice’, etc. but flagged in a way that makes it clear you’re not as­sum­ing full con­scious­ness. Like ‘quasi-de­ci­sion’, ‘quasi-ac­tion’, ‘quasi-con­scious’… Ap­plied to Zack’s case, that might sug­gest a term like ‘quasi-dis­sem­bling’ or ‘quasi-mis­lead­ing’. ‘Dis­so­nant com­mu­ni­ca­tion’ comes to mind as an­other idea.

  When I want to em­pha­size that there’s op­ti­miza­tion go­ing on but it’s not nec­es­sar­ily con­scious, I some­times speak im­per­son­ally of “Bob’s brain is do­ing X”, or “a Bob-part/​agent/​sub­agent is do­ing X”.

• Rob Bensinger 17 Aug 2019 13:03 UTC

  29 points

  in reply to: jessicata's comment on: Par­tial sum­mary of de­bate with Ben­quo and Jes­si­cata [pt 1]

  I per­son­ally wouldn’t point to “When Will AI Ex­ceed Hu­man Perfor­mance?” as an ex­em­plar on this di­men­sion, be­cause it isn’t clear about the in­ter­est­ing im­pli­ca­tions of the facts it’s re­port­ing. Katja’s take-away from the pa­per was:

  In the past, it seemed pretty plau­si­ble that what AI re­searchers think is a de­cent guide to what’s go­ing to hap­pen. I think we’ve pretty much demon­strated that that’s not the case. I think there are a va­ri­ety of differ­ent ways we might go about try­ing to work out what AI timelines are like, and talk­ing to ex­perts is one of them; I think we should weight that one down a lot.

  I don’t know whether Katja’s co-au­thors agree with her about that sum­mary, but if there’s dis­agree­ment, I think the pa­per still could have in­cluded more dis­cus­sion of the ques­tion and which find­ings look rele­vant to it.

  The ac­tual Dis­cus­sion sec­tion makes the op­po­site ar­gu­ment in­stead, list­ing a bunch of rea­sons to think AI ex­perts are good at fore­see­ing AI progress. The in­tro­duc­tion says “To pre­pare for these challenges, ac­cu­rate fore­cast­ing of trans­for­ma­tive AI would be in­valuable. [...] The pre­dic­tions of AI ex­perts provide cru­cial ad­di­tional in­for­ma­tion.” And the pa­per in­cludes a list of four “key find­ings”, none of which even raise the ques­tion of sur­vey re­spon­dents’ fore­cast­ing chops, and all of which are worded in ways that sug­gest we should in fact put some weight on the re­spon­dents’ views (some­times switch­ing be­tween the phras­ing ‘re­searchers be­lieve X’ and ‘X is true’).

  The ab­stract men­tions the main find­ing that un­der­mines how be­liev­able the re­sponses are, but does so in such a way that some­one read­ing through quickly might come away with the op­po­site im­pres­sion. The ab­stract’s struc­ture is:

  To adapt pub­lic policy, we need to bet­ter an­ti­ci­pate [AI ad­vances]. Re­searchers pre­dict [A, B, C, D, E, and F]. Re­searchers be­lieve [G and H]. Th­ese re­sults will in­form dis­cus­sion amongst re­searchers and poli­cy­mak­ers about an­ti­ci­pat­ing and man­ag­ing trends in AI.

  If it slips past your at­ten­tion that G and H are mas­sively in­con­sis­tent, it’s easy for the reader to come away think­ing the ab­stract is say­ing ‘Here’s a list of of cred­ible state­ments from ex­perts about their area of ex­per­tise’ as op­posed to ‘Here’s a demon­stra­tion that what AI re­searchers think is not a de­cent guide to what’s go­ing to hap­pen’.

• Rob Bensinger 6 Aug 2019 22:11 UTC

  7 points

  in reply to: habryka's comment on: Oc­cam’s Ra­zor: In need of sharp­en­ing?

  Hu­mans might not be a low-level atom, but ob­vi­ously we have to priv­ilege the hy­poth­e­sis ‘some­thing hu­man-like did this’ if we’ve already ob­served a lot of hu­man-like things in our en­vi­ron­ment.

  Sup­pose I’m a mem­ber of a pre­his­toric tribe, and I see a fire in the dis­tance. It’s fine for me to say ‘I have a low-ish prior on a hu­man start­ing the fire, be­cause (AFAIK) there are only a few dozen hu­mans in the area’. And it’s fine for me to say ‘I’ve never seen a hu­man start a fire, so I don’t think a hu­man started this fire’. But it’s not fine for me to say ‘It’s very un­likely a hu­man started that fire, be­cause hu­man brains are more com­pli­cated than other phe­nom­ena that might start fires’, even if I cor­rectly in­tuit how and why hu­mans are more com­pli­cated than other phe­nom­ena.

  The case of Thor is a bit more com­pli­cated, be­cause gods are differ­ent from hu­mans. If Eliezer and cousin_it dis­agree on this point, maybe Eliezer would say ‘The com­plex­ity of the hu­man brain is the biggest rea­son why you shouldn’t in­fer that there are other, as-yet-un­ob­served species of hu­man-brain-ish things that are very differ­ent from hu­mans’, and maybe cousin_it would say ‘No, it’s pretty much just the differ­ent­ness-from-ob­served-hu­mans (on the “has di­rect con­trol over el­e­men­tal forces” di­men­sion) that mat­ters, not the fact that it has a com­pli­cated brain.’

  If that’s a good char­ac­ter­i­za­tion of the dis­agree­ment, then it seems like Eliezer might say ‘In an­cient so­cieties, it was much more rea­son­able to posit mind­less “su­per­nat­u­ral” phe­nom­ena (i.e., mind­less phys­i­cal mechanisms wildly differ­ent from any­thing we’ve ob­served) than to posit in­tel­li­gent su­per­nat­u­ral phe­nom­ena.’ Whereas the hy­po­thet­i­cal cousin-it might say that an­cient peo­ple didn’t have enough ev­i­dence to con­clude that gods were any more un­likely than mind­less mechanisms that were similarly differ­ent from ex­pe­rience. Ex­am­ple ques­tion: what prob­a­bil­ity should an­cient peo­ple have as­signed to

  The reg­u­lar mo­tion of the planets is due to a ran­dom pro­cess plus a mind­less in­visi­ble force, like the mind­less in­visi­ble force that causes re­cently-cooked food to cool down all on its own.

  vs.

  The reg­u­lar mo­tion of the planets is due to de­liber­ate de­sign /​ in­tel­li­gent in­ter­ven­tion, like the in­tel­li­gent in­ter­ven­tion that ar­ranges and cooks food.

• Rob Bensinger 6 Aug 2019 18:36 UTC

  3 points

  in reply to: Rob Bensinger's comment on: AI Align­ment Open Thread Au­gust 2019

  Also the dis­cus­sion of de­con­fu­sion re­search in https://​​in­tel­li­gence.org/​​2018/​​11/​​22/​​2018-up­date-our-new-re­search-di­rec­tions/​​ and https://​​www.less­wrong.com/​​posts/​​Gg9a4y8reWKtLe3Tn/​​the-rocket-al­ign­ment-prob­lem , and the sketch of ‘why this looks like a hard prob­lem in gen­eral’ in https://​​www.less­wrong.com/​​posts/​​zEvqFtT4AtTzt­fYC4/​​op­ti­miza­tion-am­plifies and https://​​ar­bital.com/​​p/​​al­ign­ing_adds_time/​​ .

• Rob Bensinger 6 Aug 2019 18:30 UTC

  13 points

  in reply to: FactorialCode's comment on: AI Align­ment Open Thread Au­gust 2019

  MIRIx events are funded by MIRI, but we don’t de­cide the top­ics or any­thing. I haven’t taken a poll of MIRI re­searchers to see how en­thu­si­as­tic differ­ent peo­ple are about for­mal ver­ifi­ca­tion, but AFAIK Nate and Eliezer don’t see it as su­per rele­vant. See https://​​www.less­wrong.com/​​posts/​​xCpuSfT5Lt6kkR3po/​​my-take-on-agent-foun­da­tions-for­mal­iz­ing-metaphilo­soph­i­cal#cGuMRFSi224RCNBZi and the idea of a “safety-story” in https://​​www.less­wrong.com/​​posts/​​8gqrbnW758qjHFTrH/​​se­cu­rity-mind­set-and-or­di­nary-para­noia for bet­ter at­tempts to char­ac­ter­ize what MIRI is look­ing for.

  ETA: From the end of the lat­ter di­alogue,

  In point of fact, the real rea­son the au­thor is list­ing out this method­ol­ogy is that he’s cur­rently try­ing to do some­thing similar on the prob­lem of al­ign­ing Ar­tifi­cial Gen­eral In­tel­li­gence, and he would like to move past “I be­lieve my AGI won’t want to kill any­one” and into a headspace more like writ­ing down state­ments such as “Although the space of po­ten­tial weight­ings for this re­cur­rent neu­ral net does con­tain weight com­bi­na­tions that would figure out how to kill the pro­gram­mers, I be­lieve that gra­di­ent de­scent on loss func­tion L will only ac­cess a re­sult in­side sub­space Q with prop­er­ties P, and I be­lieve a space with prop­er­ties P does not in­clude any weight com­bi­na­tions that figure out how to kill the pro­gram­mer.”

  Though this it­self is not re­ally a re­duced state­ment and still has too much goal-laden lan­guage in it.

  Rather than putting the em­pha­sis on be­ing able to ma­chine-ver­ify all im­por­tant prop­er­ties of the sys­tem, this puts the em­pha­sis on hav­ing strong tech­ni­cal in­sight into the sys­tem; I usu­ally think of for­mal proofs more as a means to that end. (Again caveat­ing that some peo­ple at MIRI might think of this differ­ently.)

• Rob Bensinger 2 Jun 2019 1:34 UTC

  4 points

  in reply to: MakoYass's comment on: Feed­back Re­quested! Draft of a New About/​Wel­come Page for LessWrong

  A tricky thing about this is that there’s an el­e­ment of cog­ni­tive dis­tor­tion in how most peo­ple eval­u­ate these ques­tions, and play-act­ing at “this dis­tor­tion makes sense” can worsen the dis­tor­tion (at the same time that it helps win more trust from peo­ple who have the dis­tor­tion).

  If it turned out to be a good idea to try to speak to this per­spec­tive, I’d recom­mend first med­i­tat­ing on a few re­ver­sal tests. Like: “Hmm, I wouldn’t feel any need to add a dis­claimer here if the text I was recom­mend­ing were The Brothers Kara­ma­zov, though I’d want to briefly say why it’s rele­vant, and I might worry about the length. I’d feel a bit wor­ried about recom­mend­ing a young adult novel, even an un­usu­ally di­dac­tic one, be­cause peo­ple rightly ex­pect YA nov­els to be op­ti­mized for less use­ful and ed­ify­ing things than the “liter­ary clas­sics” refer­ence class. The in­sights tend to be shal­lower and less com­mon. YA nov­els and fan­fic­tion are similar in all those re­spects, and they pro­voke ba­si­cally the same feel­ing in me, so I can maybe use that re­ver­sal test to de­ter­mine what kinds of dis­claimers or added con­text make sense here.”

• Rob Bensinger 1 Jun 2019 22:43 UTC

  2 points

  in reply to: Rob Bensinger's comment on: FB/​Dis­cord Style Reacts

  (If I want to ex­press stronger grat­i­tude than that, I’d rather write it out.)

• Rob Bensinger 1 Jun 2019 22:42 UTC

  2 points

  in reply to: Raemon's comment on: FB/​Dis­cord Style Reacts

  On slack, Thumbs Up, OK, and Horns hand signs meet all my minor needs for thank­ing peo­ple.

• Rob Bensinger 30 May 2019 1:28 UTC

  6 points

  in reply to: Raemon's comment on: Drown­ing chil­dren are rare

  Can’t in­di­vi­d­u­als just list ‘Reign of Ter­ror’ and then spec­ify in their per­son­al­ized de­scrip­tion that they have a high bar for ter­ror?

• Rob Bensinger 14 May 2019 19:46 UTC

  5 points

  in reply to: Said Achmiz's comment on: Co­her­ent de­ci­sions im­ply con­sis­tent utilities

  We’d talked about get­ting a dump out as well, and your plan sounds great to me! The LW team should get back to you with a list at some point (un­less they think of a bet­ter idea).

• Rob Bensinger 14 May 2019 3:44 UTC

  16 points

  in reply to: Said Achmiz's comment on: Co­her­ent de­ci­sions im­ply con­sis­tent utilities

  I asked Eliezer if it made sense to cross-post this from Ar­bital, and did the cross-post­ing when he ap­proved. I’m sorry it wasn’t clear that this was a cross-post! I in­tended to make this clearer, but my idea was bad (putting the in­for­ma­tion on the se­quence page) and I also im­ple­mented it wrong (the se­quence didn’t pre­vi­ously dis­play on the top of this post).

  This post was origi­nally writ­ten as a non­tech­ni­cal in­tro­duc­tion to ex­pected util­ity the­ory and co­her­ence ar­gu­ments. Although it be­gins in me­dia res stylis­ti­cally, it doesn’t have any pre­reqs or con­text be­yond “this is part of a col­lec­tion of in­tro­duc­tory re­sources cov­er­ing a wide va­ri­ety of tech­ni­cal and semitech­ni­cal top­ics.”

  Per the first sen­tence, the main pur­pose is for this to be a link­able re­source for con­ver­sa­tions/​in­quiry about hu­man ra­tio­nal­ity and con­ver­sa­tions/​in­quiry about AGI:

  So we’re talk­ing about how to make good de­ci­sions, or the idea of ‘bounded ra­tio­nal­ity’, or what suffi­ciently ad­vanced Ar­tifi­cial In­tel­li­gences might be like; and some­body starts drag­ging up the con­cepts of ‘ex­pected util­ity’ or ‘util­ity func­tions’. And be­fore we even ask what those are, we might first ask, Why?

  There have been loose plans for a while to cross-post con­tent from Ar­bital to LW (maybe all of it; maybe just the best or most in­ter­est­ing stuff), but as I men­tioned down­thread, we’re do­ing more cross-post ex­per­i­ments sooner than we would have be­cause Ar­bital’s been hav­ing se­ri­ous perfor­mance is­sues.

• Rob Bensinger 14 May 2019 3:34 UTC

  5 points

  in reply to: jimrandomh's comment on: Co­her­ent de­ci­sions im­ply con­sis­tent utilities

  I as­sume you mean ‘no one has this re­spon­si­bil­ity for Ar­bital any­more’, and not that there’s some­one else who has this re­spon­si­bil­ity.