Wei_Dai

The Main Sources of AI Risk?

• Wei_Dai 20 Mar 2019 19:36 UTC

  6 points

  AF

  in reply to: paulfchristiano's comment on: What’s wrong with these analo­gies for un­der­stand­ing In­formed Over­sight and IDA?

  In that case, you can still try to be a straight­for­ward Bayesian about it, and say “our in­tu­ition sup­ports the gen­eral claim that pro­cess P out­puts true state­ments;” you can then ap­ply that reg­u­lar­ity to trust P on some new claim even if it’s not the kind of claim you could ver­ify, as long as “P out­puts true state­ments” had a higher prior than “P out­puts true state­ments just in the cases I can check.”

  If that’s what you do, it seems “P out­puts true state­ments just in the cases I can check.” could have a pos­te­rior that’s al­most 50%, which doesn’t seem safe, es­pe­cially in an iter­ated scheme where you have to de­pend on such prob­a­bil­ities many times? Do you not need to re­duce the pos­te­rior prob­a­bil­ity to a neg­ligible level in­stead?

  See the sec­ond and third ex­am­ples in the post in­tro­duc­ing as­crip­tion uni­ver­sal­ity.

  Can you quote these ex­am­ples? The word “ex­am­ple” ap­pears 27 times in that post and look­ing at the literal sec­ond and third ex­am­ples, they don’t seem very rele­vant to what you’ve been say­ing here so I won­der if you’re refer­ring to some other ex­am­ples.

  There is definitely a lot of fuzzi­ness here and it seems like one of the most im­por­tant places to tighten up the defi­ni­tion /​ one of the big re­search ques­tions for whether as­crip­tion uni­ver­sal­ity is pos­si­ble.

  What I’m in­fer­ring from this (as far as a di­rect an­swer to my ques­tion) is that an over­seer try­ing to do In­formed Over­sight on some ML model doesn’t need to re­verse en­g­ineer the model enough to fully un­der­stand what it’s do­ing, only enough to make sure it’s not do­ing some­thing ma­lign, which might be a lot eas­ier, but this isn’t quite re­flected in the for­mal defi­ni­tion yet or isn’t a clear im­pli­ca­tion of it yet. Does that seem right?

[Question] What’s wrong with these analo­gies for un­der­stand­ing In­formed Over­sight and IDA?

• Wei_Dai 18 Mar 2019 8:13 UTC

  3 points

  AF

  in reply to: Stuart_Armstrong's comment on: A the­ory of hu­man values

  There is the is­sue of avoid­ing ig­no­rant-yet-con­fi­dent meta-prefer­ences, which I’m work­ing on writ­ing up right now (par­tially thanks to you very com­ment here, thanks!)

  I look for­ward to read­ing that. In the mean­time can you ad­dress my par­en­thet­i­cal point in the grand-par­ent com­ment: “cor­rectly ex­tract­ing William MacAskill’s meta-prefer­ences seems equiv­a­lent to learn­ing metaphilos­o­phy from William”? If it’s not clear, what I mean is that sup­pose Will wants to figure out his val­ues by do­ing philos­o­phy (which I think he ac­tu­ally does), does that mean that un­der you scheme the AI needs to learn how to do philos­o­phy? If so, how do you plan to get around the prob­lems with ap­ply­ing ML to metaphilos­o­phy that I de­scribed in Some Thoughts on Me­taphilos­o­phy?

• Wei_Dai 18 Mar 2019 7:59 UTC

  38 points

  AF

  on: More re­al­is­tic tales of doom

  I think AI risk is dis­junc­tive enough that it’s not clear most of the prob­a­bil­ity mass can be cap­tured by a sin­gle sce­nario/​story, even as broad as this one tries to be. Here are some ad­di­tional sce­nar­ios that don’t fit into this story or aren’t made very salient by it.

  1. AI-pow­ered memetic war­fare makes all hu­mans effec­tively in­sane.

  2. Hu­mans break off into var­i­ous groups to colonize the uni­verse with the help of their AIs. Due to in­suffi­cient “metaphilo­soph­i­cal pa­ter­nal­ism”, they each con­struct their own ver­sion of utopia which is ei­ther di­rectly bad (i.e., some of the “utopias” are ob­jec­tively ter­rible or sub­jec­tively ter­rible ac­cord­ing to my val­ues), or bad be­cause of op­por­tu­nity costs.

  3. AI-pow­ered economies have much higher economies of scale be­cause AIs don’t suffer from the kind of co­or­di­na­tion costs that hu­mans have (e.g., they can merge their util­ity func­tions and be­come clones of each other). Some coun­tries may try to pre­vent AI-man­aged com­pa­nies from merg­ing for ide­olog­i­cal or safety rea­sons, but oth­ers (in or­der to gain a com­pet­i­tive ad­van­tage on the world stage) will ba­si­cally al­low their whole econ­omy to be con­trol­led by one AI, which even­tu­ally achieves a de­ci­sive ad­van­tage over the rest of hu­man­ity and does a treach­er­ous turn.

  4. The same in­cen­tive for AIs to merge might also cre­ate an in­cen­tive for value lock-in, in or­der to fa­cil­i­tate the merg­ing. (AIs that don’t have util­ity func­tions might have a harder time co­or­di­nat­ing with each other.) Other in­cen­tives for pre­ma­ture value lock-in might in­clude defense against value ma­nipu­la­tion/​cor­rup­tion/​drift. So AIs end up em­body­ing locked-in ver­sions of hu­man val­ues which are ter­rible in light of our true/​ac­tual val­ues.

  5. I think the origi­nal “stereo­typed image of AI catas­tro­phe” is still quite plau­si­ble, if for ex­am­ple there is a large amount of hard­ware over­hang be­fore the last piece of puz­zle for build­ing AGI falls into place.

• Wei_Dai 18 Mar 2019 6:09 UTC

  7 points

  in reply to: John_Maxwell_IV's comment on: More re­al­is­tic tales of doom

  Sounds like a new fram­ing of the “dae­mon” idea.

  That’s my im­pres­sion as well. If it’s cor­rect, seems like it would be a good idea to men­tion that ex­plic­itly in the post, so peo­ple can link up the new con­cept with their old con­cept.

• Wei_Dai 18 Mar 2019 5:57 UTC

  5 points

  in reply to: Chris Leong's comment on: Com­par­i­son of de­ci­sion the­o­ries (with a fo­cus on log­i­cal-coun­ter­fac­tual de­ci­sion the­o­ries)

  See “Ex­am­ple 1: Coun­ter­fac­tual Mug­ging” in Towards a New De­ci­sion The­ory.

• Wei_Dai 18 Mar 2019 0:19 UTC

  13 points

  in reply to: Chris Leong's comment on: Com­par­i­son of de­ci­sion the­o­ries (with a fo­cus on log­i­cal-coun­ter­fac­tual de­ci­sion the­o­ries)

  I think it’s needed just to define what it means to con­di­tion on an ac­tion, i.e., if an agent con­di­tions on “I make this de­ci­sion” in or­der to com­pute its ex­pected util­ity, what does that mean for­mally? You could make “I” a prim­i­tive el­e­ment in the agent’s on­tol­ogy, but I think that runs into all kinds of prob­lems. My solu­tion was to make it a log­i­cal state­ment of the form “source code X out­puts ac­tion/​policy Y”, and then to con­di­tion on it you need a log­i­cally un­cer­tain dis­tri­bu­tion.

• Wei_Dai 18 Mar 2019 0:01 UTC

  4 points

  on: More re­al­is­tic tales of doom

  There’s a bunch of bul­let points be­low Part 1 and Part 2. Are these in­tended to be par­allel with them on the same level, or in­stances/​sub­cat­e­gories of them?

  Oh, this is only on GW. On LW it looks very differ­ent. Pre­sum­ably the LW ver­sion is the in­tended ver­sion.

• Wei_Dai 17 Mar 2019 21:19 UTC

  17 points

  in reply to: Chris Leong's comment on: Com­par­i­son of de­ci­sion the­o­ries (with a fo­cus on log­i­cal-coun­ter­fac­tual de­ci­sion the­o­ries)

  Chris asked me via PM, “I’m cu­ri­ous, have you writ­ten any posts about why you hold that po­si­tion?”

  I don’t think I have, but I’ll give the rea­sons here:

  1. “ev­i­den­tial-style con­di­tion­ing on a log­i­cally un­cer­tain dis­tri­bu­tion” seems sim­pler /​ more el­e­gant to me.

  2. I’m not aware of a com­pel­ling ar­gu­ment for “causal-graph-style coun­ter­pos­si­ble rea­son­ing”. There are definitely some un­re­solved prob­lems with ev­i­den­tial-style UDT and I do en­dorse peo­ple look­ing into causal-style FDT as an al­ter­na­tive but I’m not con­vinced the solu­tions ac­tu­ally lie in that di­rec­tion. (https://​side­ways-view.com/​2018/​09/​30/​edt-vs-cdt-2-con­di­tion­ing-on-the-im­pos­si­ble/​ and links therein are rele­vant here.)

  3. Part of it is just his­tor­i­cal, in that UDT was origi­nally speci­fied as “ev­i­den­tial-style con­di­tion­ing on a log­i­cally un­cer­tain dis­tri­bu­tion” and if I added my name as a co-au­thor to a pa­per that fo­cuses on causal-style de­ci­sion the­ory, peo­ple would nat­u­rally won­der if some­thing made me change my mind.

• Wei_Dai 17 Mar 2019 6:55 UTC

  5 points

  in reply to: jessicata's comment on: Privacy

  OK, look­ing at the ar­gu­ment, I think it makes sense that sig­nal­ling equil­ibria can po­ten­tially be Pareto-worse than non-sig­nal­ling equil­ibria, as they can have more of a “mar­ket for lemons” prob­lem.

  Not sure what the con­nec­tion to “mar­ket for lemons” is. Can you ex­plain more (if it seems im­por­tant)?

  (I think “no one gets ed­u­ca­tion, ev­ery­one gets paid av­er­age pro­duc­tivity” is still a Nash equil­ibrium)

  I agree that is still a Nash equil­ibrium and I think even a Perfect Bayesian Equil­ibrium, but there may be a stronger for­mal equil­ibrium con­cept that rules it out? (It’s been a while since I stud­ied all those equil­ibrium re­fine­ments so I can’t tell you which off the top of my head.)

  I think un­der Perfect Bayesian Equil­ibrium, off-the-play-path nodes for­mally hap­pen with prob­a­bil­ity 0 and the play­ers are al­lowed to up­date in an ar­bi­trary way on those nodes, in­clud­ing not up­date at all. But in­tu­itively if some­one does de­vi­ate from the pro­posed equil­ibrium strat­egy and get some ed­u­ca­tion, it seems im­plau­si­ble that em­ploy­ers don’t up­date to­wards them be­ing type H and there­fore offer them a higher salary.

• Wei_Dai 17 Mar 2019 5:43 UTC

  5 points

  in reply to: jessicata's comment on: Privacy

  It looks like the code that turns a URL into a link made the colon into part of the link. I re­moved it so the link should work now. The ar­gu­ment should be in the PDF. Ba­si­cally you just solve the game as­sum­ing the abil­ity to sig­nal and com­pare that to the game where sig­nal­ing isn’t pos­si­ble, and see that the sig­nal­ing equil­ibrium makes ev­ery­one worse off (in that par­tic­u­lar game).

• Wei_Dai 17 Mar 2019 5:10 UTC

  12 points

  on: Privacy

  We need a realm shielded from sig­nal­ing and judg­ment.

  To sup­port this, there are re­sults from eco­nomics /​ game the­ory show­ing that sig­nal­ing equil­ibria can be worse than non-sig­nal­ing equil­ibria (in the sense of Pareto in­effi­ciency). Quot­ing one ex­am­ple from http://​fac­ulty.econ.uc­davis.edu/​fac­ulty/​bo­nanno/​teach­ing/​200C/​Sig­nal­ing.pdf

  So the bench­mark is rep­re­sented by the situ­a­tion where no sig­nal­ing takes place and em­ploy­ers—not be­ing able to dis­t­in­guish be­tween more pro­duc­tive and less pro­duc­tive ap­pli­cants and not hav­ing any el­e­ments on which to base a guess—offer the same wage to ev­ery ap­pli­cant, equal to the av­er­age pro­duc­tivity. Call this the non-sig­nal­ing equil­ibrium. In a sig­nal­ing equil­ibrium (where em­ploy­ers’ be­liefs are con­firmed, since less pro­duc­tive peo­ple do not in­vest in ed­u­ca­tion, while the more pro­duc­tive do) ev­ery­body may be worse off than in the non-sig­nal­ing equil­ibrium. This oc­curs if the wage offered to the non-ed­u­cated is lower than the av­er­age pro­duc­tivity (= wage offered to ev­ery­body in the non-sig­nal­ing equil­ibrium) and that offered to the ed­u­cated peo­ple is higher, but be­comes lower (than the av­er­age pro­duc­tivity) once the costs of ac­quiring ed­u­ca­tion are sub­tracted. The pos­si­ble Pareto in­effi­ciency of sig­nal­ing equil­ibria is a strong re­sult and a wor­ry­ing one: it means that so­ciety is wast­ing re­sources in the pro­duc­tion of ed­u­ca­tion. How­ever, it is not per se enough to con­clude that ed­u­ca­tion (i.e. the sig­nal­ing ac­tivity) should be elimi­nated. The re­sult is not that, in gen­eral, elimi­na­tion of the sig­nal­ing ac­tivity leads to a Pareto im­prove­ment: Spence sim­ply pointed out that this is a pos­si­bil­ity.

  So in the­ory it seems quite pos­si­ble that pri­vacy is a sort of co­or­di­na­tion mechanism for avoid­ing bad sig­nal­ing equil­ibria. Whether or not it ac­tu­ally is, I’m not sure. That seems to re­quire em­piri­cal in­ves­ti­ga­tion and I’m not aware of such re­search.

• Wei_Dai 15 Mar 2019 19:00 UTC

  7 points

  in reply to: Rob Bensinger's comment on: Ques­tion: MIRI Cor­rig­bil­ity Agenda

  Is Jes­sica Tay­lor’s A first look at the hard prob­lem of cor­rigi­bil­ity still a good refer­ence or is it out­dated?

• Wei_Dai 15 Mar 2019 4:00 UTC

  9 points

  in reply to: Sherrinford's comment on: A the­ory of hu­man values

  I think in terms of eco­nomics, vNM ex­pected util­ity is clos­est to how we tend to think about util­ity/​prefer­ences. The prob­lem with vNM (from our per­spec­tive) is that it as­sumes a co­her­ent agent (i.e., an agent that satis­fies the vNM ax­ioms) but hu­mans aren’t co­her­ent, in part be­cause we don’t know what our val­ues are or should be. (“Hu­mans don’t have util­ity func­tions” is a com­mon re­frain around here.) From academia in gen­eral, the ap­proach that comes clos­est to how we tend to think about val­ues is re­flec­tive equil­ibrium, al­though other meta-eth­i­cal views are not un­rep­re­sented around here.

  For util­ity com­par­i­sons be­tween peo­ple, I think a lot of think­ing here have been based on or in­spired by game the­ory, e.g., bar­gain­ing games.

  Of course there is a lot of dis­agree­ment and un­cer­tainty be­tween and within in­di­vi­d­u­als on LW, so spe­cific posts may well be based on differ­ent foun­da­tions or are just in­for­mal ex­plo­ra­tions that aren’t based on any the­o­ret­i­cal foun­da­tions.

  In this post, Stu­art seems to be try­ing to con­struct an ex­trap­o­lated/​syn­the­sized (vNM or vNM-like) util­ity func­tion out of a sin­gle hu­man’s in­com­plete and in­con­sis­tent prefer­ences and meta-prefer­ences, which I don’t think has much of a liter­a­ture in eco­nomics?

• Wei_Dai 15 Mar 2019 2:54 UTC

  12 points

  on: Spec­u­la­tions on Duo Standard

  Hi Zvi, may I sug­gest that you tag your Magic the Gather­ing posts with [MtG] or some­thing similar in the ti­tle? Since you blog about both MtG top­ics and other top­ics, I imag­ine a lot of peo­ple on LW clicked on this post won­der­ing what it’s about, and then im­me­di­ately went back out af­ter see­ing that it’s a post about MtG. (I ac­tu­ally had to Google “Duo Stan­dard” to figure that out be­cause the post doesn’t men­tion MtG or Magic in the first few para­graphs.)

  Also, am I cor­rect in as­sum­ing that these MtG posts are just about MtG, and are not meant to illus­trate more gen­eral prin­ci­ples or some­thing like that?

• Wei_Dai 14 Mar 2019 3:45 UTC

  10 points

  AF

  in reply to: Lukas_Gloor's comment on: A the­ory of hu­man values

  Prob­a­bly you were think­ing of some­thing like teach­ing AIs metaphilos­o­phy in or­der to per­haps im­prove the pro­ce­dure? This would be the main al­ter­na­tive I see, and it does feel more ro­bust. I am won­der­ing though whether we’ll know by that point whether we’ve found the right way to do metaphilosophy

  I think there’s some (small) hope that by the time we need it, we can hit upon a solu­tion to metaphilos­o­phy that will just be clearly right to most (philo­soph­i­cally so­phis­ti­cated) peo­ple, like how math and sci­ence were prob­a­bly once method­olog­i­cally quite con­fus­ing but now ev­ery­one mostly agrees on how math and sci­ence should be done. Failing that, we prob­a­bly need some sort of global co­or­di­na­tion to pre­vent com­pet­i­tive pres­sures lead­ing to value lock-in (like the kind that would fol­low from Stu­art’s scheme). In other words, if there wasn’t a race to build AGI, then there wouldn’t be a need to solve AGI safety, and there would be no need for schemes like Stu­art’s that would lock in our val­ues be­fore we solve metaphilos­o­phy.

  it doesn’t feel ob­vi­ous why some­thing like Stu­art’s anti-re­al­ism isn’t already close to there

  Stu­art’s scheme uses each hu­man’s own meta-prefer­ences to de­ter­mine their own (fi­nal) ob­ject-level prefer­ences. I would less con­cerned if this was used on some­one like William MacAskill (with the caveat that cor­rectly ex­tract­ing William MacAskill’s meta-prefer­ences seems equiv­a­lent to learn­ing metaphilos­o­phy from William) but a lot of hu­mans have seem­ingly ter­rible meta-prefer­ences or at least differ­ent meta-prefer­ences which likely lead to differ­ent ob­ject-level prefer­ences (so they can’t all be right, as­sum­ing moral re­al­ism).

  To put it an­other way, my po­si­tion is that if moral re­al­ism or rel­a­tivism (po­si­tions 1-3 in this list) is right, we need “metaphilo­soph­i­cal pa­ter­nal­ism” to pre­vent a “ter­rible out­come”, and that’s not part of Stu­art’s scheme.

• Wei_Dai 13 Mar 2019 20:22 UTC

  10 points

  in reply to: Vladimir_Nesov's comment on: How dan­ger­ous is it to ride a bi­cy­cle with­out a helmet?

  It feels to me like peo­ple in our com­mu­nity aren’t be­ing skep­ti­cal enough or push­ing back enough on the idea of acausal co­or­di­na­tion for hu­mans. I’m kind of con­fused about this be­cause it seems like a weirder idea and has less good ar­gu­ments for it than for ex­am­ple the im­por­tance of AI risk which does get sub­stan­tial skep­ti­cism and push back.

  In an old post I ar­gued that for acausal co­or­di­na­tion rea­sons it seems as if you should fur­ther mul­ti­ply this value by the num­ber of peo­ple in the refer­ence class of those mak­ing the de­ci­sion the same way (dis­counted by how lit­tle you care about strangers vs. your­self).

  But if “the same way” in­cludes not only the same kind of ex­plicit cost/​benefit anal­y­sis but also “fur­ther mul­ti­ply this value by the num­ber of peo­ple in the refer­ence class of those mak­ing the de­ci­sion the same way”, the num­ber of peo­ple in this refer­ence class must be tiny be­cause no­body is do­ing this for de­cid­ing whether to wear bike helmets.

  Sup­pose two peo­ple did “fur­ther mul­ti­ply this value by the num­ber of peo­ple in the refer­ence class of those mak­ing the de­ci­sion the same way”, but their de­ci­sion mak­ing pro­cesses are slightly differ­ent, e.g., they use differ­ent heuris­tics to do things like find­ing sources for the num­bers that go into the cost/​benefit anal­y­sis, I don’t know how to figure out whether they are still in the same refer­ence class, or how to gen­er­al­ize be­yond “same refer­ence class” when the agents are hu­mans as op­posed to AIs (and even with the lat­ter we don’t have a com­plete math­e­mat­i­cal the­ory).

  peo­ple talk about this ar­gu­ment mostly in the con­text of voting

  I’m skep­ti­cal about this too. I’m not ac­tu­ally aware of a good ar­gu­ment for acausal co­or­di­na­tion in the con­text of vot­ing. A search on LW yields only this short com­ment from Eliezer.

• Wei_Dai 13 Mar 2019 19:48 UTC

  7 points

  AF

  on: A the­ory of hu­man values

  This seems to as­sume a fairly spe­cific (i.e., anti-re­al­ist) metaethics. I’m quite un­cer­tain about metaethics and I’m wor­ried that if moral re­al­ism is true (and say for ex­am­ple that to­tal he­do­nic util­i­tar­i­anism is the true moral the­ory), and what you pro­pose here causes the true moral the­ory to be able to con­trol only a small frac­tion of the re­sources of our uni­verse, that would con­sti­tute a ter­rible out­come. Given my state of knowl­edge, I’d pre­fer not to make any plans that im­ply com­mit­ment to a spe­cific metaeth­i­cal the­ory, like you seem to be do­ing here.

  What’s your re­sponse to peo­ple with other metaethics or who are very un­cer­tain about metaethics?

  How­ever, for ac­tual hu­mans, the first sce­nario seems to loom much larger.

  I don’t think this is true for me, or maybe I’m mi­s­un­der­stand­ing what you mean by the two sce­nar­ios.

• Wei_Dai 13 Mar 2019 7:39 UTC

  3 points

  AF

  in reply to: michaelcohen's comment on: Asymp­tot­i­cally Benign AGI

  If the as­sump­tion is true, we could de­mand that A use their words, and counter us be­ing mind-hacked by pok­ing holes in what B is say­ing rather than de­mand­ing we stop listen­ing to B. And if A is able to con­vince us that B was mind-hack­ing, even af­ter some more back and forth, B will be pun­ished for that.

  Oh, I see, I didn’t un­der­stand “it is harder to mys­tify a judge than it is to pierce through some­one else mys­tify­ing a judge” cor­rectly. So this as­sump­tion ba­si­cally rules out a large class of pos­si­ble vuln­er­a­bil­ities in the judge, right? For ex­am­ple, if the judge had the equiv­a­lent of a buffer overflow bug in a net­work stack, the scheme would fail. In that case, A would not be able to “pierce through” B’s at­tack and stop it with its words if the judge keeps listen­ing to B (and B was ac­tu­ally at­tack­ing).

  I don’t think the “AI safety via de­bate” pa­per ac­tu­ally makes ar­gu­ments for this as­sump­tion (at least I couldn’t find where it does). Do you have rea­sons to think it’s true, or ideas for how to ver­ify that it’s true, short of putting a hu­man in a BoMAI?