SDM

• SDM 12 Feb 2020 17:48 UTC

  2 points

  in reply to: Daniel Kokotajlo's comment on: Si­mu­la­tion of tech­nolog­i­cal progress (work in progress)

  Eye­bal­ling the graphs you pro­duced, it looks like the sin­gu­lar­i­ties you keep get­ting are hy­per­bolic growth, which we already have in real life (com­pare log(world GDP) to your graph of log(pro­jects com­pleted) - their shapes are al­most iden­ti­cal).

  So far as I can tell, what you’ve shown is that you al­most always get a big speedup of hy­per­bolic growth as AI ad­vances but with­out dis­con­ti­nu­ities, which is what the ‘con­tin­u­ous take­off’ peo­ple like Chris­ti­ano already say they are ex­pect­ing.

  AI is just an­other, faster step in the hy­per­bolic growth we are cur­rently ex­pe­rienc­ing, which cor­re­sponds to a fur­ther in­crease in rate but not a dis­con­ti­nu­ity (or even a dis­con­ti­nu­ity in rate).

  So per­haps this is ev­i­dence of con­tin­u­ous take­off still be­ing quite fast.

• SDM 3 Feb 2020 18:14 UTC

  1 point

  on: Cri­tiquing “What failure looks like”

  First off, if we have spe­cific ev­i­dence (an an­swer to Ob­jec­tion 2) then the his­tor­i­cal anal­ogy in Ob­jec­tion 1 looks a lot weaker, as any real ev­i­dence of WFLL1 aris­ing now would sug­gest that the his­tor­i­cal cases of other al­gorithms that gave patholog­i­cal re­sults just aren’t rep­re­sen­ta­tive. I think they aren’t rep­re­sen­ta­tive.

  (I think the dis­con­ti­nu­ity-based ar­gu­ments largely do make the “this time is differ­ent” case, roughly be­cause gen­eral in­tel­li­gence seems clearly game-chang­ing. WFLL2 seems some­where in be­tween these, and I’m un­sure where my be­liefs fall on that.)

  The key differ­ence ‘this time’ (be­fore we get any­where near WFLL2 or AGI), as I see it, is that those early al­gorithms give recom­men­da­tions to peo­ple that they could im­ple­ment or avoid, so the ‘ex­plo­ra­tory phase’ where we poked around to find out what they were ca­pa­ble of was pretty much risk-free, while WFLL1 im­plies that the sys­tems have some de­gree of au­ton­omy and ac­tu­ally have a chance to do un­ex­pected things with­out hu­mans re­al­iz­ing straight away. Danzig’s lin­ear op­ti­miza­tion lead­ing to a catas­tro­phe would have re­quired more care­less­ness and stu­pidity than (cur­rent or very near-fu­ture) deep RL, be­cause deep RL’s mis­takes are sub­tler and be­cause it has to be loosed on some en­vi­ron­ment to achieve re­sults and give us use­ful in­for­ma­tion on its be­havi­our. As for ev­i­dence, Stu­art Rus­sell thinks that we are already see­ing WFLL1 in so­cial me­dia ad al­gorithms:

  “Con­sider the so-called ‘filter bub­ble’ of so­cial me­dia. The re­in­force­ment learn­ing al­gorithm is try­ing to max­i­mize click throughs. From the view of the hu­man, the pur­pose of the ma­chine is to max­i­mize click­throughs. But from the view of the ma­chine, it is chang­ing the state of the world to max­i­mize clicks. It is chang­ing you to make you more pre­dictable. A rav­ing fas­cist or com­mu­nist is more pre­dictable and will lap up rav­ing con­tent. The ma­chines can change our mind about our ob­jec­tive func­tion so we are eas­ier to satisfy. Ad­ver­tisers have done this for decades.” [I ar­gued with him about this feed­back loop, and Yann Le Cun says this changed at Face­book a while ago]
  
  “The re­in­force­ment learn­ing al­gorithm in so­cial me­dia has de­stroyed the EU, NATO and democ­racy. And that’s just 50 lines of code.”

  I won­der if this hy­poth­e­sis was in Paul’s mind when he wrote the es­say. If Rus­sell is right about any of this that sug­gests that one of the first times we gave deep RL any abil­ity to in­fluence the world it suc­cumbed to a failure sce­nario al­most im­me­di­ately. That’s not a good track record.

• SDM 6 Jan 2020 18:35 UTC

  LW: 4 AF: 3

  AF

  on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  The biggest dis­agree­ment be­tween me and more pes­simistic re­searchers is that I think grad­ual take­off is much more likely than dis­con­tin­u­ous take­off (and in fact, the first, third and fourth para­graphs above are quite weak if there’s a dis­con­tin­u­ous take­off).

  It’s been ar­gued be­fore that Con­tin­u­ous is not the same as Slow by any nor­mal stan­dard, so the strat­egy of ‘deal­ing with things as they come up’, while more vi­able un­der a con­tin­u­ous sce­nario, will prob­a­bly not be suffi­cient.

  It seems to me like you’re as­sum­ing longter­mists are very likely not re­quired at all in a case where progress is con­tin­u­ous. I take con­tin­u­ous to just mean that we’re in a world where there won’t be sud­den jumps in ca­pa­bil­ity, or ap­par­ently use­less sys­tems sud­denly cross­ing some thresh­old and be­com­ing su­per­in­tel­li­gent, not where progress is slow or easy to re­verse. We could still pick a com­pletely wrong ap­proach that makes al­ign­ment much more difficult and set our­selves on a likely path to­wards dis­aster, even if the fol­low­ing is true:

  So far as I can tell, the best one-line sum­mary for why we should ex­pect a con­tin­u­ous and not a fast take­off comes from the in­ter­view Paul Chris­ti­ano gave on the 80k pod­cast: ‘I think if you op­ti­mize AI sys­tems for rea­son­ing, it ap­pears much, much ear­lier.’

  So far as I can tell, Paul’s point is that ab­sent spe­cific rea­sons to think oth­er­wise, the prima fa­cie case that any time we are try­ing hard to op­ti­mize for some crite­ria, we should ex­pect the ‘many small changes that add up to one big effect’ situ­a­tion.

  Then he goes on to ar­gue that the spe­cific ar­gu­ments that AGI is a rare case where this isn’t true (like nu­clear weapons) are ei­ther wrong or aren’t strong enough to make dis­con­tin­u­ous progress plau­si­ble.

  In a world where con­tin­u­ous but mod­er­ately fast take­off is likely, I can eas­ily imag­ine doom sce­nar­ios that would re­quire long term strat­egy or con­cep­tual re­search early on to avoid, even if none of them in­volve FOOM. Imag­ine that the ac­cepted stan­dard for al­igned AI is fol­lows some par­tic­u­lar re­search agenda, like Co­op­er­a­tive In­verse Re­in­force­ment Learn­ing, but it turns out that CIRL starts to be­have patholog­i­cally and tries to wire­head it­self as it gets more and more ca­pa­ble, and that its a fairly deep flaw that we can only patch and not avoid.

  Let’s say that over the course of a cou­ple of years failures of CIRL sys­tems start to ap­pear and com­pound very rapidly un­til they con­sti­tute an Ex­is­ten­tial dis­aster. Maybe peo­ple re­al­ize what’s go­ing on, but by then it would be too late, be­cause the right ap­proach would have been to try some other ap­proach to AI al­ign­ment but the re­search to do that doesn’t ex­ist and can’t be done any­where near fast enough. Like Paul Chris­ti­ano’s what failure looks like

• SDM 6 Dec 2019 15:00 UTC

  2 points

  in reply to: Rob Bensinger's comment on: Mis­con­cep­tions about con­tin­u­ous takeoff

  MIRI thinks that the fact Evolu­tion hasn’t been putting much effort into op­ti­miz­ing for gen­eral in­tel­li­gence is a rea­son to ex­pect dis­con­tin­u­ous progress? Ap­par­ently, Paul’s point is that once we re­al­ize evolu­tion has been putting lit­tle effort into op­ti­miz­ing for gen­eral in­tel­li­gence, we re­al­ize we can’t tell much about the likely course of AGI de­vel­op­ment from evolu­tion­ary his­tory, which leaves us in the de­fault po­si­tion of ig­no­rance. Then, he fur­ther ar­gues that the de­fault case is that progress is con­tin­u­ous.

  So far as I can tell, Paul’s point is that ab­sent spe­cific rea­sons to think oth­er­wise, the prima fa­cie case that any time we are try­ing hard to op­ti­mize for some crite­ria, we should ex­pect the ‘many small changes that add up to one big effect’ situ­a­tion.

  Then he goes on to ar­gue that the spe­cific ar­gu­ments that AGI is a rare case where this isn’t true (like nu­clear weapons) are ei­ther wrong or aren’t strong enough to make dis­con­tin­u­ous progress plau­si­ble.

  From what you just wrote, it seems like the folks at MIRI agree that we should have the prima fa­cie ex­pec­ta­tion of con­tin­u­ous progress, and I’ve read el­se­where that Eliezer thinks the case for re­cur­sive self-im­prove­ment lead­ing to a dis­con­ti­nu­ity is weaker or less cen­tral than it first seemed. So, are MIRI’s main rea­sons for dis­agree­ing with Paul down to other ar­gu­ments (hence the switch from the in­tel­li­gence ex­plo­sion hy­poth­e­sis to the gen­eral idea of rapid ca­pa­bil­ity gain)?

  I would think the most likely place to dis­agree with Paul (if not on the in­tel­li­gence ex­plo­sion hy­poth­e­sis) would be if you ex­pected the right com­bi­na­tion of break­throughs ex­ceeds to a ‘gen­er­al­ity thresh­old’ (or ‘se­cret sauce’ as Paul calls it) that leads to a big jump in ca­pa­bil­ity, but in­ad­e­quate achieve­ment on any one of the break­throughs won’t do.

  Stu­art Rus­sell gives a list of the el­e­ments he thinks will be nec­es­sary for the ‘se­cret sauce’ of gen­eral in­tel­li­gence in Hu­man Com­pat­i­ble: hu­man-like lan­guage com­pre­hen­sion, cu­mu­la­tive learn­ing, dis­cov­er­ing new ac­tion sets and man­ag­ing its own men­tal ac­tivity. (I would add that some­body mak­ing that list 30 years ago would have added per­cep­tion and ob­ject recog­ni­tion, and some­body mak­ing it 60 years ago would have also added effi­cient log­i­cal rea­son­ing from known facts). Let’s go with Rus­sell’s list, so we can be a bit more con­crete. Per­haps this is your dis­agree­ment:

  An AI with (e.g.) good per­cep­tion and ob­ject recog­ni­tion, lan­guage com­pre­hen­sion, cu­mu­la­tive learn­ing ca­pa­bil­ity and abil­ity to dis­cover new ac­tion sets but a merely ad­e­quate or bad abil­ity to man­age its men­tal ac­tivity would be (Paul thinks) rea­son­ably ca­pa­ble com­pared to an AI that is good at all of these things, but (MIRI thinks) it would be much less ca­pa­ble. MIRI has con­cep­tual ar­gu­ments (to do with the na­ture of gen­eral in­tel­li­gence) and em­piri­cal ar­gu­ments (com­par­ing hu­man/​chimp brains and prag­matic ca­pa­bil­ities) in favour of this hy­poth­e­sis, and Paul thinks the con­cep­tual ar­gu­ments are too murky and un­clear to be per­sua­sive and that the em­piri­cal ar­gu­ments don’t show what MIRI thinks they show. Am I on the right track here?

• SDM 5 Dec 2019 20:04 UTC

  6 points

  on: Mis­con­cep­tions about con­tin­u­ous takeoff

  Sum­mary of my re­sponse: chimps are nearly use­less be­cause they aren’t op­ti­mized to be use­ful, not be­cause evolu­tion was try­ing to make some­thing use­ful and wasn’t able to suc­ceed un­til it got to hu­mans.

  So far as I can tell, the best one-line sum­mary for why we should ex­pect a con­tin­u­ous and not a fast take­off comes from the in­ter­view Paul Chris­ti­ano gave on the 80k pod­cast: ‘I think if you op­ti­mize AI sys­tems for rea­son­ing, it ap­pears much, much ear­lier.’ Which is to say, the equiv­a­lent of the ‘chimp’ mile­stone on the road to hu­man-level AI does not have ap­prox­i­mately the eco­nomic util­ity of a chimp, but a de­cent frac­tion of the util­ity of some­thing that is ‘hu­man-level’. This strikes me as an im­por­tant ar­gu­ment that he’s re­peated here, and dis­cussed here last april but other than that it seems to have gone largely un­no­ticed and I’m won­der­ing why.

  I have a the­ory about why this didn’t get dis­cussed ear­lier—there is a much more fa­mous bad ar­gu­ment against AGI be­ing an ex­is­ten­tial risk, the ‘in­tel­li­gence isn’t a su­per­power’ ar­gu­ment that sounds similar. From Chol­let vs Yud­kowsky:

  In­tel­li­gence is not a su­per­power; ex­cep­tional in­tel­li­gence does not, on its own, con­fer you with pro­por­tion­ally ex­cep­tional power over your cir­cum­stances.

  …said the Homo sapi­ens, sur­rounded by countless pow­er­ful ar­ti­facts whose abil­ities, let alone mechanisms, would be ut­terly in­com­pre­hen­si­ble to the or­ganisms of any less in­tel­li­gent Earthly species.

  I worry that in ar­gu­ing against the claim that gen­eral in­tel­li­gence isn’t a mean­ingful con­cept or can’t be used to com­pare differ­ent an­i­mals, some peo­ple have been im­plic­itly as­sum­ing that evolu­tion has been putting a de­cent amount of effort into op­ti­miz­ing for gen­eral in­tel­li­gence. Alter­na­tively, that ar­gu­ing for one sounds like an­other, or that a lot of peo­ple have been ar­gu­ing for both to­gether and haven’t dis­t­in­guished be­tween them.

  Claiming that you can mean­ingfully com­pare evolved minds on the gen­er­al­ity of their in­tel­li­gence needs to be dis­t­in­guished from claiming that evolu­tion has been op­ti­miz­ing for gen­eral in­tel­li­gence rea­son­ably hard be­fore hu­mans came about.

• SDM 3 Dec 2019 14:13 UTC

  1 point

  on: Mis­con­cep­tions about con­tin­u­ous takeoff

  From one of the linked ar­ti­cles, Chris­ti­ano talk­ing about take­off speeds:

  I be­lieve that be­fore we have in­cred­ibly pow­er­ful AI, we will have AI which is merely very pow­er­ful. This won’t be enough to cre­ate 100% GDP growth, but it will be enough to lead to (say) 50% GDP growth. I think the likely gap be­tween these events is years rather than months or decades.

  In par­tic­u­lar, this means that in­cred­ibly pow­er­ful AI will emerge in a world where crazy stuff is already hap­pen­ing (and prob­a­bly ev­ery­one is already freak­ing out). If true, I think it’s an im­por­tant fact about the strate­gic situ­a­tion.

  and in your post:

  Still, the gen­eral strat­egy of “deal­ing with things as they come up” is much more vi­able un­der con­tin­u­ous take­off. There­fore, if a con­tin­u­ous take­off is more likely, we should fo­cus our at­ten­tion on ques­tions which fun­da­men­tally can’t be solved as they come up.

  I agree that the con­tin­u­ous/​slow take­off is more likely than fast take­off, though I have low con­fi­dence in that be­lief (and in most of my be­liefs about AGI timelines) but the world of a con­tin­u­ous/​slow take­off, badly man­aged still seems like an ex­treme dan­ger and a case where it would be too late to deal with many prob­lems in e.g. the same year that they arise. Are you imag­in­ing some­thing like this?

• SDM 2 Dec 2019 16:13 UTC

  1 point

  in reply to: rohinmshah's comment on: The Value Defi­ni­tion Problem

  The sce­nario where ev­ery hu­man gets an in­tent-al­igned AGI, and each AGI learns their own par­tic­u­lar val­ues would be a case where each in­di­vi­d­ual AGI is fol­low­ing some­thing like ‘Distil­led Hu­man Prefer­ences’, or pos­si­bly just ‘Am­bi­tious Learned Value Func­tion’ as its Value Defi­ni­tion, so a fairly Direct sce­nario. How­ever, the over­all out­come would be more to­wards the in­di­rect end—be­cause a mul­ti­po­lar world with lots of pow­er­ful Hu­mans us­ing AGIs and try­ing to com­pro­mise would (you an­ti­ci­pate) end up con­verg­ing on our CEV, or Mo­ral Truth, or some­thing similar. I didn’t con­sider di­rect vs in­di­rect in the con­text of mul­ti­po­lar sce­nar­ios like this (nor did Bostrom, I think) but it seems suffi­cient to just say that the in­di­vi­d­ual AGIs use a fairly di­rect Value Defi­ni­tion while the out­come is in­di­rect.

• SDM 25 Nov 2019 14:04 UTC

  LW: 1 AF: 1

  AF

  in reply to: rohinmshah's comment on: The Value Defi­ni­tion Problem

  I ap­pre­ci­ate the sum­mary, though the way you state the VDP isn’t quite the way I meant it.

  what should our AI sys­tem <@try to do@>(@Clar­ify­ing “AI Align­ment”@), to have the best chance of a pos­i­tive out­come?

  To me, this reads like, ‘we have a par­tic­u­lar AI, what should we try to get it to do’, wheras I meant it as ‘what Value Defi­ni­tion should we be build­ing our AI to pur­sue’. So, that’s why I stated it as ′ what should we aim to get our AI to want/​tar­get/​de­cide/​do’ or, to be con­sis­tent with your way of writ­ing it ‘what should we try to get our AI sys­tem to do to have the best chance of a pos­i­tive out­come’, not ‘what should our AI sys­tem try to do to have the best chance of a pos­i­tive out­come’. Aside from that minor ter­minolog­i­cal differ­ence, that’s a good sum­mary of what I was try­ing to say.

  I fall more on the side of prefer­ring in­di­rect ap­proaches, though by that I mean that we should del­e­gate to fu­ture hu­mans, as op­posed to defin­ing some par­tic­u­lar value-find­ing mechanism into an AI sys­tem that even­tu­ally pro­duces a defi­ni­tion of val­ues.

  I think your opinion is prob­a­bly the ma­jor­ity opinion—my ma­jor point with the ‘scale of di­rect­ness’ was to em­pha­size that our ‘par­tic­u­lar value-find­ing mechanisms’ can have more or fewer de­grees of free­dom, since from a cer­tain per­spec­tive ‘del­e­gate ev­ery­thing to a simu­la­tion of fu­ture hu­mans’ is also a ‘par­tic­u­lar mechanism’ just with a lot more de­grees of free­dom, so even if you strongly favour in­di­rect ap­proaches you will still have to make some de­ci­sions about the na­ture of the del­e­ga­tion.

  The origi­nal rea­son that I wrote this post was to get peo­ple to ex­plic­itly no­tice the point that we will prob­a­bly have to do some philo­soph­i­cal labour our­selves at some point, and then I dis­cov­ered Stu­art Arm­strong had already made a similar ar­gu­ment. I’m cur­rently work­ing on an­other post (also based on the same work at EA Ho­tel) with some more spe­cific ar­gu­ments about why we should con­struct a par­tic­u­lar value-find­ing mechanism that doesn’t fix us to any par­tic­u­lar nor­ma­tive eth­i­cal the­ory, but does fix us to an un­der­stand­ing of what val­ues are—some­thing I call a Co­her­ent Ex­trap­o­lated Frame­work (CEF). But again, Stu­art Arm­strong an­ti­ci­pated a lot (but not all!) of what I was go­ing to say.

• SDM 22 Nov 2019 21:07 UTC

  LW: 3 AF: 2

  AF

  in reply to: G Gordon Worley III's comment on: The Value Defi­ni­tion Problem

  Thanks for point­ing that out to me; I had not come across your work be­fore! I’ve had a look through your post and I agree that we’re say­ing similar things. I would say that my ‘Value Defi­ni­tion Prob­lem’ is an (in­ten­tion­ally) va­guer and broader ques­tion about what our re­search pro­gram should be—as I ar­gued in the ar­ti­cle, this is mostly an ax­iolog­i­cal ques­tion. Your fi­nal state­ment of the Align­ment Prob­lem (in­for­mally) is:

  A must learn the val­ues of H and H must know enough about A to be­lieve A shares H’s val­ues

  while my Value Defi­ni­tion Prob­lem is

  “Given that we are try­ing to solve the In­tent Align­ment prob­lem for our AI, what should we aim to get our AI to want/​tar­get/​de­cide/​do, to have the best chance of a pos­i­tive out­come?”

  I would say the VDP is about what our ‘guid­ing prin­ci­ple’ or ‘tar­get’ should be in or­der to have the best chance of solv­ing the al­ign­ment prob­lem. I used Chris­ti­ano’s ‘in­tent al­ign­ment’ for­mu­la­tion but yours ac­tu­ally fits bet­ter with the VDP, I think.

• SDM 18 Nov 2019 23:17 UTC

  1 point

  on: In­ter­nal­iz­ing Existentialism

  If you stum­bled upon this and didn’t re­al­ize moral­ity wasn’t es­sen­tial, well, um, I’m not go­ing to try to con­vince you of that.

  Per­haps this makes lit­tle differ­ence to the rest of your post, but it’s worth not­ing that the mind-de­pen­dence moral­ity isn’t all-or-noth­ing. A com­mon view is that there are facts about the right and wrong ways to ag­gre­gate prefer­ences or turn non-prefer­ences into prefer­ences with­out there be­ing un­con­di­tional facts about what we should do.

  And if I want to find out if moral­ity re­ally does not ex­ist as an es­sen­tial prop­erty of the uni­verse, it’s worth­while to try to take it out of my lan­guage and see if it comes up miss­ing.

  Mo­ral lan­guage be­ing re­ducible to non-moral lan­guage is a sep­a­rate (though en­tan­gled) ques­tion to whether there are such things as moral facts. A lot of moral an­tire­al­ists would say there is some­thing spe­cial and in­dis­pens­able about moral lan­guage, and that it means some­thing more than lik­ing or dis­lik­ing, e.g. pre­scrip­tivist. Or take this from Three Wor­lds Col­lide (writ­ten by an an­tire­al­ist):

  The Babyeaters strive to do the baby-eat­ing thing to do, the Su­per­hap­pies out­put the Su­per Happy thing to do. None of that tells us any­thing about the right thing to do. They are not ask­ing the same ques­tion we are—no mat­ter what word of their lan­guage the trans­la­tor links to our ‘should’. If you’re con­fused at all about that, my lord, I might be able to clear it up.”

  Even if moral re­al­ism is true and even if moral claims are spe­cial in some way, I still think this part is true at least of aes­thetic claims (which all your ex­am­ples were)

  there’s no sense in which some­thing can “look good” if there is no ob­server to as­sess the qual­ity, so it seems through lan­guage we ca­su­ally mis­take prefer­ences for essences.

The Value Defi­ni­tion Problem

• SDM 26 Oct 2019 20:41 UTC

  4 points

  on: Iron: From myth­i­cal to mundane

  It’s always nice to take in a bit of Joy in the Merely Real. I have some­times found it a use­ful ex­er­cise to con­sider just which things would or would not be shock­ing to peo­ple in the past. For ex­am­ple, any­one from be­fore 1500 would be ut­terly shocked and mys­tified by any page of printed text or any piece of cloth­ing from 1800 or af­ter.