Donald Hobson

• Donald Hobson 28 Feb 2020 0:03 UTC

  1 point

  in reply to: Nornagest's comment on: Harry Pot­ter and the Meth­ods of Ra­tion­al­ity dis­cus­sion thread, March 2015, chap­ter 116

  The ni­ave pro­to­col, any time you get a note, you come up with a ran­dom pass­word. That pass­word has to be on the bot­tom of the note. If the pass­word has even a few bits of en­tropy, rel­a­tive to out­siders, this will work. (Me­mory charms or time turn­ers can get around this, but its still a good pre­cau­tion)

• Donald Hobson 26 Feb 2020 17:26 UTC

  4 points

  on: New ar­ti­cle from Oren Etzioni

  Prepar­ing any­way even if it’s very low prob­a­bil­ity be­cause of ex­treme con­se­quences is Pas­cal’s Wager

  ASI is prob­a­bly com­ing sooner or later. Some­one has to pre­pare at some point, the ques­tion is when.

  I con­sider AI de­vel­op­ment to be a field that I have lit­tle definite info about. Its hard to as­sign less than 1% prob to state­ments about ASI. (Ex­cept­ing the highly con­joined ones.) I don’t con­sider things like dinosaur kil­ling as­ter­oids with 1 in 100 mil­lion probs to be pas­cal mug­gings.

  If and when a ca­nary “col­lapses” we will have am­ple time to de­sign off switches and iden­tify red lines we don’t want AI to cross

  We have a tricky task, and we don’t know how long it will take. Hav­ing hit one of these switches doesn’t help us to do the task much. A stu­dent is given an as­sign­ment in Au­gust, the due date is March next year. They de­cide to put it off un­til it snows. Snow­fall is an in­di­ca­tor that the due date is com­ing soon, but not a good one. But ei­ther way, it doesn’t help you do the as­sign­ment.

  What is a “fully self driv­ing” car, we have had al­gorithms that kind of usu­ally work for years, and a sub­stan­tial part of mod­ern progress in the field looks like gath­er­ing more data, and de­vel­op­ing driv­ing spe­cific tricks. Sup­pose that you needed 100 mil­lion hours of driv­ing data to train cur­rent AI sys­tems. A com­pany pays drivers to put a lit­tle record­ing box in their car. It will take 5 years to gather enough data, and af­ter that we will have self driv­ing cars. What are you go­ing to do in 5 years time that you can’t do now. In re­al­ity, we aren’t sure if you need 50, 100 or 500 mil­lion hours of driv­ing data with cur­rent al­gorithms, and aren’t sure how many peo­ple will want the boxes in­stalled. (Th­ese boxes are usu­ally built into sat­navs or lane con­trol sys­tems in mod­ern cars)

  Limited ver­sions of the Tur­ing test (like Wino­grad Schemas)

  What per­centage do you want, and what will you do when gpt-5 hits it?

  We are decades away from the ver­sa­tile abil­ities of a 5 year old

  A “re­sult” got by fo­cussing on the things that 5 year olds are good at,

  Some­times you have a prob­lem like look­ing at an image of some ev­ery­day scene and say­ing whats hap­pen­ing in it that 5 year olds are (or at least were a few years ago) much bet­ter at. Look­ing at a load of stock data and us­ing lin­ear re­gres­sion to find cor­re­la­tions be­tween prices, noth­ing like that ex­isted in the en­vi­ron­ment of evolu­tion­ary adapt­ed­ness, hu­man brains aren’t built to do that.

  If and when a ca­nary “col­lapses” we will have am­ple time to de­sign off switches and iden­tify red lines we don’t want AI to cross

  Even if that was true, how would you know that? Tech­nolog­i­cal progress is hard to pre­dict. De­sign­ing off switches is ut­terly triv­ial if the sys­tem isn’t try­ing to avoid the off switch be­ing pressed, and ac­tu­ally quite hard if the AI is smart enough to know about the off switch and re­move it.

• Donald Hobson 26 Feb 2020 16:46 UTC

  3 points

  on: Thoughts about Dr Stone and Mythology

  I agree that get­ting chi­nese whispered into non­sense is a real failure mode, pop sci quan­tum me­chan­ics shows this well enough. I think it un­likely that there was that much of a real, true and im­por­tant point be­hind most myths. Re­peat­edly rere­cord a sound on analogue me­dia enough and you get static. Re­peat­edly retell a story and you get mythol­ogy. We can’t tell much about what the sig­nal started as, but I doubt it was all brilli­ant ra­tio­nal­ity, be­cause most of our re­ally old doc­u­ments aren’t that ra­tio­nal.

• Donald Hobson 26 Feb 2020 13:13 UTC

  LW: 1 AF: 1

  AF

  on: How Low Should Fruit Hang Be­fore We Pick It?

  Sup­pose the AI finds a plan with 10^50 im­pact and 10^1000 util­ity. I don’t want that plan to be run. Its prob­a­bly a plan that in­volves tak­ing over the uni­verse and then do­ing some­thing re­ally high util­ity. I think a con­straint is bet­ter than a scal­ing fac­tor.

• Donald Hobson 23 Feb 2020 20:24 UTC

  LW: 2 AF: 1

  AF

  on: Will AI un­dergo dis­con­tin­u­ous progress?

  I am not con­vinced that the dis­tinc­tion be­tween con­tin­u­ous and dis­con­tin­u­ous ap­proaches is a fea­ture of the ter­ri­tory. Zoom in in suffi­cient de­tail, and you see a con­tin­u­ous wave­func­tion of elec­tron in­ter­act­ing with the con­tin­u­ous wave­func­tion of a sili­con atom. Zoom out to evolu­tion­ary timescales, and the jump from ho­minids with pointy sticks to ASI is al­most in­stant. The math­e­mat­i­cal defi­ni­tion of con­ti­nu­ity re­lies on your func­tion be­ing math­e­mat­i­cally for­mal. Is dis­tance from earth to the moon an even num­ber of plank-lengths? Well there are a huge num­ber of slightly differ­ent mea­sure­ments you could make, de­pend­ing on when you mea­sure and ex­actly what points you mea­sure be­tween, and how you deal with re­li­tivis­tic length con­trac­tion, the an­swer will be differ­ent. In a microsec­ond in which the code that is a foom­ing AI is do­ing garbage col­lec­tion, is AI progress hap­pen­ing? You have iden­ti­fied an em­piri­cal vari­able called AI progress, but whether or not it is con­tin­u­ous de­pends on ex­actly how you fill in the de­tails.

  Imag­ine su­per­in­tel­li­gence has hap­pened and we are dis­cussing the de­tails af­ter­words. We were in a world ba­si­cally like this one, and then some­one ran some code, which gained to­tal cos­mic power within a mil­lisec­ond. Some­one tries to ar­gue that this was a con­tin­u­ous im­prove­ment, just a very fast one. What ev­i­dence would con­vince you one way or the other on this?

• Donald Hobson 23 Feb 2020 16:21 UTC

  3 points

  on: The­ory and Data as Constraints

  I would bet that all the data needed in prin­ci­ple to, say, find a cure for Alzheimers is already available on­line—if only we knew how to effec­tively lev­er­age it.

  I agree. If “effec­tively lev­er­age it” means a su­per­in­tel­li­gence with un­limited com­pute, then this is a some­what weak state­ment. I would ex­pect that a su­per­in­tel­li­gence given the hu­man genome would figure out how to cure all dis­eases. I would ex­pect it to be able to figure out a lot from any book on biol­ogy. I would ex­pect it to be able to look at a few hol­li­day pho­tos, figure out the fun­da­men­tal equa­tions of re­al­ity, and that evolu­tion hap­pened on a planet ,that it was cre­ated by evolved in­tel­li­gences with tech ect. From this, it could de­sign nanobots pro­grammed to find hu­mans and cure them, even if it had no idea what hu­mans look like, it just pro­grams the nanobots to find the most in­tel­li­gent life forms around.

• Donald Hobson 23 Feb 2020 14:37 UTC

  LW: 3 AF: 3

  AF

  on: Will AI un­dergo dis­con­tin­u­ous progress?

  Sup­pose that differ­ent tasks take differ­ent lev­els of AI to do bet­ter than hu­mans.

  Firstly AI can do ar­ith­matic, then play chess, then drive cars ect. Lets also as­sume that AI is much faster than hu­mans. So imag­ine that AI re­search abil­ity was ris­ing from al­most noth­ing to su­per­hu­man over the course of a year. A few months in and its in­vent­ing stuff like lin­ear re­gres­sion, im­pres­sive, but not as good as cur­rent hu­man work on AI. There are a few months where the AI is worse than a se­ri­ous team of top re­searchers, but bet­ter than an in­tern. So if you have a nieche use for AI, that can be au­to­mat­i­cally au­to­mated. The AI re­search AI de­signs a wid­get build­ing AI. The hu­mans could have made a wid­get build­ing AI them­selves, but so few wid­gets are pro­duced, that it wasn’t worth it.

  Then the AI be­comes as good as a top hu­man re­search team and FOOM. How crazy the world gets be­fore foom de­pends on how much other stuff is au­to­mated first. Is it eas­ier to make an AI teacher, or an AI AI re­searcher? Also re­mem­ber that bearo­cratic de­lays are a thing, there is a differ­ence be­tween hav­ing an AI that does med­i­cal di­ag­no­sis in a lab, and it be­ing used in ev­ery hos­pi­tal.

• Donald Hobson 22 Feb 2020 23:44 UTC

  1 point

  on: Does ra­tio­nal­ism af­fect your dreams?

  I had a dream where I was fly­ing by in­cre­ment­ing my own x and y co­or­diates. Some­what re­lated to simu­lated wor­lds, but also to straight pro­gram­ming.

• Donald Hobson 22 Feb 2020 22:27 UTC

  1 point

  on: What would you do with an Evil AI?

  If I am con­fi­dant that I have the origi­nal source code, as writ­ten by hu­mans, I read that. I am look­ing for deep ab­stract prin­ci­ples. I am look­ing only for ab­stract ideas that are gen­eral to the field of AI.

  If I can en­crypt the code in a way that only a fu­ture su­per­in­tel­li­gence can crack, and I feel hope­ful about FAI, I do that. Other­wise, se­cure erase, pos­si­bly in­volv­ing any­thing that can slag the hard drives that is ly­ing around.

• Donald Hobson 11 Feb 2020 16:05 UTC

  4 points

  on: Si­mu­la­tion of tech­nolog­i­cal progress (work in progress)

  You might be pro­duc­ing some use­ful info, but mostly about whether an ar­bi­trary sys­tem ex­ibits un­limited ex­po­nen­tial growth. If you got 1000 differ­ent pro­gram­mers to each throw to­gether some model of tech progress, some based on com­plet­ing tasks, some based on ex­tract­ing re­sources, some based on ran­dom differ­en­tial equa­tions ect, and see what pro­por­tion of them give ex­po­nen­tial growth and then stag­na­tion. Ac­tu­ally, there isn’t a scale on your model, so who can say if the run­ning out of tasks, or stag­na­tion are next year or in 100000 years. At best, you will be able to tell how strongly out­side view pri­ors should fa­vor exp growth over growth and then de­cay. (Pure growth is clearly sim­pler, but how much sim­pler?)

In­tel­li­gence with­out causality

• Donald Hobson 7 Feb 2020 19:05 UTC

  1 point

  on: In­tel­li­gence is prob­a­bly not that good

  A lot of your ar­gu­ment seems to be com­par­ing an ar­ti­fact of hu­man tech­nol­ogy with an evolved sys­tem. “is there a way to de­stroy the moon, given only the abil­ity to post 10k char­ac­ters to less­wrong.com?”

  To make the dis­cus­sion clearer, lets pick a par­tic­u­lar evolved sys­tem and tech­nol­ogy, say an aero­plane wing and a in­sect wing. Sup­pose that the aero­plane wing wins on some crite­ria, like speed, the in­sect wing wins on effi­ciency and it all bal­ances out over­all.

  To say there­for that in­tel­li­gence isn’t that great is a mix­ing of lev­els. There are two in­tel­li­gences in the game, hu­mans and evolu­tion. Both have pro­duced a great va­ri­ety of highly op­ti­mized ar­ti­facts. Both are of roughly com­pa­rable power. By com­par­ing two aero­planes, you can also com­pare the skill of the de­sign­ers, but it is mean­ingless to try to com­pare an aero­plane to an aero­plane de­signer. The in­sect is the plane, not the de­signer.

  Some of your com­par­i­sons make even less sense, like abil­ity to sur­vive in ex­treme en­vi­ron­ments. Com­par­ing a fish and an un­tooled hu­man in abil­ity to sur­vive in the ocean is a straight con­test of fish evolu­tion vs hu­man evolu­tion. If the hu­man drowns be­fore they have a chance to think any­thing, the power of the hu­man brain is not shown in the slight­est.

  Also com­par­ing hu­man in­tel­li­gence be­tween hu­mans is like com­par­ing the run­ning speed of chee­tahs, all your re­sults will be similar. So one hu­man beat­ing an­other tells you lit­tle about in­tel­li­gence.

  So what would a real com­par­i­son of in­tel­li­gence with some­thing else look like? I think the ques­tion “Is in­tel­li­gence good?” is not that mean­ingful.

  What we can do is ask “is there a way to X given only Y” For in­stance “is there a way to make a fire, given only the abil­ity to con­tract mu­cles of a hu­man body in a for­est?” or “is there a way to de­stroy the moon, given only the abil­ity to post 10k char­ic­ters to less­wrong.com?” Th­ese are to­tally for­mal­iz­able ques­tions and could in prin­ci­ple be an­swered by simu­lat­ing an ex­po­nen­tial num­ber of uni­verses.

  We can also ask ques­tions about which al­gorithms will ac­tu­ally find a way to achieve a goal. We know that there ex­ists a pat­tern of elec­tri­cal in­puts that win the game pong, but want to know if some gra­di­ent de­scent based al­gorithm will find one.

  We can then say there are a wide va­ri­ety of tasks and goals that hu­mans can fulfill given our prim­i­tive ac­tion of mus­cle con­trac­tion. Given that chimps have a similar mus­cu­la­ture, but less in­tel­li­gence and can’t do most of these tasks, and many of the routes to fulfill­ment of the goals go through lay­ers of in­di­rec­tion, then it seems that an in­tel­li­gence com­pa­rable to hu­mans with some other out­put chan­nel would be similarly good at achiev­ing goals.

• Donald Hobson 7 Feb 2020 15:37 UTC

  LW: 1 AF: 1

  AF

  on: Plau­si­bly, al­most ev­ery pow­er­ful al­gorithm would be manipulative

  How dan­ger­ous would you con­sider a per­son with ba­sic pro­gram­ming skills and a hy­per­com­puter? I mean I could make some­thing very dan­ger­ous, given hy­per­com­pute. I’m not sure if I could make much that was safe and still use­ful. How com­mon would it be to ac­ci­den­tally evolve a race of aliens in the garbage col­lec­tion?

  At the mo­ment, my best guess at what pow­er­ful al­gorithms look like is some­thing that lets you max­i­mize func­tions with­out search­ing through all the in­puts. Gra­di­ent de­scent can of­ten find a high point with­out that much com­pute, so is more pow­er­ful than ran­dom search. If your pow­er­ful al­gorithm is more like re­ally good com­pu­ta­tion­ally bounded op­ti­miza­tion, I sus­pect it will be about as ma­nipu­la­tive as brute forc­ing the search space. (I see no strong rea­son for strate­gies la­beled ma­nipu­la­tive to be that much eas­ier or harder to find than those that aren’t.)

• Donald Hobson 7 Feb 2020 14:35 UTC

  1 point

  on: Don­ald Hob­son’s Shortform

  Sup­pose an early AI is try­ing to un­der­stand its pro­gram­mers and makes mil­lions of hy­poth­e­sis that are them­selves peo­ple. Later it be­comes a friendly su­per­in­tel­li­gence that figures out how to think with­out mind­crime. Sup­pose all those im­perfect vir­tual pro­gram­mers have been saved to disk by the early AI, the su­per­in­tel­li­gence can look through it. We end up with a post sin­gu­lar­ity utopia that con­tains mil­lions of cit­i­zens al­most but not quite like the pro­gram­mers. We don’t need to solve the non­per­son pred­i­cate our­selves to get a good out­come, just avoid minds we would re­gret cre­at­ing.

• Donald Hobson 4 Feb 2020 17:45 UTC

  9 points

  in reply to: RedMan's comment on: Ab­sent co­or­di­na­tion, fu­ture tech­nol­ogy will cause hu­man extinction

  Quote from wikipe­dia on fukushima

  Deaths 1 can­cer death at­tributed to ra­di­a­tion ex­po­sure by gov­ern­ment panel.[4][5]

  Non-fatal in­juries 16 with phys­i­cal in­juries due to hy­dro­gen ex­plo­sions,[6]
  2 work­ers taken to hos­pi­tal with pos­si­ble ra­di­a­tion burns[7]

  I think this puts the in­ci­dent squarely in the class of minor ac­ci­dents that the me­dia had a panic about. Un­less you think it had a 50 % chance of wiping out japan and we were just lucky, it is ir­rele­vant to the dis­cus­sion of X-risk.

  With CO2, it de­pends what you mean by buis­ness as usual. We don’t have 500 years of fos­sil fuels left, and we are already switch­ing to re­new­ables. I don’t think that the earth will be­come un­in­hab­it­able to tech­nolog­i­cally ad­vanced hu­man life. A In a sce­nario where hu­mans are us­ing air con­di­tion­ers and de­sal­i­na­tors to sur­vive the 80C Nor­we­gian deserts, the world is still “hab­it­able”. (I don’t think it will get that bad, but I think hu­mans would sur­vive if it did. )

  If the time it takes for a black ball to kill us is more than a few gen­er­a­tions it’s re­ally hard to plan around fix­ing it.

  No those are the ones that are re­ally easy to plan around, you have plenty of time to fix them. Its the ones that kill you in­stantly that are hard to plan around.

• Donald Hobson 4 Feb 2020 17:32 UTC

  1 point

  on: Ab­sent co­or­di­na­tion, fu­ture tech­nol­ogy will cause hu­man extinction

  Con­sider these 5 states

  1)FAI

  2)UFAI

  3) Tech progress fails. No one is do­ing tech re­search.

  4) We co­or­di­nate to avoid UFAI, and don’t know how to make FAI.

  5) No co­or­di­na­tion to avoid UFAI, no one has made one yet. (State we are cur­rently in)

  In the first 3 sce­nar­ios, hu­man­ity won’t be wiped out by some other tech. If we can co­or­di­nate around AI, I would sus­pect that we would man­age to co­or­di­nate around other black balls. (AI tech seems un­usu­ally hard to co­or­di­nate around, as we don’t know where the dan­ger­ous re­gions are, tech near the dan­ger­ous re­gions is likely to be very prof­itable, it is an ob­ject en­tirely of in­for­ma­tion, thus eas­ily copied and hid­den. ) In state 5, it is pos­si­ble for some other black ball to wipe out hu­man­ity.

  So con­di­tional on some black ball tech other than UFAI wiping out hu­man­ity, the most likely sce­nario is that it came sooner than UFAI could. I would be sur­prised if hu­man­ity stayed in state 5 for the next 100 years. (I would be most wor­ried about grey goo here)

  The other thread of pos­si­bil­ity is that hu­man­ity co­or­di­nates around stop­ping UFAI be­ing de­vel­oped, and then gets wiped out by some­thing else. This re­quires an im­pres­sive amount of co­or­di­na­tion. It also re­quires that FAI isn’t de­vel­oped (or is stopped by the co­or­di­na­tion to avoid UFAI) Given this hap­pens, I would ex­pect that hu­mans had got bet­ter at co­or­di­nat­ing, that peo­ple who cared about X-risk were in po­si­tions of power, and that stan­dards and pres­i­dents had been set. Any­thing that wipes out a hu­man­ity that well co­or­di­nated would have to be re­ally hard to co­or­di­nate around.

• Donald Hobson 2 Feb 2020 22:57 UTC

  1 point

  on: The Case for Ar­tifi­cial Ex­pert In­tel­li­gence (AXI): What lies be­tween nar­row and gen­eral AI?

  I think that there is a scale from the to­tally spe­cific al­gorithms to to­tally gen­eral ones.

• Donald Hobson 31 Jan 2020 15:11 UTC

  LW: 3 AF: 2

  AF

  on: Would I think for ten thou­sand years?

  Peo­ple will have to do a lot of maths and philos­o­phy to get an AI sys­tem that works at all.

  Sup­pose you have a lead of 1 week over any ufai pro­jects, and you have your AI sys­tem to the point where it can pre­dict what you would do in a box. (Ac­tu­ally, we can say the AI has de­vel­oped mind up­load­ing tech + lotsa com­pute) The hu­man team needs say 5 years of think­ing to come up with bet­ter metaethics, defense against value drift or what­ever. You want to simu­late the hu­mans in some rea­son­ably hu­man friendly en­vi­ron­ment for a few years to work this thing out. You pick a nice town, and ask the AI to cre­ate a vir­tual copy of the town. (More speci­fi­cally, you ran­domly sam­ple from the AI’s prob­a­bil­ity dis­tri­bu­tion, af­ter con­di­tion­ing on enough data that the town will be town­like.) The vir­tual town is cre­ated with no peo­ple ex­cept the re­search team in it. All the ser­vices are set to work with­out any main­te­nance. (Water in vir­tual pipes, food in vir­tual shops, vir­tual in­ter­net works.). The team of peo­ple up­loaded into this town is at least 30, ideally a few hun­dred, in­clud­ing plenty of friends and fam­ily.

  This “vir­tual me in a box” seems likely to be use­ful and un­likely to be dan­ger­ous. I agree that any vir­tual box trick that in­volves peo­ple think­ing for a long time com­pared to cur­rent lifes­pans is dan­ger­ous. A sin­gle per­son trapped in low res poly­gon land would likely go crazy from the sen­sory de­pri­va­tion.

  You need an en­vi­ron­ment with a re­al­is­tic level of so­cial­iz­ing and leisure ac­tivi­ties to sup­port psy­colog­i­cally healthy hu­mans. Any well done “vir­tual me in a box” is go­ing to look more like a vir­tual AI safety camp or re­search de­part­ment than 1 per­son in a blank white room con­tain­ing only a key­board.

  Un­for­tu­nately, all those de­tails would be hard to man­u­ally hard code in. You seem to need an AI that can be trusted to fol­low rea­son­ably clear and spe­cific goals with­out ad­ver­sar­ial op­ti­miza­tion. You want a vir­tual park, man­u­ally cre­at­ing it would be a lot of hard work, see cur­rent video games. You need an AI that can fill in thou­sands of lit­tle de­tails in a manor not op­ti­mized to mess with hu­mans. This is not an es­pe­cially high bar.

• Donald Hobson 29 Jan 2020 9:35 UTC

  3 points

  on: Al­gorithms vs Compute

  The al­gorithms that are used nowa­days are ba­si­cally the same as the al­gorithms that were known then, just with a bunch of tricks like dropout.

  Sup­pose that you have 100 ideas that seem like they might work. You test them, and one of them does work. You then find a math­e­mat­i­cal rea­son why it works. Is this in­sight of com­pute?

  Even if most of the im­prove­ment is in com­pute, there could be much bet­ter al­gorithms that we just aren’t find­ing. I would be un­sur­prised if there ex­ists an al­gorithm that would be re­ally scary on vac­uum tubes.

• Donald Hobson 28 Jan 2020 14:35 UTC

  15 points

  on: The Episte­mol­ogy of AI risk

  Differ­ent minds use differ­ent crite­ria to eval­u­ate an ar­gu­ment. Sup­pose that half the pop­u­la­tion were perfect ra­tio­nal­ists, whose crite­ria for judg­ing an ar­gu­ment de­pended only on Oc­cam’s ra­zor and Bayesian up­dates. The other half are hard-coded bibli­cal liter­al­ists, who only be­lieve state­ments based on re­li­gious au­thor­ity. So half the pop­u­la­tion will con­sider “Here are the short equa­tions, show­ing that this con­cept has low Komel­gorov com­plex­ity” to be a valid ar­gu­ment, the other half con­sider, “Pope Cle­ment said …” to be a strong ar­gu­ment.

  Sup­pose that any po­si­tion that has strong re­li­gious and strong ra­tio­nal­ist ar­gu­ments for it is so ob­vi­ous that no one is doubt­ing or dis­cussing it. Then most propo­si­tions be­lieved by half the pop­u­la­tion have strong ra­tio­nal­ist sup­port, or strong re­li­gious sup­port, but not both. If you are a ra­tio­nal­ist and see one fairly good ra­tio­nal­ist ar­gu­ment for X, you search for more info about X. Any re­li­gious ar­gu­ments get dis­missed as non­sense.

  The end re­sult is that the ra­tio­nal­ists are hav­ing a se­ri­ous dis­cus­sion about AI risk among them­selves. The re­li­gous dis­miss AI as lu­dicrous based on some bible verse.

  The re­li­gious peo­ple are hav­ing a se­ri­ous dis­cus­sion about the sec­ond com­ing of Christ and judge­ment day, which the ra­tio­nal­ists dis­miss as lu­dicrous.

  The end re­sult is a so­ciety where most of the peo­ple who have read much about AI risk think its a thing, and most of the peo­ple who have read much about judge­ment day think its a thing.

  If you took some per­son from one side and forced them to read all the ar­gu­ments on the other, they still wouldn’t be­lieve. Each side has the good ar­gu­ments un­der their crite­ria of what a good ar­gu­ment is.

  The ra­tio­nal­ists say that the re­li­gious have poor epistemic luck, there is noth­ing we can do to help them now, when su­per-in­tel­li­gence comes it can rewire their brains. The re­li­gious say that the ra­tio­nal­ists are cursed by the devil, when judge­ment day comes, they will be con­verted by the glory of god.

  The ra­tio­nal­ists are de­sign­ing a su­per-in­tel­li­gence, the re­li­gious are pray­ing for judge­ment day.

  Bad ideas and good ones can have similar so­cial dy­nam­ics be­cause most of the so­cial dy­nam­ics around an idea de­pends on hu­man na­ture.