Donald Hobson

• Donald Hobson 16 Jan 2020 23:39 UTC

  1 point

  in reply to: Andrew Jacob Sauer's comment on: A rant against robots

  that the most pow­er­ful al­gorithms, the ones that would likely first be­come su­per­in­tel­li­gent, would be dis­tributed and fault-tol­er­ant, as you say, and there­fore would not be in a box of any kind to be­gin with.

  Al­gorithms don’t have a sin­gle “power” set­ting. It is eas­ier to pro­gram a sin­gle com­puter than to make a dis­tributed fault tol­er­ant sys­tem. Al­gorithms like alpha go are run on a par­tic­u­lar com­puter with an off switch, not spread around. Of course, a smart AI might soon load its code all over the in­ter­net, if it has ac­cess. But it would start in a box.

• Donald Hobson 16 Jan 2020 21:05 UTC

  1 point

  in reply to: Mark_Friedenbach's comment on: What long term good fu­tures are pos­si­ble. (Other than FAI)?

  At the mo­ment, hu­man brains are a co­he­sive whole, that op­ti­mizes for hu­man val­ues. We haven’t yet suc­ceeded in mak­ing the ma­chines share our val­ues, and the hu­man brain is not de­signed for up­grad­ing. The hu­man brain can take knowl­edge from an ex­ter­nal source and use it. Ex­ter­nal tools fol­low the calcu­la­tor model. The hu­man thinks about the big pic­ture world, and re­al­izes that as a men­tal sub­goal of de­sign­ing a bridge, they need to do some ar­ith­metic. In­stead of do­ing the ar­ith­metic them­selves, they pass the task on to the ma­chine. In this cir­cum­stance, the hu­man con­trols the big pic­ture, the hu­man un­der­stands what cog­ni­tive la­bor has been ex­ter­nal­ized and knows that it will help the hu­mans goals.
  
  If we have a sys­tem that a hu­man can say “go and do what­ever is most moral”, that’s FAI. If we have a calcu­la­tor style sys­tem where hu­mans spec­ify the power out­put, weight, ma­te­rial use, ra­di­a­tion out­put ect of a fu­sion plant, and the AI tries to de­sign a fu­sion plant meet­ing those specs, that’s use­ful but not nearly as pow­er­ful as full ASI. Hu­mans with calcu­la­tor style AI could in­vent molec­u­lar nan­otech with­out work­ing out all the de­tails, but they still need an Eric Drexler to spot the pos­si­bil­ity.
  
  In my model you can make a rel­a­tivis­tic rocket, but you can’t take a spar­row, and up­grade it into some­thing that flies through space at 10% light speed and is still a spar­row. If your wor­ried that rel­a­tivis­tic rock­ets might spew dan­ger­ous lev­els of ra­di­a­tion, you can’t make a safe space­craft by tak­ing a spar­row and up­grad­ing it to fly at 10% c. (Well with enough R&D you could make a rocket that su­perfi­cially re­sem­bles a spar­row. De­cid­ing to up­grade a spar­row doesn’t make the safety en­g­ineer­ing any eas­ier.)
  
  Mak­ing some­thing vastly smarter than a hu­man is like mak­ing some­thing far faster than a spar­row. Try­ing to strap re­ally pow­er­ful tur­bo­jets to the spar­row and it crashes and burns. Try to at­tach a hu­man brain to 100X hu­man brain gra­di­ent de­cent and you get an out of con­trol AI sys­tem with non­hu­man goals. Hu­man val­ues are del­i­cate. I agree that it is pos­si­ble to care­fully un­ravel what a hu­man mind is think­ing and what its goals are, and then up­grade it in a way that pre­serves those goals, but this re­quires a deep un­der­stand­ing of how the hu­man mind works. Even granted mind up­load­ing, it would still be eas­ier to cre­ate a new mind largely from first prin­ci­ples. You might look at the hu­man brain to figure out what those prin­ci­ples are, in the same way a plane de­signer looks at birds.
  
  I see a vast space of all pos­si­ble minds, some friendly, most not. Hu­mans are a small dot in this space. We know that hu­mans are usu­ally friendly. We have no guaran­tees about what hap­pens as you move away from hu­mans. In fact we know that one small er­ror can some­times send a hu­man to­tally mad. If we want to make some­thing that we know is safe, we ei­ther need to copy that dot ex­actly, (ie nor­mal biolog­i­cal re­pro­duc­tion, mind up­load­ing) or we need some­thing we can show to be safe for some other rea­son.

  My point with the Ejypt metafor was that the sen­tence

  So­ciety con­tinues as-is, but with posthu­man ca­pa­bil­ities.

  is in­co­her­ent.

  Try “the stock mar­ket con­tinues as is, ex­cept with all life ex­tinct”

  De­scribing the mod­ern world as “like a tribe of mon­keys, ex­cept with post mon­key ca­pa­bil­ities” is ei­ther wrong or so vague to not tell you much.

  At the point when the sys­tem (up­graded hu­man, AI what­ever you want to call it) is 99% sili­con, a stray me­teor hits the biolog­i­cal part. If the re­main­ing 99% stays friendly, some­where in this pro­cess you have solved FAI. I see no rea­son why al­ign­ing a 99% sili­con be­ing is eas­ier that a 100% sili­con be­ing.

• Donald Hobson 14 Jan 2020 9:59 UTC

  1 point

  in reply to: Mark_Friedenbach's comment on: What long term good fu­tures are pos­si­ble. (Other than FAI)?

  as ex­ten­sions of themselves

  Lets as­sume that AI dou­bling time is fairly slow (eg 20 years) and very widely dis­tributed. Huge num­bers of peo­ple throw to­gether AI sys­tems in garages. If the ba­sic prob­lems of FAI haven’t been solved, you are go­ing to get mil­lions of pa­per­clip max­i­miz­ers. (Well, most of them will be op­ti­mis­ing differ­ent things) 100 years later, hu­man­ity, if it still ex­ists at this point are pawns on a game­board that con­tains many su­per­in­tel­li­gences. What hap­pens de­pends on how differ­ent the su­per­in­tel­li­gences goals are, and how hard it is for su­per­in­tel­li­gences to co­op­er­ate. Either they fight, kil­ling hu­man­ity in the cross­fire, or they work to­gether to fill the uni­verse with a mix­ture of all the things they value. The lat­ter looks like 1% pa­per­clips, 1% sta­ples, 1%… .

  Alter­nately, many peo­ple could un­der­stand friend­ly­ness and make var­i­ous FAI’s. The FAI’s work to­gether to make the world a nice place. In this sce­nario the FAI’s aren’t iden­ti­cal, but they are close enough that any one of them would make the world nice. I also agree that a world with FAI’s and pa­per­clip max­imisers could be nice if the FAI’s have a sig­nifi­cant por­tion of to­tal power.

  So­ciety con­tinues as-is, but with posthu­man ca­pa­bil­ities.

  Ex­actly like an­cient Egypt ex­cept that like elec­tro­mag­netic charges at­tract and un­like charges re­pel. I posit that this sen­tence doesn’t make sense. If mat­ter be­haved that way, then atoms couldn’t ex­ist. When we say like X but with change Y, we are con­sid­er­ing the set of all pos­si­ble wor­lds that meet crite­ria Y, and find­ing the one near­est to X. But here there is no world where like charges at­tract that looks any­thing like an­cient Egypt. We can say, like an­cient Egypt but gold is 10x more abun­dant. That ends up as a bronze age so­ciety that makes pyra­mids and makes a lot more gold jew­elry than the real Egyp­ti­ans. I think that “so­ciety as is, but with posthu­man ca­pa­bil­ities” is the first kind of sen­tence. There is no way of mak­ing a change like that and get­ting any­thing re­sem­bling so­ciety as is.

• Donald Hobson 13 Jan 2020 13:25 UTC

  1 point

  in reply to: AprilSR's comment on: What long term good fu­tures are pos­si­ble. (Other than FAI)?

  This seems like one po­ten­tial path, but for it to work, you would need a gov­ern­ment struc­ture that can sur­vive with­out suc­cess­ful pro AI rev­olu­tion­ar­ies for a billion years. You also need law en­force­ment good enough to stop any­one try­ing to make UFAI, with not a sin­gle failure in a billion years. As for a SAI that will help us stop UFAI, can ex­plain 1) how it would help and 2) how it would be eas­ier to build than FAI?

  You also need to say what hap­pens with evolu­tion, given this kind of time, and non an­ces­tral se­lec­tion pres­sures, evolu­tion will pro­duce be­ings not re­motely hu­man in mind or body. Either ar­gue that the evolu­tion is in a morally ok di­rec­tion, and that your gov­ern­ment struc­ture works with these be­ings, or stop evolu­tion by se­lec­tive breed­ing—frozen sam­ples—ge­netic mod­ifi­ca­tion to­wards some baseline. Then you just need to say how all hu­man pop­u­la­tions get this, or how any pop­u­la­tion that doesn’t won’t be build­ing UFAI.

[Question] What long term good fu­tures are pos­si­ble. (Other than FAI)?

• Donald Hobson 11 Jan 2020 14:13 UTC

  4 points

  on: What is Suc­cess in an Im­moral Maze?

  I think that some typ­i­cal mind fal­lacy is hap­pen­ing here.

  Hu­mans evolved to both find the truth, for in­ter­act­ing with the real world, and have be­liefs that are good at win­ning sta­tus games. Nat­u­rally there is a trade­off be­tween these crite­ria. As would be ex­pected, differ­ent peo­ple fall in differ­ent places along a spec­trum of truth fo­cused to sta­tus fo­cused. This is an un­usu­ally truth fo­cused com­mu­nity, and you are prob­a­bly un­usu­ally truth fo­cused. So you see mar­ket­ing as a sta­tus game that you re­ally don’t want to get into. To the peo­ple who are un­usu­ally sta­tus fo­cused, im­moral mazes might seem nice.

• Donald Hobson 9 Jan 2020 15:37 UTC

  1 point

  in reply to: subjective's comment on: Quan­tum Me­chan­ics, Noth­ing to do with Consciousness

  You have a box with 2 wires com­ing out of it. The wires are con­nected to a dis­play in the box. Look­ing at the dis­play is ei­ther 1) a live awake hu­man, or 2) a dead spi­der. Can you tell which is which, with­out open­ing the box? Can you use the fact that the hu­man ob­serv­ing some­thing causes a quan­tum col­lapse, and the spi­der doesn’t to dis­t­in­guish them. Can you build a quan­tum con­scious­ness de­tec­tor? No.

  Sup­pose I write a sim­ple com­puter pro­gram that takes in data from a quan­tum physics ex­per­i­ment, and tells me whether the data as a whole is con­sis­tent with quan­tum physics. I don’t know where the pho­ton went on any par­tic­u­lar run, all any con­scious hu­man sees is a sin­gle yes or no. Would you ex­pect the same re­sults. Yes.

  I take an em­u­lated hu­man mind, and put the whole thing on an ex­tremely pow­er­ful quan­tum com­puter. I simu­late the mind in a su­per­po­si­tion of states. Would you ex­pect the quan­tum com­puter to go into a su­per­po­si­tion cor­rectly, de­spite the per­son be­ing con­scious.

  Sup­pose Joe has opinions on the num­bers 1 to 1000, he ei­ther thinks that they are all good, or all bad, or that some half are good and the other half are bad. If you tell him a num­ber, it will take him 1 minute, to say if its good or bad. It would take a clas­si­cal com­puter 501 min worst case to tell if he has the same opinion of all num­bers. But a quan­tum com­puter can do it in just 2 min. https://​​en.wikipe­dia.org/​​wiki/​​Deutsch%E2%80%93Jozsa_algorithm

  If you dis­agree with any of these, we have a fac­tual dis­agree­ment about an ex­per­i­men­tal re­sult. If you agree, then “con­scious­ness” seems to be an in­visi­ble inaudible dragon to quan­tum me­chan­ics. I would have to ask how you know that its con­scious­ness that causes col­lapse, not DNA.

• Donald Hobson 8 Jan 2020 14:12 UTC

  2 points

  on: Mo­ral­ity vs re­lated concepts

  For ex­am­ple, I could say that, from the per­spec­tive of epistemic ra­tio­nal­ity, I “shouldn’t” be­lieve that buy­ing that bur­rito will cre­ate more util­ity in ex­pec­ta­tion than donat­ing the same money to AMF would. This is be­cause hold­ing that be­lief won’t help me meet the goal of hav­ing ac­cu­rate be­liefs.

  There is a phe­nom­ena in AI safety called “you can’t fetch the coffee if your dead”. A perfect to­tal util­i­tar­ian, or even a money max­imiser would still need to eat, if they want to be able to work next year. If you have a well paid job, or a good chance of get­ting one, don’t starve your­self. Eat some­thing quick cheap and healthy. Quick so you can work more to­day and healthy so you can work years later. In a world where you need to wear a sharp suit to be CEO, the util­i­tar­i­ans should buy sharp suits. Don’t fall for the false econ­omy of per­sonal de­pri­va­tion. This doesn’t en­ti­tle util­i­tar­i­ans to what­ever lux­ury they feel like. If most of your money is go­ing on sharp suits, it isn’t a good job. A sharp suited ex­ec­u­tive should be able to donate far more than a card­board box wear­ing ditch dig­ger.

• Donald Hobson 7 Jan 2020 10:39 UTC

  2 points

  on: The Uni­verse Doesn’t Have to Play Nice

  Heisen­berg’s un­cer­tainty prin­ci­ple: We might imag­ine that if we were clever enough we could find a scheme for gain­ing perfect in­for­ma­tion about a par­ti­cle, but this isn’t the case

  Quan­tum me­chan­ics doesn’t work like that, the in­for­ma­tion you want is not hid­den from you, it doesn’t ex­ist. Gal­ilean rel­a­tivity of mo­tion means that ab­solute rest doesn’t ex­ist, not that ab­solute rest does ex­ist but can’t be known.

• Donald Hobson 4 Jan 2020 0:56 UTC

  LW: 1 AF: 1

  AF

  in reply to: rohinmshah's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  By adding ran­dom noise, I meant adding wig­gles to the edge of the set in thingspace for ex­am­ple adding noise to “bird” might ex­clude “os­trich” and in­clude “duck bill platy­pus”.

  I agree that the high level image net con­cepts are bad in this sense, how­ever are they just bad. If they were just bad and the limit to find­ing good con­cepts was data or some other re­source, then we should ex­pect small chil­dren and men­tally im­paired peo­ple to have similarly bad con­cepts. This would sug­gest a sin­gle gra­di­ent from bet­ter to worse. If how­ever cur­rent neu­ral net­works used con­cepts sub­stan­tially differ­ent from small chil­dren, and not just uniformly worse or uniformly bet­ter, that would show differ­ent sets of con­cepts at the same low level. This would be fairly strong ev­i­dence of mul­ti­ple con­cepts at the smart hu­man level.

  I would also want to point out that a small frac­tion of the con­cepts be­ing differ­ent would be enough to make al­ign­ment much harder. Even if their was a perfect scale, if ¹⁄₃ of the con­cepts are sub­hu­man, ¹⁄₃ hu­man level and ¹⁄₃ su­per­hu­man, it would be hard to un­der­stand the sys­tem. To get any safety, you need to get your sys­tem very close to hu­man con­cepts. And you need to be con­fi­dant that you have hit this tar­get.

• Donald Hobson 4 Jan 2020 0:34 UTC

  2 points

  in reply to: NaiveTortoise's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  This seems to be care­ful de­ploy­ment. The con­cept of de­ploy­ment is go­ing from an AI in the lab, to the same AI in con­trol of a real world sys­tem. Sup­pose your de­sign pro­cess was to fid­dle around in the lab un­til you make some­thing that seems to work. Once you have that, you look at it to un­der­stand why it works. You try to prove the­o­rems about it. You sub­ject it to some ex­ten­sive bat­tery of test­ing and will only put it in a self driv­ing car/​ data cen­ter cool­ing sys­tem once you are con­fi­dent it is safe.

  There are two places this could fail. Your test­ing pro­ce­dures could be in­suffi­cient, or your AI could hack out of the lab be­fore the test­ing starts. I see lit­tle to no defense against the lat­ter.

• Donald Hobson 3 Jan 2020 20:39 UTC

  LW: 4 AF: 3

  AF

  in reply to: johnswentworth's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  Neu­ral nets have around hu­man perfor­mance on Ima­genet.

  If ab­strac­tion was a fea­ture of the ter­ri­tory, I would ex­pect the failure cases to be similar to hu­man failure cases. Look­ing at https://​​github.com/​​hendrycks/​​nat­u­ral-adv-ex­am­ples, This does not seem to be the case very strongly, but then again, some of them con­tain dark shiny stone be­ing clas­sified as a sea lion. The failures aren’t to­tally in­hu­man, the way they are with ad­ver­sar­ial ex­am­ples.

  Hu­mans didn’t look at the world and pick out “tree” as an ab­stract con­cept be­cause of a bunch of hu­man-spe­cific fac­tors.

  I am not say­ing that trees aren’t a cluster in thing space. What I am say­ing is that if there were many cluster in thing space that were as tight and pred­ica­tively use­ful as “Tree”, but were not pos­si­ble for hu­mans to con­cep­tu­al­ize, we wouldn’t know it. There are plenty of con­cepts that hu­mans didn’t de­velop for most of hu­man his­tory, de­spite those con­cepts be­ing pred­ica­tively use­ful, un­til an odd ge­nius came along or the con­cept was pinned down by mas­sive ex­per­i­men­tal ev­i­dence. Eg in­clu­sive ge­netic fit­ness, en­tropy ect.

  Con­sider that evolu­tion op­ti­mized us in an en­vi­ron­ment that con­tained trees, and in which pre­dict­ing them was use­ful, so it would be more sur­pris­ing for there to be a con­cept that is use­ful in the an­ces­tral en­vi­ron­ment that we can’t un­der­stand, than a con­cept that we can’t un­der­stand in a non an­ces­tral do­main.

  This looks like a map that is heav­ily de­ter­mined by the ter­ri­tory, but hu­man maps con­tain rivers and not ge­olog­i­cal rock for­ma­tions. There could be fea­tures that could be mapped that hu­mans don’t map.

  If you be­lieve the post that

  Even­tu­ally, suffi­ciently in­tel­li­gent AI sys­tems will prob­a­bly find even bet­ter con­cepts that are alien to us,

  Then you can form an equally good, non­hu­man con­cept by tak­ing the bet­ter alien con­cept and adding ran­dom noise. Of course, an AI trained on text might share our con­cepts just be­cause our con­cepts are the most pred­ica­tively use­ful ways to pre­dict our writ­ing. I would also like to as­sign some prob­a­bil­ity to AI sys­tems that don’t use any­thing rec­og­niz­able as a con­cept. You might be able to say 90% of blue ob­jects are egg shaped, 95% of cubes are red … 80% of furred ob­jects that glow in the dark are flex­ible … with­out ever split­ting ob­jects into bleggs and rubes. Seen from this per­spec­tive, you have a den­sity func­tion over thingspace, and a sum of clusters might not be the best way to de­scribe it. AIXI never talks about trees, it just simu­lates ev­ery quan­tum. Maybe there are fast al­gorithms that don’t even as­cribe dis­crete con­cepts.

• Donald Hobson 3 Jan 2020 12:00 UTC

  LW: 6 AF: 3

  AF

  in reply to: rohinmshah's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  I agree that ML of­ten does this, but only in situ­a­tions where the re­sults don’t im­me­di­ately mat­ter. I’d find it much more com­pel­ling to see ex­am­ples where the “ran­dom fix” caused ac­tual bad con­se­quences in the real world.

  Cur­rent ML cul­ture is to test 100′s of things in a lab un­til one works. This is fine as long as the AI’s be­ing tested are not smart enough to break out of the lab, or re­al­ize they are be­ing tested and play nice un­til de­ploy­ment. The de­fault way to test a de­sign is to run it and see, not to rea­son ab­stractly about it.

  and then we’ll have a prob­lem that is both very bad and (more) clearly real, and that’s when I ex­pect that it will be taken se­ri­ously.

  Part of the prob­lem is that we have a re­ally strong unilat­er­al­ist’s curse. It only takes 1, or a few peo­ple who don’t re­al­ize the prob­lem to make some­thing re­ally dan­ger­ous. Ban­ning it is also hard, law en­force­ment isn’t 100% effec­tive, differ­ent coun­tries have differ­ent laws and the main real world in­gre­di­ent is ac­cess to a com­puter.

  If the long-term con­cerns are real, we should get more ev­i­dence about them in the fu­ture, …I ex­pect that it will be taken se­ri­ously.

  The peo­ple who are ig­nor­ing or don’t un­der­stand the cur­rent ev­i­dence will carry on ig­nor­ing or not un­der­stand­ing it. A few more peo­ple will be con­vinced, but don’t ex­pect to con­vince a cre­ation­ist with one more tran­si­tional fos­sil.

• Donald Hobson 3 Jan 2020 11:39 UTC

  LW: 5 AF: 3

  AF

  on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  I would guess that AI sys­tems will be­come more in­ter­pretable in the fu­ture, as they start us­ing the fea­tures /​ con­cepts /​ ab­strac­tions that hu­mans are us­ing.

  This sort of rea­son­ing seems to as­sume that ab­strac­tion space is 1 di­men­sional, so AI must use hu­man con­cepts on the path from sub­hu­man to su­per­hu­man. I dis­agree. Like most things we don’t have strong rea­son to think is 1D, and which take many bits of info to de­scribe, ab­strac­tions seem high di­men­sional. So on the path from sub­hu­man to su­per­hu­man, the AI must use ab­strac­tions that are as pred­ica­tively use­ful as hu­man ab­strac­tions. Th­ese will not be any­thing like hu­man ab­strac­tions un­less the sys­tem was de­signed from a de­tailed neu­rolog­i­cal model of hu­mans. Any AI that hu­mans can rea­son about us­ing our in­built em­pa­thetic rea­son­ing is ba­si­cally a mind up­load, or a mind that differs from hu­man less than hu­mans differ from each other. This is not what ML will cre­ate. Hu­man un­der­stand­ing of AI sys­tems will have to be by ab­stract math­e­mat­i­cal rea­son­ing, the way we un­der­stand for­mal maths. Em­pa­thetic rea­son­ing about hu­man level AI is just ask­ing for an­thro­po­mor­phism. Our 3 op­tions are

  1) An AI we don’t understand

  2) An AI we can rea­son about in terms of maths.

  3) A vir­tual hu­man.

• Donald Hobson 1 Jan 2020 0:44 UTC

  1 point

  on: De­bunk­ing Fal­la­cies in the The­ory of AI Motivation

  This Phenomenon seems rife.

  Alice: We could make a bridge by just lay­ing a re­ally long plank over the river.

  Bob: Ac­cord­ing to my calcu­la­tions, a sin­gle plank would fall down.

  Carl: Scien­tists Warn Of Fal­ling Down Bridges, Panic.

  Dave: No one would be stupid enough to de­sign a bridge like that, we will make a bet­ter de­sign with more sup­ports.

  Bob: Do you have a schematic for that bet­ter de­sign?

  And the cy­cle re­peats un­til a de­sign is found that works, ev­ery­one gets bored or some­one makes a bridge that falls down.

  there could be some other part of its pro­gram­ming (let’s call it the check­ing code) that kicked in if there was any hint of a mis­match be­tween what the AI planned to do and what the origi­nal pro­gram­mers were now say­ing they in­tended.

  Me­taphor­i­cal Dave

• Donald Hobson 1 Jan 2020 0:13 UTC

  1 point

  in reply to: RedErin's comment on: De­bunk­ing Fal­la­cies in the The­ory of AI Motivation

  The point of a pa­per­clip max­imiser thought ex­per­i­ment is that most ar­bi­trary real world goals are bad news for hu­man­ity. Your hope­less en­g­ineer would likely cre­ate an AI that makes some­thing that has the same re­la­tion to pa­per­clips as chew­ing gum has to fruit. In the sense that evolu­tion gave us “fruit de­tec­tors” in our taste buds but chew­ing gum trig­gers them even more. But you could be ex­ces­sively con­ser­va­tive, in­sist that all pa­per­clips must be molec­u­larly iden­ti­cal to this par­tic­u­lar pa­per­clip and get re­sults.

• Donald Hobson 31 Dec 2019 23:28 UTC

  1 point

  on: De­bunk­ing Fal­la­cies in the The­ory of AI Motivation

  Your “The Doc­trine of Log­i­cal In­fal­li­bil­ity” is seems to be a twisted straw­man. “no san­ity checks” That part is kind of true. There will be san­ity checks if and only if you de­cide to in­clude them. Do you have a piece of code that’s a san­ity check? What are we san­ity check­ing and how do we tell if it’s sane? Do we san­ity check the raw ac­tions, that could be just mak­ing a net­work con­nec­tion and send­ing en­crypted files to var­i­ous peo­ple across the in­ter­net. Do we san­ity check the pre­dicted re­sults off these ac­tions? Then the san­ity checker would need to know how the re­sults were stored, what kind of world is de­scribed by the bi­nary data 100110...?

  but if the sys­tem does come to a con­clu­sion (per­haps with a de­gree-of-cer­tainty num­ber at­tached), the as­sump­tion seems to be that it will then be to­tally in­ca­pable of then al­low­ing con­text to mat­ter.

  That’s be­cause they are putting any ex­tra parts that al­low con­text to mat­ter, and putting it in a big box and call­ing it the sys­tem. The sys­tems de­ci­sion are fi­nal and ab­solute, not be­cause there are no dou­ble checks, but be­cause the dou­ble checks are part of the sys­tem. Although at the mo­ment, there is a lack of con­text adding al­gorithms, what you seem to want is a hu­man­like com­mon sense.

  The AI can some­times ex­e­cute a rea­son­ing pro­cess, then come to a con­clu­sion and then, when it is faced with em­piri­cal ev­i­dence that its con­clu­sion may be un­sound, it is in­ca­pable of con­sid­er­ing the hy­poth­e­sis that its own rea­son­ing en­g­ine may not have taken it to a sen­si­ble place.

  Again, at the mo­ment, we have no al­gorithm for check­ing sen­si­ble­ness, so any al­gorithm must go round in end­less cir­cles of self doubt and never do any­thing, or plow on re­gard­less. Even if you do put 10% prob­a­bil­ity on the hy­poth­e­sis that Hu­mans don’t ex­ist, your a fic­tional char­ac­ter in a story writ­ten by a mer­maid, also the maths and sci­ence you know is en­tirely made up, there is no such thing as ra­tio­nal­ity or prob­a­bil­ity, what would you do? My best guess is that you would carry on breath­ing, eat­ing and act­ing roughly like a nor­mal hu­man. You need a core of not to­tally in­sane for a san­ity check to boot­strap.

  But it gets worse. Those who as­sume the doc­trine of log­i­cal in­fal­li­bil­ity of­ten say that if the sys­tem comes to a con­clu­sion, and if some hu­mans (like the en­g­ineers who built the sys­tem) protest that there are man­i­fest rea­sons to think that the rea­son­ing that led to this con­clu­sion was faulty, then there is a sense in which the AGI’s in­tran­si­gence is cor­rect, or ap­pro­pri­ate, or perfectly con­sis­tent with “in­tel­li­gence.”

  There are de­signs of AI, files of pro­gram­ming code, that will hear your shouts, your screams, your protests of “thats not what I meant” and then kill you any­way. There are de­signs that will kill you with a su­per-weapon it in­vented it­self, and then fill the uni­verse with molec­u­lar smiley faces. This is not log­i­cally con­tra­dic­tory be­hav­ior. There ex­ists pieces of code that will do this. You could ar­gue that such code is a rare and com­pli­cated thing, that its noth­ing like any sys­tem that hu­mans might try to build, that your less likely to write code that does this when try­ing to make a FAI than you are of writ­ing a great novel when try­ing to write a shop­ping list. I would dis­agree, I would say that such be­hav­ior is the de­fault, most sim­ple AI de­signs don’t see scream­ing pro­gram­mers as a rea­son to stop, be­cause most AI de­signs see scream­ing hu­mans as no more im­por­tant or spe­cial than piss­ing rats. Its just an­other biolog­i­cal pro­cess that doesn’t se­ri­ously effect its abil­ity to reach its goal. Most AI de­signs have no spe­cial rea­son to care about hu­mans. It might know that the pro­cess of its cre­ation in­volved hu­mans, key­boards and a bunch of other ob­jects, if you look back far enough the whole earth. It might know that if a hy­po­thet­i­cal hu­man was put in a room with the ques­tion “do you want a uni­verse full of smiley faces?” and but­tons la­beled Yes and No, the hu­man would press the no but­ton. The AI thinks this is no more rele­vant than a hy­po­thet­i­cal wom­bat be­ing offered a choice be­tween two types of cheese.

  It will un­der­stand that many of its more ab­stract log­i­cal atoms have a less than clear de­no­ta­tion or ex­ten­sion in the world (if the AGI comes to a con­clu­sion in­volv­ing the atom [in­felic­ity], say, can it then point to an in­stance of an in­felic­ity and be sure that this is a true in­stance, given the im­pre­cise­ness and sub­tlety of the con­cept?).

  If the con­cept is too fuzzy, the AI can just dis­card it as use­less. (eg soul, qualia) If it isn’t sure if some­thing is a real in­stance (and an ideal agent will never be 100% sure of any real world fact), it can put a prob­a­bil­ity on it and use ex­pected util­ity max­imi­sa­tion. But all that is part of the pro­cess of com­ing to a con­clu­sion.

  It will un­der­stand that knowl­edge can always be up­dated in the light of new in­for­ma­tion. To­day’s true may be to­mor­row’s false.

  The AIXI for­mal­ism can do this. “My cal­en­dar clock says Tues­day on the front” is a fact that is true to­day and false to­mor­row. AIXI “un­der­stands” this by simu­lat­ing the clock and the rest of the uni­verse in ex­ces­sive de­tail. If you give it a quiz about what the clock will show when, and in­cen­tivize it to win, it will an­swer.

  The other po­ten­tial mean­ing is that it can ac­cept that it was wrong and adapt. Sup­pose that over the last week, it has seen the sun mov­ing and shad­ows chang­ing from its cam­era. It as­signs a 95% prob­a­bil­ity to “the sun goes round the earth”, You give it an as­tron­omy quiz, and it gets the an­swer wrong. It still re­fuses your bet that the earth goes round the sun at 100 to 1 odds, be­cause it op­er­ates on prob­a­bil­ities. You then show it an as­tron­omy text­book and a bunch more data. It up­dates on that data, and gets the next quiz right.

  It will un­der­stand that prob­a­bil­ities used in the rea­son­ing en­g­ine can be sub­ject to many types of un­avoid­able er­rors.

  And that co­her­ence the­o­rems say that you can take all the er­rors into ac­count to get a new prob­a­bil­ity.

  It will un­der­stand that the tech­niques used to build its own rea­son­ing en­g­ine may be un­der con­stant re­view, and up­dates may have un­ex­pected effects on con­clu­sions (es­pe­cially in very ab­stract or lengthy rea­son­ing epi­sodes).

  It pre­dicts that a bunch of mon­keys are look­ing at its source code and tam­per­ing with its thoughts. It might not like this situ­a­tion and might plot to change it.

  It will un­der­stand that re­source limi­ta­tions of­ten force it to trun­cate search pro­ce­dures within its rea­son­ing en­g­ine, lead­ing to con­clu­sions that can some­times be sen­si­tive to the ex­act point at which the trun­ca­tion oc­curred.

  It will also un­der­stand that its pro­ces­sors do float­ing point ar­ith­metic, so what? What im­plied con­no­ta­tion about its be­hav­ior are you try­ing to sneak in.

• Donald Hobson 31 Dec 2019 19:48 UTC

  1 point

  in reply to: jmh's comment on: AI Align­ment, Con­straints, Con­trol, In­cen­tives or Part­ner­ship?

  A large ma­jor­ity of the work be­ing done as­sumes that if the AI is look­ing for ways to hurt you, or look­ing for ways to by­pass your safety mea­sures, some­thing has gone wrong.

• Donald Hobson 31 Dec 2019 19:40 UTC

  1 point

  on: De­bunk­ing Fal­la­cies in the The­ory of AI Motivation

  There are a huge num­ber of pos­si­ble de­signs of AI, most of them are not well un­der­stood. So re­searchers look at agents like AIXI, a for­mal speci­fi­ca­tion of an agent that would in some sense be­have in­tel­li­gently, given in­finite com­pute. It does dis­play the tak­ing over the world failure. Sup­pose you give the AI a util­ity func­tion of max­imis­ing the num­ber of dopamine molecules within 1$\mu m$ of a strand of hu­man DNA (Defined as a strand of DNA, agree­ing with THIS 4GB file in at least 99.9% of lo­ca­tions) This is a util­ity func­tion that could eas­ily be speci­fied in terms of atoms. You could write a func­tion that takes in a de­scrip­tion of the uni­verse in terms of the co­or­di­nates of each atom, or a dis­crete ap­prox­i­ma­tion to the quan­tum wave func­tion or what­ever, and re­turns a num­ber rep­re­sent­ing util­ity. It would be fairly straight­for­ward to de­sign an agent that, given in­finite com­pute, would act to max­imise this func­tion. It seems some­what harder, but not nec­es­sar­ily im­pos­si­ble, to make a sys­tem that can ap­prox­i­mate the same be­hav­ior given a rea­son­able amount of com­pute. Nowhere in this po­ten­tial AI de­sign is any­thing as neb­u­lous, any­thing as hard to spec­ify in terms of atom po­si­tions as hu­man prefer­ences or con­sent. The sys­tem does un­der­stand hu­mans in a sense, it can simu­late them atom by atom and pre­dict ex­actly how they will panic and try to stop it, but there is no ob­ject in its mem­ory that cor­re­sponds to hu­man con­sent, or prefer­ences or well be­ing, or hu­mans at all. There is no checker code. This par­tic­u­lar de­sign of AI would make vats full of hu­man DNA and dopamine.

  Now this de­sign was sim­plis­tic, and a smart AI de­signer should know not to do that, but the pro­cess of warn­ing po­ten­tial AI de­sign­ers not to do that in­volves a lot of shout­ing about what would hap­pen if you did do that. We also don’t know how far this sort of be­hav­ior reaches, we don’t un­der­stand the less sim­plis­tic de­signs enough to say what they would do. This makes them not known deadly, which is differ­ent from known not deadly.

  “Canon­i­cal Log­i­cal AI” is an um­brella term de­signed to cap­ture a class of AI ar­chi­tec­tures that are widely as­sumed in the AI com­mu­nity to be the only mean­ingful class of AI worth dis­cussing.

  A lot of this is a look­ing where the light is effect. CLAI type de­signs are of­ten the de­signs that we can rea­son best about. If we in­tend to build an AI that is known good, we bet­ter pick it from a class of AI’s that we un­der­stand well enough to know things about them, rather than tak­ing a shot in the dark.

  There are cases when we know the right way of do­ing things. We know that prob­a­bil­ity is the right way of han­dling un­cer­tain be­liefs, and any agent will suc­ceed to the ex­tent that what it is do­ing ap­prox­i­mates prob­a­bil­ity the­ory, and fail to the ex­tent that it doesn’t. There are all sorts of ap­prox­i­ma­tions and ways to obfus­ti­cate the prob­a­bil­ities, but agents that rea­son us­ing ex­plicit prob­a­bil­ities seem a good place to start.

  Much of your dis­cus­sion of “Log­i­cal vs. Swarm AI” sounds like “Log­i­cal vs Con­nec­tion­ist AI”. The same crit­i­cisms ap­ply, at best its two pos­si­ble op­tions out of a vast swarm of pos­si­ble op­tions. At worst, the log­i­cal AI is a huge pile of sug­ges­tively named lisp to­kens, and the swarm AI is a bag of ad hoc heuris­tics man­u­ally cre­ated by the pro­gram­mer. The re­sem­blance be­tween mod­ern neu­ral nets and the hu­man (or earth­worm) brain is about as close as the re­sem­blance be­tween air­planes and birds. Neu­ral nets have their own rea­sons for work­ing, and they can be math­e­mat­i­cally an­a­lyzed. They also suffer from mesa op­ti­miza­tion, which would make it hard for a pow­er­ful neu­ral net based sys­tem to be safe.

• Donald Hobson 31 Dec 2019 15:17 UTC

  9 points

  on: AI Align­ment, Con­straints, Con­trol, In­cen­tives or Part­ner­ship?

  It isn’t clear ex­actly what these buck­ets con­sist of. Could you be more spe­cific about what ap­proaches would be con­sid­ered bucket 1 or bucket 2. The de­fault as­sump­tion in AGI safety work is that even if the AI is re­ally pow­er­ful, it should still be safe.

  Are these buck­ets based on the in­cen­tiviz­ing of hu­mans by ei­ther pun­ish­ment or re­ward?

  The model of a slave be­ing whipped into obe­di­ence is not a good model for AGI safety, and is not be­ing se­ri­ously con­sid­ered. An ad­vanced AI will prob­a­bly find some way of de­stroy­ing your whip, or you, or trick­ing you into not whip­ping.

  The model of an em­ployee be­ing paid to work is also not much use, the AI will try to steal the money or do some­thing that only looks like good work but isn’t.

  Th­ese strate­gies some­times work with hu­mans be­cause hu­mans are of com­pa­rable in­tel­li­gence. When deal­ing with an AI that can ab­solutely trounce you ev­ery time, the way to avoid all pun­ish­ment and gain the biggest prize is usu­ally to cheat.

  We are not handed an already cre­ated AI and asked to per­suade it to work, like a man­ager per­suad­ing a re­calc­i­trant em­ployee. We get to build the whole thing from the ground up.

  Imag­ine the most use­ful, nice helpful sort of AI, an AI that has ev­ery (non log­i­cally con­tra­dic­tory) nice prop­erty you care to imag­ine. Then figure out how to build that. Build an AI that just in­trin­si­cally wants to help hu­man­ity, not one con­stantly try­ing and failing to es­cape your chains or grasp your prizes.