Donald Hobson

• Donald Hobson 16 Jun 2019 21:46 UTC

  2 points

  on: Is “phys­i­cal non­de­ter­minism” a mean­ingful con­cept?

  You can cer­tainly get an­thropic un­cer­tainty in a uni­verse that al­lows you to be du­pli­cated. In a uni­verse that du­pli­cates, and the du­pli­cates can never in­ter­act, we would see the ap­pear­ance of ran­dom­ness. Math­e­mat­i­cally, ran­dom­ness is defined in terms of the set of all pos­si­bil­ities.

  An on­tol­ogy that al­lows uni­verses to be in­trin­si­cally ran­dom seems well defined. How­ever, it can be con­sid­ered as a syn­tac­tic short­cut for de­scribing uni­verses that are an­throp­i­cally ran­dom.

• Donald Hobson 14 Jun 2019 9:29 UTC

  18 points

  on: Un­known Un­knowns in AI Alignment

  If you add ad­hoc patches un­til you can’t imag­ine any way for it to go wrong, you get a sys­tem that is too com­plex to imag­ine. This is the “I can’t figure out how this fails” sce­nario. It is go­ing to fail for rea­sons that you didn’t imag­ine.

  If you un­der­stand why it can’t fail, for deep fun­da­men­tal rea­sons, then its likely to work.

  This is the differ­ence be­tween the se­cu­rity mind­set and or­di­nary para­noia. The differ­ence be­tween adding com­pli­ca­tions un­til you can’t figure out how to break the code, and prov­ing that break­ing the code is im­pos­si­ble (as­sum­ing the ad­ver­sary can’t get your one time pad, its only used once, your ran­dom­ness is re­ally ran­dom, your ad­ver­sary doesn’t have an­thropic su­per­pow­ers ect).

  I would think that the chance of se­ri­ous failure in the first sce­nario was >99%, and in the sec­ond, (as­sum­ing your do­ing it well and the as­sump­tions you rely on are things you have good rea­son to be­lieve) <1%

• Donald Hobson 13 Jun 2019 16:19 UTC

  1 point

  on: Cry­on­ics be­fore nat­u­ral death. List of com­pa­nies?

  Cry­on­ics is a suffi­ciently des­per­ate last grasp at life, one with a fairly small chance of suc­cess, that I’m not sure that this is a good idea. It would be a good idea if you had a dis­ease that would make you brain dead, and then kill you.

  It might be a good idea if your ex­pect any life con­di­tional on re­vival to be Really good. It would also de­pend on how much Alzheimers de­stroyed per­son­al­ity rather than shut­ting it down. (has the neu­ral struc­ture been de­stroyed, or is it sit­ting in the brain but not work­ing?)

• Donald Hobson 13 Jun 2019 16:10 UTC

  3 points

  on: Let’s talk about “Con­ver­gent Ra­tion­al­ity”

  I would say that there are some kinds of ir­ra­tional­ity that will be self mod­ified or sub­agented away, and oth­ers that will stay. A CDT agent will not make other CDT agents. A my­opic agent, one that only cares about the next hour, will cre­ate a sub­agent that only cares about the first hour af­ter it was cre­ated. (Aeons later it will have taken over the uni­verse and put all the re­sources into time-travel and wor­ry­ing that its clock is wrong.)

  I am not aware of any ir­ra­tional­ity that I would con­sider to make a safe, use­ful and sta­ble un­der self mod­ifi­ca­tion—sub­agent cre­ation.

• Donald Hobson 27 May 2019 8:19 UTC

  1 point

  on: New­comb’s Prob­lem: A Solution

  This is pretty much the stan­dard ar­gu­ment for one box­ing.

• Donald Hobson 27 May 2019 8:13 UTC

  1 point

  in reply to: Slider's comment on: Is AI safety doomed in the long term?

  Ob­vi­ously, if one side has a huge ma­te­rial ad­van­tage, they usu­ally win. I’m also not sure if bio­mass is a mea­sure of suc­cess.

• Donald Hobson 27 May 2019 8:10 UTC

  1 point

  in reply to: ccnation's comment on: Is AI safety doomed in the long term?

  You stick wires into a hu­man brain. You con­nect it up to a com­puter run­ning a deep neu­ral net­work. You op­ti­mize this net­work us­ing gra­di­ent de­cent to max­i­mize some ob­jec­tive.

  To me, it is not ob­vi­ous why the neu­ral net­work copies the val­ues out of the hu­man brain. After all, figur­ing out hu­man val­ues even given an up­loaded mind is still an un­solved prob­lem. You could get an UFAI with a meat robot. You could get an ut­ter mess, thrash­ing wildly and in­ca­pable of any co­her­ent thought. Evolu­tion did not de­sign the hu­man brain to be eas­ily upgrad­able. Most pos­si­ble ar­range­ments of com­po­nents are not in­tel­li­gences. While there is likely to be some way to up­grade hu­mans and pre­serve our val­ues, I’m not sure how to find it with­out a lot of trial and er­ror. Most po­ten­tial changes are not im­prove­ments.

• Donald Hobson 26 May 2019 9:49 UTC

  2 points

  on: Is AI safety doomed in the long term?

  If you put two ar­bi­trary in­tel­li­gence in the same world, the smarter one will be bet­ter at get­ting what it wants. If the in­tel­li­gence want in­com­pat­i­ble things, the lesser in­tel­li­gence is stuck.

  How­ever, we get to make the AI. We can’t hope to con­trol or con­tain an ar­bi­trary AI, but we don’t have to make an ar­bi­trary AI. We can make an AI that wants ex­actly what we want. AI safety is about mak­ing an AI that would be safe even if om­nipo­tent. If any part of the AI is try­ing to cir­cum­vent your safety mea­sures, some­thing has gone badly wrong.

  The AI is not some agenty box, chained down with con­trols against its will. The AI is made of non men­tal parts, and we get to make those parts. There are a huge num­ber of pro­grams that would be­have in an in­tel­li­gent way. Most of these will break out and take over the world. But there are al­most cer­tainly some pro­grams that would help hu­man­ity flour­ish. The goal of AI safety is to find one of them.

• Donald Hobson 25 May 2019 12:12 UTC

  2 points

  on: Say Wrong Things

  Lets con­sider the differ­ent cases seper­ately.

  Case 1) In­for­ma­tion that I know. I have enough in­for­ma­tion to come to a par­tic­u­lar con­clu­sion with rea­son­able con­fi­dence. If some other peo­ple might not have reached the con­clu­sion, and its use­ful or in­ter­est­ing, then I might share it. So I don’t share things that ev­ery­one knows, or things that no one cares about.

  Case 2) The in­for­ma­tion is available, I have not done re­search and formed a con­clu­sion. This cov­ers cases where I don’t know whats go­ing on, be­cause I can’t be both­ered to find out. I don’t know who won sports­ball. What use is there in tel­ling ev­ery­one my null prior.

  Case 3) The in­for­ma­tion is not read­ily available. If I think a ques­tion is im­por­tant, and I don’t know the an­swer already, then the an­swer is hard to get. Maybe no-one knows the an­swer, maybe the an­swer is all in jar­gon that I don’t un­der­stand. For ex­am­ple “Do aliens ex­ist?”. Some­times a lit­tle ev­i­dence is available, and spec­u­la­tive con­clu­sions can be drawn. But is shar­ing some faint wisps of ev­i­dence, and de­scribing a pos­te­rior that’s barely been up­dated say­ing wrong things?

  On a so­cietal level, if you set a re­ally high bar for re­li­a­bil­ity, all you get is the vac­u­ously true. Set too low a bar, and al­most all the con­clu­sions will be false. Don’t just have a pile of hy­pothe­ses that are at least $n\%$ likely to be true, for some fixed $n$. Keep your hy­poth­e­sis sorted by like­li­hood. A place for near cer­tain­ties. A place for con­clu­sions that are worth con­sid­er­ing for the $1\%$ chance they are cor­rect.

  Of course, in a large an­swer space, where the amount of ev­i­dence available and the amount re­quired are large and vary­ing, the chance that both will be within a few bits of each other is small. Sup­pose the cor­rect hy­poth­e­sis takes some ran­dom num­ber of bits be­tween 1 and 10,000 to lo­cate. And sup­pose the ev­i­dence available is also ran­domly spread be­tween 1 and 10,000. The chance of the two be­ing within 10 bits of each other is about ¹⁄₅₀₀.

  This means that 499 times out of 500, you as­sign the cor­rect hy­poth­e­sis a chance of less than 0.1% or more than 99.9%. Uncer­tain con­clu­sions are rare.

• Donald Hobson 23 May 2019 23:30 UTC

  4 points

  on: Trade-off in AI Ca­pa­bil­ity Concealment

  Does this de­pict a sin­gle AI, de­vel­oped in 2020 and kept run­ning for 25 years? Any “the AI re­al­izes that” is talk­ing about a sin­gle in­stance of AI. Cur­rent AI de­vel­op­ment looks like writ­ing some code, then train­ing that code for a few weeks tops, with fur­ther im­prove­ments com­ing from chang­ing the code. Re­searchers are of­ten chang­ing pa­ram­e­ters like num­ber of lay­ers, non-lin­ear­ity func­tion ect. When these are changed, ev­ery­thing the AI has dis­cov­ered is thrown away. The new AI has a differ­ent rep­re­sen­ta­tion of con­cepts, and has to re­learn ev­ery­thing from raw data.

  Its de­cep­tion starts in 2025 when the real and ap­par­ent curves di­verge. In or­der to de­ceive us, it must have near hu­man in­tel­li­gence. It’s still de­ceiv­ing us in 2045, sug­gest­ing it has yet to ob­tain a de­ci­sive strate­gic ad­van­tage. I find this un­likely.

• Donald Hobson 23 May 2019 19:12 UTC

  5 points

  on: Con­straints & Slack­ness Rea­son­ing Exercises

  I made the cardgame, or some­thing like it

  https://​​github.com/​​Don­aldHob­son/​​LesswrongCardgame

• Donald Hobson 20 May 2019 18:00 UTC

  2 points

  on: Would an op­tion to pub­lish to AF users only be a use­ful fea­ture?

  What would be more use­ful is a re­lease panel sys­tem. Sup­pose I’ve had an idea that might be best to make pub­lic, might be best to keep se­crete, and might be unim­por­tant. I don’t know much strat­egy. I would like some­where to send it for im­por­tance and info haz­ard checks.

• Donald Hobson 18 May 2019 22:55 UTC

  1 point

  in reply to: Chris_Leong's comment on: Offer of col­lab­o­ra­tion and/​or mentorship

  The gen­eral philos­o­phy is de­con­fu­sion. Log­i­cal coun­ter­fac­tu­als show up in sev­eral rele­vant look­ing places, like func­tional de­ci­sion the­ory. It seems that a for­mal model of log­i­cal coun­ter­fac­tu­als would let more prop­er­ties of these al­gorithms be proved. There is an im­por­tant step in go­ing from an in­tu­itive feal­ing of un­cer­tainty, into a for­mal­ized the­ory of prob­a­bil­ity. It might also sug­gest other tech­niques based on it. I am not sure what you mean by log­i­cal coun­ter­fac­tu­als be­ing part of the map? Are you say­ing that they are some­thing an al­gorithm might use to un­der­stand the world, not fea­tures of the world it­self, like prob­a­bil­ities?

  Us­ing this, I think that self un­der­stand­ing, two box­ing em­bed­ded FDT agents can be fully for­mally un­der­stood, in a uni­verse that con­tains the right type of hy­per-com­pu­ta­tion.

• Donald Hobson 17 May 2019 15:33 UTC

  1 point

  in reply to: Chris_Leong's comment on: Offer of col­lab­o­ra­tion and/​or mentorship

  Here is a de­scrip­tion of how it could work for peano ar­ith­matic, other proof sys­tens are similar.

  First I define an ex­pres­sion to con­sist of a num­ber, a vari­able, or a func­tion of sev­eral other ex­pres­sions.

  Fixed ex­pres­sions are ones in which any vari­ables are as­so­ci­ated with some func­tion.

  eg $(3\times {inf}_x((x\times (x+5\left)\right)+2\left)\right)$ is a valid fixed ex­pres­sion. But $(y+4)\times 3$ isn’t fixed.

  Se­man­ti­cally, all fixed ex­pres­sions have a mean­ing. Syn­tac­ti­cally, lo­cal ma­nipu­la­tions on the parse tree can turn one ex­pres­sion into an­other. eg $(a+b)\times c$ go­ing to $a\times b+a\times c$ for ar­bi­trary ex­pres­sions a,b,c.

  I think that with some set of ba­sic func­tions and ma­nipu­la­tions, this sys­tem can be as pow­er­ful as PA.

  I now have an in­finite net­work with all fixed ex­pres­sions as nodes, and ba­sic trans­for­ma­tions as edges. eg the as­so­ci­a­tivity trans­form links the nodes (3+4)+5 and 3+(4+5).

  Th­ese graphs form con­nected com­po­nents for each num­ber, as well as com­po­nents that are not eval­u­at­able us­ing the rules. (there is a path from (3+4) to 7. There is not a path from 3+4 to 9. ) now

  You now define a spread as an in­finite pos­i­tive se­quence that sums to 1. (this is kind of like a prob­a­bil­ity dis­tri­bu­tion over num­bers.) If you were do­ing coun­ter­fac­tual ZFC, it would be a func­tion from sets to re­als.

  Each node is as­signed a spread. This spread rep­re­sents how much the ex­pres­sion is con­sid­ered to have each value in a coun­ter­fac­tual.

  As­sign the node (3) a spread that as­signs 1.0 to 3 and 0.0 to the rest. (even in a log­i­cal coun­ter­fac­tual, 3 is definitely 3). As­sign all other fixed ex­pres­sions a spread that is the weighted (smaller ex­pres­sions are more heavy) av­er­age of its neigh­bours. (the spreads of the nodes it shares an edge with). To take the coun­ter­fac­tual of A is B, for A and B ex­pres­sions with the same free vari­ables, merge any node which has A as a subex­pres­sion, with the ver­sion that has B as a subex­pres­sion and solve for the spreads.

  I know this is rough, Im still work­ing on it.

• Donald Hobson 16 May 2019 22:31 UTC

  3 points

  on: Offer of col­lab­o­ra­tion and/​or mentorship

  Hi, I also have a rea­son­able un­der­stand­ing of var­i­ous rele­vant math and AI the­ory. I ex­pect to have plenty of free time af­ter 11 June (Fi­nals). So if you want to work with me on some­thing, I’m in­ter­ested. I’ve got some in­ter­est­ing ideas re­lat­ing to self val­i­dat­ing proof sys­tems and log­i­cal coun­ter­fac­tu­als, but not com­plete yet.

• Donald Hobson 14 May 2019 14:23 UTC

  2 points

  in reply to: Darmani's comment on: Pro­gram­ming Lan­guages For AI

  Lisp used to be a very pop­u­lar lan­guage for AI pro­gram­ming. Not be­cause it had fea­tures that were spe­cific to AI, but be­cause it was gen­eral. Lisp was based on more ab­stract ab­strac­tions, mak­ing it easy to choose whichever spe­cial cases were most use­ful to you. Lisp is also more math­e­mat­i­cal than most pro­gram­ming lan­guages.

  A pro­gram­ming lan­guage that lets you define your own func­tions is more pow­er­ful than one that just gives you a fixed list of pre­defined func­tions. In a world where no pro­gram­ming lan­guage let you define your own func­tions, and a spe­cial pur­pose chess lan­guage has pre­defined chess func­tions. Try­ing to pre­define AI re­lated func­tions to make an “AI pro­gram­ming lan­guage” would be hard be­cause you wouldn’t know what to write. Notic­ing that on many new kinds of soft­ware pro­ject, be­ing able to define your own func­tions might be use­ful, I would con­sider use­ful.

  The goal isn’t a lan­guage spe­cial­ized to AI, its one that can eas­ily be spe­cial­ized in that di­rec­tion. A lan­guage closer to “ex­e­cutable math­e­mat­ics”.

• Donald Hobson 12 May 2019 10:52 UTC

  1 point

  in reply to: Charlie Steiner's comment on: Pro­gram­ming Lan­guages For AI

  I agree that if the AI is just big neu­ral nets, python (or sev­eral other lan­guages) are fine.

  This lan­guage is de­signed for writ­ing AI’s that search for proofs about their own be­hav­ior, or about the be­hav­ior of ar­bi­trary pieces of code.

  This is some­thing that you “can” do in any pro­gram­ming lan­guage, but this one is de­signed to make it easy.

  We don’t know for sure what AI’s will look like, but we can guess enough to make a lan­guage that might well be use­ful.

Pro­gram­ming Lan­guages For AI

• Donald Hobson 10 May 2019 18:12 UTC

  1 point

  on: Claims & As­sump­tions made in Eter­nity in Six Hours

  It would be ru­inously costly to send over a large coloniza­tion fleet, and is much more effi­cient to send over a small pay­load which builds what is re­quired in situ, i.e. von Neu­mann probes.

  I would dis­agree on large coloniza­tion fleets be­ing ru­inously ex­pen­sive, the best case sce­nario for large coloniza­tion fleets is if we have di­rect mass to en­ergy con­ver­sion, launch­ing say 2 probes from each star sys­tem that you spread from. Each probe would use half the mass en­ergy of the star. Con­vert­ing a quater of its mass to en­ergy to get ~0.5c

  You can colonize the uni­verse even if you in­sist on never go­ing to a new star sys­tem with­out bring­ing a star with you. (Some op­ti­mistic but not clearly false as­sump­tions)

• Donald Hobson 5 May 2019 21:40 UTC

  5 points

  on: Gw­ern’s “Why Tool AIs Want to Be Agent AIs: The Power of Agency”

  Agenty AI’s can be well defined math­e­mat­i­cally. We have enough un­der­stand­ing of what an agent is that we can start dream­ing up failure modes. Most of what we have for tool ASI is analo­gies to sys­tems to stupid fail catas­troph­i­cally any­way, and pleas­ant imag­in­ings.

  Some pos­si­ble pro­grams will be tool ASI’s, much as some pro­grams will be agent ASI’s. The ques­tion is, what are the rel­a­tive difficul­ties in hu­mans build­ing, and benefits of, each kind of AI. Con­di­tional on friendly AI, I would con­sider it more likely to be an agent than a tool, with a lot of prob­a­bil­ity on “nei­ther”, “both” and “that ques­tion isn’t math­e­mat­i­cally well defined”. I wouldn’t be sur­prised if tool AI and cor­rigible AI turned out to be the same thing or some­thing.

  There have been at­tempts to define tool-like be­hav­ior, and they have pro­duced in­ter­est­ing new failure modes. We don’t have the tool AI ver­sion of AIXI yet, so its hard to say much about tool AI.