orthonormal

• orthonormal 14 May 2019 0:01 UTC

  2 points

  on: Co­her­ent de­ci­sions im­ply con­sis­tent utilities

  For­mat­ting re­quest: can the foot­note num­bers be aug­mented with links that jump to the foot­note text? (I pre­sume this worked in Ar­bital but broke when it was moved here.)

• orthonormal 12 May 2019 6:06 UTC

  12 points

  on: Com­plex Be­hav­ior from Sim­ple (Sub)Agents

  I had a no­tion here that I could stochas­ti­cally in­tro­duce a new goal that would min­i­mize to­tal suffer­ing over an agent’s life-his­tory. I tried this, and the most sta­ble solu­tion turned out to be thus: in­tro­duce an over­whelm­ingly aver­sive goal that causes the agent to run far away from all of its other goals scream­ing.

  did you mean: anhedonia

  (No, se­ri­ously, your para­graph is an apt de­scrip­tion of a long bout I had of de­pres­sion-in­duced an­he­do­nia; I felt so averse to ev­ery ac­tion that I ceased to feel wants, and I con­sis­tently marked my mood as neu­tral rather than nega­tive de­spite be­ing ob­jec­tively more severely de­pressed than I was at other times when I put nega­tive num­bers in my mood tracker.)

• orthonormal 5 May 2019 23:41 UTC

  21 points

  on: Dishon­est Up­date Reporting

  The ideal thing is to judge Bob as if he were mak­ing the same pre­dic­tion ev­ery day un­til he makes a new one, and log-score all of them when the event is re­vealed. (That is, if Bob says 75% on Jan­uary 1st and 60% on Fe­bru­ary 1st, and then on March 1st the event is re­vealed to have hap­pened, Bob’s score equals 31*log(.25) + 28*log(.4). Then Bob’s best strat­egy is to up­date his pre­dic­tion to his ac­tual cur­rent es­ti­mate as of­ten as pos­si­ble; past pre­dic­tions are sunk costs.

  The real-world ver­sion is re­mem­ber­ing to dock peo­ple’s bad pre­dic­tions more, the longer they per­sisted in them. But of course this is hard.

  538 did do this with their self-eval­u­a­tion, which is a good way to try and es­tab­lish a norm in the do­main of model-driven re­port­ing.

• orthonormal 4 May 2019 14:57 UTC

  4 points

  on: Peck­ing Order and Flight Leadership

  Let’s note differ­ences of de­gree here. Poli­ti­cal sys­tems differ mas­sively in how eas­ily de­ci­sion­mak­ers can claim large spoils for them­selves, and these differ­ences seem to cor­re­late with how pro-so­cial the de­ci­sions tend to be. In par­tic­u­lar, the dol­lar amounts of graft be­ing alleged for poli­ti­ci­ans in liberal democ­ra­cies are usu­ally small com­pared to what despots reg­u­larly claim with­out con­se­quence. (Which is not to say that it would be wise to ig­nore cor­rup­tion in liberal democ­ra­cies!)

• orthonormal 26 Apr 2019 4:24 UTC

  20 points

  on: [An­swer] Why wasn’t sci­ence in­vented in China?

  I’m not con­vinced that Europe had more in­tel­lec­tual free­dom on av­er­age than China, but be­cause of the patch­work of prin­ci­pal­ities, it cer­tainly had more vari­a­tion in in­tel­lec­tual free­dom than did a China that was at any given time ei­ther mostly unified or mostly at war; and all that you need for an in­tel­lec­tual rev­olu­tion is the ex­is­tence of a bas­tion of in­tel­lec­tual free­dom some­where.

• orthonormal 15 Feb 2019 22:29 UTC

  17 points

  on: How does OpenAI’s lan­guage model af­fect our AI timeline es­ti­mates?

  It doesn’t move much prob­a­bil­ity mass to the very near term (i.e. 1 year or less), be­cause both this and AlphaS­tar aren’t re­ally do­ing con­se­quen­tial­ist rea­son­ing, they’re just able to get a sur­pris­ing perfor­mance with sim­pler tricks (the very Marko­vian na­ture of hu­man writ­ing, a good po­si­tion eval­u­a­tion func­tion) given a whole lot of com­pute.
  

  How­ever, it does shift my prob­a­bil­ities for­ward in time, in the sense that one new weird trick to do de­duc­tive or con­se­quen­tial­ist rea­son­ing, plus a lot of com­pute, might get you there re­ally quickly.

• orthonormal 5 Oct 2018 18:56 UTC

  53 points

  on: The Rocket Align­ment Problem

  I ex­pect that some oth­er­wise con­vince­able read­ers are not go­ing to re­al­ize that in this fic­tional world, peo­ple haven’t dis­cov­ered New­ton’s physics or calcu­lus, and those read­ers are there­fore go­ing to miss the anal­ogy of “this is how MIRI would talk about the situ­a­tion if they didn’t already know the fun­da­men­tal con­cepts but had rea­sons for search­ing in the right di­rec­tion”. (I’m not think­ing of read­ers in­ca­pable of han­dling that coun­ter­fac­tual, but of read­ers who aren’t great at in­fer­ring im­plicit back­ground facts from a writ­ten di­alogue. Such read­ers might get very con­fused at the un­ex­pected turns of the di­alogue and quit rather than figure out what they’re baf­fled by.)

  I’d sug­gest adding to the pream­ble some­thing like “In a weird world where peo­ple had figured out work­able aero­nau­tics and ba­sic rocket propul­sion by trial and er­ror, but hadn’t figured out New­ton’s laws or calcu­lus”.

• orthonormal 7 May 2018 20:51 UTC

  9 points

  in reply to: Jason Gross's comment on: Fuzzy Boundaries, Real Concepts

  I like your defi­ni­tion, though, and want to try to make a bet­ter one (and I ac­knowl­edge this is not the point of this post).

  I think that’s a perfectly valid thing to do in the com­ments here! How­ever, I think your at­tempt,

  My stab at a re­fine­ment of “con­sent” is “re­spect for an­other’s choices”, where “dis­re­spect” is “de­liber­ately(?) do­ing some­thing to un­der­mine”

  is far too vague to be a use­ful con­cept.

  In most re­al­is­tic cases, I can give a definite an­swer to whether A touched B in a way B clearly did not want to be touched. In the case of my hon­esty defi­ni­tion, it does in­volve in­tent and so I can only in­fer statis­ti­cally when some­one else is be­ing dishon­est vs mis­taken, but for my­self I usu­ally have an an­swer about whether say­ing X to per­son C would be hon­est or not.

  I don’t think I could do the same for your defi­ni­tion; “am I re­spect­ing their choices” is a tough query to bot­tom out in ba­sic facts.

Fuzzy Boundaries, Real Concepts

• orthonormal 6 May 2018 21:21 UTC

  12 points

  in reply to: Wei_Dai's comment on: Lo­cal Val­idity as a Key to San­ity and Civilization

  My com­ment was meant to ex­plain what I un­der­stood Eliezer to be say­ing, be­cause I think you had mis­in­ter­preted that. The OP is sim­ply say­ing “don’t give weight to ar­gu­ments that are lo­cally in­valid, re­gard­less of what else you like about them”. Of course you need to use pri­ors, heuris­tics, and in­tu­itions in ar­eas where you can’t find an ar­gu­ment that car­ries you from be­gin­ning to end. But be­ing able to think “oh, if I move there, then they can take my queen, and I don’t see any­thing else good about that po­si­tion, so let’s not do that then” is a fair bit eas­ier than prov­ing your move op­ti­mal.

• orthonormal 22 Apr 2018 3:14 UTC

  21 points

  in reply to: Wei_Dai's comment on: Lo­cal Val­idity as a Key to San­ity and Civilization

  Rely­ing purely on lo­cal val­idity won’t get you very far in play­ing chess

  The equiv­a­lent of lo­cal val­idity is just me­chan­i­cally check­ing “okay, if I make this move, then they can make that move” for a bunch of cases. Which, first, is a ma­jor de­vel­op­men­tal mile­stone for kids learn­ing chess. So we only think it “won’t get you very far” be­cause all the high-level hu­man play ex­plic­itly or im­plic­itly takes it for granted.

  And sec­ondly, it’s pretty analo­gous to do­ing math; prov­ing the­o­rems is based on the abil­ity to check the lo­cal val­idity of each step, but math­e­mat­i­ci­ans aren’t just brute-forc­ing their way to proofs. They have to de­velop higher-level heuris­tics, some of which are re­ally hard to ex­press in lan­guage, to sug­gest av­enues, and then check lo­cal val­idity once they have a skele­ton of some part of the ar­gu­ment. But if math­e­mat­i­ci­ans stopped do­ing that an­noy­ing bit, well, then af­ter a while you’ll end up with an­other crisis of anal­y­sis when the brilli­ant in­tu­itions are miss­ing some tiny in­gre­di­ent.

  Lo­cal val­idity is an in­cred­ibly im­por­tant part of any sci­en­tific dis­ci­pline; the fact that it’s not a part of most poli­ti­cal dis­course is merely a re­flec­tion that our so­ciety is at about the de­vel­op­men­tal level of a seven-year-old when it comes to poli­ti­cal rea­son­ing.

• orthonormal 4 Apr 2018 5:16 UTC

  4 points

  on: Non-Ad­ver­sar­ial Good­hart and AI Risks

  Bro­ken link on the text “real kil­ling of birds to re­duce pests in China has never been tried”.

• orthonormal 4 Apr 2018 4:52 UTC

  4 points

  on: The Costly Co­or­di­na­tion Mechanism of Com­mon Knowledge

  Much of this ma­te­rial is cov­ered very similarly in Melt­ing Asphalt, es­pe­cially the posts Ads Don’t Work That Way and Doesn’t Mat­ter, Warm Fuzzies.

• orthonormal 4 Apr 2018 4:27 UTC

  14 points

  on: LessWrong Di­as­pora Jar­gon Survey

  If you do fu­ture sur­veys of this sort, I’d like you to ask peo­ple for their prob­a­bil­ities rather than just their best guesses. If peo­ple are un­cer­tain but de­cently cal­ibrated, I’d ar­gue there’s not much of a prob­lem; if peo­ple are con­fi­dently wrong, I’d ar­gue there’s a real prob­lem.

• orthonormal 4 Apr 2018 0:40 UTC

  10 points

  in reply to: Marcello's comment on: The Mean­ing of Right

  This com­ment got linked a decade later, and so I thought it’s worth stat­ing my own thoughts on the ques­tion:

  We can con­sider a refer­ence class of CEV-seek­ing pro­ce­dures; one (mas­sively-un­der­speci­fied, but that’s not the point) ex­am­ple is “em­u­late 1000 copies of Paul Chris­ti­ano liv­ing to­gether com­fortably and im­mor­tally and dis­cussing what the AI should do with the phys­i­cal uni­verse; once there’s a large su­per­ma­jor­ity in fa­vor of an en­actable plan (which can in­clude fur­ther such del­e­gated de­ci­sions), the AI does that”.

  I agree that this is go­ing to be chaotic, in the sense that even slightly differ­ent el­e­ments of this refer­ence class might end up steer­ing the AI to differ­ent bas­ins of at­trac­tion.

  I as­sert, how­ever, that I’d con­sider it a pretty good out­come over­all if the fu­ture of the world were de­ter­mined by a gen­uinely ran­dom draw from this refer­ence class, hon­estly in­stan­ti­ated. (Again with the mas­sive un­der­speci­fi­ca­tion, I know.)

  CEV may be un­der­de­ter­mined and many-val­ued, but that doesn’t mean pa­per­clip­ping is as good an an­swer as any.

  Re: no bas­ins, it would be a bad situ­a­tion in­deed if the vast ma­jor­ity of the refer­ence class never ended up out­putting an ac­tion plan, in­stead defer­ring and del­e­gat­ing for­ever. I don’t have cached thoughts about that.

• orthonormal 1 Apr 2018 20:41 UTC

  32 points

  on: An­nounc­ing: Karma 2.0

  I for one wel­come our new ty­po­graph­i­cal over­lords.

• orthonormal 31 Mar 2018 22:47 UTC

  10 points

  in reply to: Kaj_Sotala's comment on: A model I use when mak­ing plans to re­duce AI x-risk

  That’s a le­git thing to be frus­trated by, but I think you know the rea­son why AI safety re­searchers don’t want “we don’t see a way to get to a good out­come ex­cept for an al­igned pro­ject to grab a de­ci­sive strate­gic ad­van­tage” to filter into pub­lic dis­course: it pat­tern-matches too well to “trust us, you need to let us run the uni­verse”.

• orthonormal 31 Mar 2018 19:51 UTC

  10 points

  in reply to: Kaj_Sotala's comment on: A model I use when mak­ing plans to re­duce AI x-risk

  To be clear, I am mak­ing the claim that, of the peo­ple who have made use­ful ad­vances on Or­a­cle AI safety re­search (Arm­strong counts here; I don’t think Yam­polskiy does), all of them be­lieve that the goal of hav­ing a safe Or­a­cle AI is to achieve a de­ci­sive strate­gic ad­van­tage quickly and get to an al­igned fu­ture. I rec­og­nize that this is a hard claim to eval­u­ate (e.g. be­cause this isn’t a state­ment one could put in a Se­ri­ous Aca­demic Jour­nal Ar­ti­cle in the 2010s, it would have to be dis­cussed on their blog or in pri­vate cor­re­spon­dence), but if any­one has a clear coun­terex­am­ple, I’d be in­ter­ested in see­ing it.

• orthonormal 31 Mar 2018 19:36 UTC

  9 points

  in reply to: Zvi's comment on: Circling

  Yes, this. NVC should be treated with a similar sort of pa­ram­e­ters to Crocker’s Rules, which you can de­clare for your­self at any time, you can in­vite peo­ple to a con­ver­sa­tion where it’s known that ev­ery­one will be us­ing them, but you can­not hold it against any­one if you in­vite them to de­clare Crocker’s Rules and they re­fuse.

• orthonormal 31 Mar 2018 17:39 UTC

  10 points

  in reply to: c0rw1n's comment on: The abrupt­ness of nu­clear weapons

  There’s a lot of Ac­tu­ally Bad things an AI can do just by mak­ing elec­trons move.