Dagon

• Dagon 2 Sep 2019 17:36 UTC

  6 points

  on: The Value of Your Time, or Why You Should Buy an AC

  This is a good re­minder for those who haven’t thought of it or haven’t ex­am­ined their voices and habits for awhile. But https://​​www.less­wrong.com/​​posts/​​6NvbSwuSAooQxxf7f/​​be­ware-of-other-op­ti­miz­ing, and watch out for a ra­tio­nal­ist habit to over­weight the leg­ible (mea­surable, calcu­la­ble) por­tions of life choices and ig­nore the hid­den and hard-to-quan­tify.

  Con­sider that Philip may preFer not to have got­ten much done on the di­men­sions he can com­mu­ni­cate, and the heat gives him an easy jus­tifi­ca­tion. Or that the emo­tional cost of pick­ing and com­mit­ting to spend for the unit is higher than the benefit for a week or two.

• Dagon 29 Aug 2019 18:07 UTC

  2 points

  in reply to: dadadarren's comment on: Sleep­ing Beauty: Is Disagree­ment While Shar­ing All In­for­ma­tion Pos­si­ble in An­thropic Prob­lems?

  What re­ward (and more im­por­tantly, what util­ity) does the pre­dic­tor re­ceive/​lose for a cor­rect/​in­cor­rect guess?

  To the ex­tent that “you” care about your clones, you should guess in ways that max­i­mize the ag­gre­gate pay­out to all guessers. If you don’t, then guess to max­i­mize the guesser’s pay­out even at the ex­pense of clones (who will make the same guess, but be wrong more of­ten).

  Self-lo­cat­ing prob­a­bil­ities ex­ist only to the ex­tent that they in­fluence how much util­ity the cur­rent de­ci­sion-maker as­signs to ex­pe­riences of pos­si­bly-you en­tities.

• Dagon 28 Aug 2019 23:18 UTC

  3 points

  on: The Miss­ing Math of Map-Making

  I am still find­ing it difficult to un­der­stand how the fo­cus on causal­ity of map­mak­ing is more helpful than ex­am­in­ing the in­tent to sum­ma­rize (which en­com­passes which in­for­ma­tion gets thrown away based on what do­main of pre­dic­tion the map is cre­ated for) and the (pretty pure bayesean) ac­cu­racy of pre­dic­tions.

• Dagon 28 Aug 2019 17:57 UTC

  4 points

  in reply to: johnswentworth's comment on: Don’t Pull a Bro­ken Chain

  Note that map­ping is a type of ab­strac­tion that’s in­de­pen­dent of causal chains and feed­back/​con­trol loops. You can make an ex­cel­lent ther­mo­stat that doesn’t un­der­stand a thing about ther­mo­dy­nam­ics, air flow, or con­trol the­ory (though you may need to know some­thing about these things in or­der to make the ther­mo­stat work well).

• Dagon 27 Aug 2019 23:41 UTC

  2 points

  on: Car­to­graphic Processes

  Hmm. I might have a sense of where you’re go­ing, but the ter­minol­ogy is con­fus­ing to me. Noth­ing hap­pens spon­ta­neously, ev­ery fu­ture state hap­pens be­cause of the past state of the uni­verse, so your in­tro makes very lit­tle sense to me. I think the dis­tinc­tion you’re point­ing to isn’t spon­ta­neous/​caused, I think it’s nat­u­ral/​ar­tifi­cial, or maybe au­to­matic/​planned, or maybe in­evitable/​in­ten­tional. In any case, it seems to be about hu­man con­scious de­ci­sions to cre­ate the map. I’m not sure why this doesn’t ap­ply to the hu­man con­scious de­ci­sion to cre­ate the roads be­ing mapped, but I sus­pect there’s an el­e­ment of ob­jec­tive/​sub­jec­tive in there or full-fidelity/​sim­plified-model.

  I’m also un­sure if the “car­to­graphic pro­cess” is the hu­man in­tent to make a map/​model, or the phys­i­cal steps (mea­sure­ments, up­date of dis­play, etc.) that gen­er­ate the map.

• Dagon 27 Aug 2019 22:19 UTC

  2 points

  in reply to: TAG's comment on: Chris_Leong’s Shortform

  Not new that I could tell—it is a re­fresh­ing clar­ity for strict de­ter­minism—free will is an illu­sion, and “pos­si­ble” is in the map, not the ter­ri­tory. “De­cid­ing” is how a brain feels as it ex­e­cutes it’s al­gorithm and takes the pre­de­ter­mined (but not pre­vi­ously known) path.

  He does not re­solve the con­flict that it feels SOOO real as it hap­pens.

• Dagon 27 Aug 2019 21:24 UTC

  4 points

  in reply to: Said Achmiz's comment on: A Per­sonal Ra­tion­al­ity Wishlist

  Un­der­stand­ing how it works and re­mem­ber­ing de­tails when asked out of con­text may be very differ­ent things. I wish the par­ti­ci­pants had been given fol­low-up ques­tions about how it works, and then the ex­er­cises re­peated when a bi­cy­cle was pre­sent.

• Dagon 27 Aug 2019 18:55 UTC

  3 points

  in reply to: TAG's comment on: Chris_Leong’s Shortform

  The topic is in­ter­est­ing, but no dis­cus­sion about it is in­ter­est­ing. Th­ese are not con­tra­dic­tory.

  The open ques­tion about strong de­ter­minism vs liber­tar­ian free will is in­ter­est­ing, and there is a yet-un­ex­plained con­tra­dic­tion be­tween my felt ex­pe­rience (and oth­ers re­ported ex­pe­riences) and my fun­da­men­tal phys­i­cal model of the uni­verse. The fact that no­body has any al­ter­na­tive model or ev­i­dence (or even ideas about what ev­i­dence is pos­si­ble) that helps with this in­ter­est­ing ques­tion makes the dis­cus­sion un­in­ter­est­ing.

• Dagon 26 Aug 2019 19:09 UTC

  2 points

  in reply to: Stuart_Armstrong's comment on: Grat­ifi­ca­tion: a use­ful con­cept, maybe new

  True. There are two dis­tinc­tions (I think) you’re mak­ing from base util­i­tar­i­anism (prefer­ences over state of the uni­verse in terms of agent-ex­pe­rienced util­ity):

  1) This is about path, not state. You have an opinion about some­thing to do/​ex­pe­rience that’s in­de­pen­dent of any differ­ence in ex­pected value of a fu­ture state. It’s also (I think) ex­plic­itly in­dex­i­cal—you care that it’s you hav­ing this ex­pe­rience, not that it’s ex­pe­rienced by more peo­ple.

  2) This is about … some­thing … which isn’t on the pain/​plea­sure axis. I’m less sure of this one, as I tend to ex­pe­rience iden­tity-af­firm­ing things as some­what plea­surable and I’m not sure that’s any less com­pa­rable on this di­men­sion than any other plea­sure or per­sonal dis­ap­point­ment.

  The tor­ture ex­am­ple is similar on the first point, but misses the sec­ond. Is that roughly cor­rect?

• Dagon 26 Aug 2019 18:40 UTC

  4 points

  on: Grat­ifi­ca­tion: a use­ful con­cept, maybe new

  I’ve heard this referred to as “ex­pe­rience prefer­ence”, or some­times “ex­pe­ri­en­tial util­ity”, in that they are things you want to ex­pe­rience, dis­tinct from states you want the uni­verse to be in. (skip­ping the rab­bit hole of whether all ex­pe­rience is mem­ory or whether this is a prefer­ence for hav­ing a mem­ory vs hav­ing an ex­pe­rience).

  It oc­curs a lot in the nega­tive as well—things you don’t want to ex­pe­rience (or don’t want ANYONE to ex­pe­rience), re­gard­less of the state of the uni­verse af­ter­ward. Many tor­ture-trade­off dis­cus­sions hinge on this point—to a lot of peo­ple, suffer­ing is bad not be­cause of con­se­quences or be­cause is re­duces a he­do­nic sum, but is a di­men­sion of bad in it­self.

• Dagon 26 Aug 2019 18:33 UTC

  7 points

  on: Schel­ling Cat­e­gories, and Sim­ple Mem­ber­ship Tests

  I re­ally ap­pre­ci­ate the clar­ity of in­clud­ing both the math and the good ex­am­ples. It may be use­ful to ac­knowl­edge that many of the ob­ser­va­tion/​com­mu­ni­ca­tion limits are in fact con­tin­gent on some­thing un­stated. What is it that keeps you from ob­serv­ing an­other vari­able or draw­ing bet­ter-fit cat­e­gories?

  but that’s no rea­son to not as­pire to the true pre­ci­sion of the Bayes-struc­ture to what­ever ex­tent pos­si­ble

  Which brings us to the ac­tual hard ques­tion: to what ex­tent _is it_ pos­si­ble? A lot of the time, the right an­swer is to no­tice that the com­pro­mise that’s com­monly made is un­der­shoot­ing the pre­ci­sion that you could han­dle, and de­ter­mine which of your au­di­ence or con­ver­sa­tional part­ners are ahead of or be­hind you in your mod­el­ing of the ac­tual world. In your first ex­am­ple, you can ABSOLUTELY talk about fi­nan­cial eman­ci­pa­tion, con­tract en­force­ment, or other adult rights that can be ap­plied to some minors. Some things are taboo or sa­cred and you prob­a­bly can’t openly talk about them with­out severely vi­o­lat­ing norms, but you can still model the de­tails in­side your head to avoid truth-deflect­ing norms/​laws.

  Pick­ing the right level of de­tail for the per­son or group (or in­ter­nally, topic) you’re deal­ing with is, IMO, far more im­por­tant that pick­ing the ac­tual sharp lines to draw on a fuzzy can­vas.

• Dagon 25 Aug 2019 16:58 UTC

  3 points

  on: Am I go­ing for a job in­ter­view with a woo pusher?

  If you haven’t in­ter­viewed in awhile, no harm in prac­tic­ing on them. It does seem pretty woo-ey, but one can make a pretty strong ar­gu­ment that most pop­u­lar woo con­tains some ac­tu­ally helpful el­e­ments. I do worry that the job is ac­tu­ally a sales job in the guise of a tech­ni­cian (like it’s based on com­mis­sions or quo­tas for clients you bring in), and if that’s not what you want, you should be ex­tremely clear about it be­fore you ac­cept.

• Dagon 25 Aug 2019 16:27 UTC

  4 points

  in reply to: JakeArgent's comment on: Tor­ture and Dust Specks and Joy—Oh my! or: Non-Archimedean Utility Func­tions as Pseu­do­graded Vec­tor Spaces

  I like this in­sight—not only non­lin­ear but ac­tu­ally dis­con­tin­u­ous. There are some marginal in­stants of tor­ture that are hugely nega­tive, mixed in with those that are more mildly nega­tive. This is due to some­thing that’s of­ten for­got­ten in these dis­cus­sions: on­go­ing im­pact of a mo­men­tary ex­pe­rience.

  Be­ing “bro­ken” by tor­ture may make it im­pos­si­ble to ever re­cover enough for any fu­ture ex­pe­riences to be pos­i­tive. There may be a few quanta of bro­ken­ness, but it’s not the case that ev­ery marginal sec­ond is all that bad, only some of them.

• Dagon 23 Aug 2019 14:20 UTC

  2 points

  on: The “Com­mit­ment Races” problem

  I think you’re miss­ing at least one key el­e­ment in your model: un­cer­tainty about fu­ture pre­dic­tions. Com­mit­ments have a very high cost in terms of fu­ture con­se­quence-effect­ing de­ci­sion space. Con­se­quen­tial­ism does _not_ im­ply a very high dis­count rate, and we’re al­lowed to rec­og­nize the limits of our pre­dic­tion and to give up some power in the short term to re­serve our flex­i­bil­ity for the fu­ture.

  Also, one of the rea­sons that this kind of in­ter­ac­tion is rare among hu­mans is that com­mit­ment is im­pos­si­ble for hu­mans. We can change our minds even af­ter mak­ing an oath—of­ten with some rep­u­ta­tional con­se­quences, but still pos­si­ble if we deem it worth­while. Even so, we’re rightly re­luc­tant to make se­ri­ous com­mitt­ments. An agent who can ac­tu­ally en­force it’s self-limi­ta­tions is go­ing to be or­ders of mag­ni­tude more hes­i­tant to do so.

  All that said, it’s worth rec­og­niz­ing that an agent that’s sig­nifi­cantly bet­ter at pre­dict­ing the con­se­quences of po­ten­tial com­mit­ments will pay a lower cost for the best of them, and has a ma­te­rial ad­van­tage over those who need flex­i­bil­ity be­cause they don’t have in­for­ma­tion. This isn’t a race in time, it’s a race in knowl­edge and un­der­stand­ing. I don’t think there’s any way out of that race—more pow­er­ful agents are go­ing to beat weaker ones most of the time.

• Dagon 22 Aug 2019 23:01 UTC

  2 points

  on: Time Travel, AI and Trans­par­ent Newcomb

  Let us sup­pose <im­pos­si­ble thing>. Now <im­pos­si­ble re­sult> re­mains im­pos­si­ble, how? Maybe the uni­verse has a mys­te­ri­ous agency we can trick or bar­gain with!

  I think you’ll need to back up a bit fur­ther if you want to ex­plore this. “time travel is pos­si­ble” isn’t well enough defined to be able to rea­son about, ex­cept in the hu­man con­cep­tual space with no physics at­tached. And if you’re as­sum­ing away physics, you don’t need to ex­plain any­thing, just let the para­doxes hap­pen.

• Dagon 20 Aug 2019 23:51 UTC

  3 points

  in reply to: Davis_Kingsley's comment on: Davis_Kingsley’s Shortform

  I can’t tell if this is just an­other ex­am­ple that strate­gic choices tend to be valuable (guaran­teed non-nega­tive, but in prac­tice usu­ally pos­i­tive). OF COURSE an op­po­nent’s choice is go­ing to re­duce your value in a zero-sum game.

  I do want to warn against ap­ply­ing to other as­pects of life that aren’t purely zero-sum and aren’t de­signed by a hu­man to bal­ance the power be­tween both par­ties. See also https://​​www.less­wrong.com/​​posts/​​rHBd­cHGLJ7KvLJQPk/​​the-log­i­cal-fal­lacy-of-gen­er­al­iza­tion-from-fic­tional

• Dagon 20 Aug 2019 23:24 UTC

  4 points

  in reply to: steve2152's comment on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  I don’t un­der­stand why https://​​en.wikipe­dia.org/​​wiki/​​The­ory_of_the_sec­ond_best doesn’t get more con­sid­er­a­tion. In a com­plex in­ter­con­nected sys­tem, V can not only be much less than E, it can be much less than would be ob­tained with ~C. You may not get mere utopia, you may get se­ri­ous dystopia.

• Dagon 20 Aug 2019 23:03 UTC

  8 points

  on: What are the rea­sons to *not* con­sider re­duc­ing AI-Xrisk the high­est pri­or­ity cause?

  Other rea­sons that peo­ple may have (I have some of these rea­sons, but not all):

  • not a clas­si­cal utilitarian

  • don’t be­lieve those timelines

  • too dis­tant to feel an emo­tional tie to

  • un­clear what to do even if it is a priority

  • very high dis­count rate for fu­ture humans

  • be­lief that moral value is rel­a­tive with cog­ni­tive abil­ity (an ex­tremely smart AI may be worth a few quitil­lion hu­mans in a moral/​ex­pe­ri­en­tial sense)

  Of these, I think the one that I’m per­son­ally least moved by while ac­knowl­eg­ing it as one of the bet­ter ar­gu­ments against util­i­tar­i­anism is the last. It’s clear that there’s SOME differ­ence in moral weight for differ­ent ex­pe­riences of differ­ent ex­pe­riencers. Which means there’s some di­men­sion on which a util­ity mon­ster is con­ceiv­able. If it’s a di­men­sion that AGI will ex­cel on, we can max­i­mize util­ity by giv­ing it what­ever it wants.

• Dagon 20 Aug 2019 17:08 UTC

  2 points

  in reply to: Swimmer963's comment on: Swim­mer963′s Shortform

  (don’t write fic­tion, but have run and playtested a lot of RPGs, which share many of the wor­ld­build­ing el­e­ments).

  Among the hard parts is figur­ing out how much sus­pen­sion of dis­be­lief your au­di­ence will will­ingly bring, on what top­ics. This _is_ fic­tion, so we’re not gen­er­ally try­ing to truly pre­dict a hy­po­thet­i­cal “pos­si­ble” out­come, we’re try­ing to high­light similar­i­ties and differ­ences from our own. This VERY OFTEN im­plies as­sum­ing a similar­ity (where the point of de­par­ture has less effect that is likely) and then jus­tify­ing it or con­strain­ing the de­par­ture so it’s less difficult to main­tain that this el­e­ment of so­ciety would still be rec­og­niz­able.

• Dagon 20 Aug 2019 16:14 UTC

  2 points

  on: Nega­tive “eeny meeny miny moe”

  Ehn. For kids who will EVER ac­cept this as fair, you’re putting too much thought into poli­tics. If the kids are this ma­nipu­la­ble, they’ll prob­a­bly ac­cept your au­thor­ity in the one-shot case as well.

  Also, more iter­a­tions gives them more time to re­al­ize that you’re cheat­ing (by shifts in how to count syl­la­bles) or that the game is fully de­ter­minis­tic (and you’re cheat­ing by de­cid­ing who to start with).

  This is only us­able for such low-stakes cases where the par­ti­ci­pants don’t mind that it’s not fair. And in those cases, don’t waste time on pointless com­plex­ity. Of course, if this is part of the en­ter­tain­ment, I re­verse that ad­vice—choose the sin­gle-elimi­na­tion method to ex­tend the du­ra­tion of the ten­sion of not know­ing.