Charlie Steiner

• Charlie Steiner 24 Jun 2019 6:06 UTC

  9 points

  AF

  on: Model­ing AGI Safety Frame­works with Causal In­fluence Diagrams

  The rea­son I don’t per­son­ally find these kinds of rep­re­sen­ta­tion su­per use­ful is be­cause each of those boxes is a quite com­pli­cated func­tion, and what’s in the boxes usu­ally in­volves many more bits worth of in­for­ma­tion about an AI sys­tem than how the boxes are con­nected. And some­times one makes differ­ent choices in how to chop an AI’s op­er­a­tion up into causally linked boxes, which can lead to an ap­ples-and-or­anges prob­lem when com­par­ing di­a­grams (for ex­am­ple, the di­a­grams you use for CIRL and IDI are very differ­ent chop­pings-up of the al­gorithms).

  I ac­tu­ally have a draft sit­ting around of how one might rep­re­sent value learn­ing schemes with a hi­er­ar­chi­cal di­a­gram of in­for­ma­tion flow. Even­tu­ally I de­cided that the idea made lots of sense for a few paradigm cases and was more trou­ble than it was worth for ev­ery­thing else. When you need to care­fully re­fer to the text de­scrip­tion to un­der­stand a di­a­gram, that’s a sign that maybe you should use the text de­scrip­tion.

  This isn’t to say I think one should never see any­thing like this. Differ­ent ways of pre­sent­ing the same in­for­ma­tion (like di­a­grams) can help drive home a par­tic­u­larly im­por­tant point. But I am skep­ti­cal that there’s a one-size-fits-all solu­tion, and in­stead think that di­a­gram us­age should be tai­lored to the par­tic­u­lar point it’s in­tended to make.

• Charlie Steiner 21 Jun 2019 14:27 UTC

  5 points

  on: Is your un­cer­tainty re­solv­able?

  I think I must be the odd one out here in terms of com­fort us­ing prob­a­bil­ities close to 1 and 0. Be­cause 90% and 99% are not “near cer­tainty” to me.

  How sure are you that the English guy who you’ve been told helped in­vent calcu­lus and did stuff with grav­ity and op­tics was called “Isaac New­ton”? We’re talk­ing about prob­a­bil­ities like 99.99999% here. (Con­di­tion­ing on no dumb gotchas from hu­man com­mu­ni­ca­tion, e.g. me us­ing a uni­code char­ac­ter from a differ­ent lan­guage and claiming it’s no longer the same, which has sud­denly be­come much more salient to you and me both. An “in­ter­nal” prob­a­bil­ity, if you will.)

  Maybe it would help to think of this as about 20 bits of in­for­ma­tion past 50%? Every bit of in­for­ma­tion you can spec­ify about some­thing means you are as­sign­ing a more ex­treme prob­a­bil­ity dis­tri­bu­tion about that thing. The prob­a­bil­ity of the an­swer be­ing “Isaac New­ton” has a very tiny prior for any given ques­tion, and only rises to 50% af­ter lots of bits of in­for­ma­tion. And if you could get to 50%, it’s not strange that you could have quite a few more bits left over, be­fore even­tu­ally run­ning into the limits set by the re­li­a­bil­ity of your own brain.

  So when you say some plans re­quire near cer­tainty, I’m not sure if you mean what I mean but chose smaller prob­a­bil­ities, or if you mean some some­what differ­ent point about so­cial norms about when num­bers are big/​small enough that we are al­lowed to stop/​start wor­ry­ing about them. Or maybe you mean a third thing about leg­i­bil­ity and com­mu­ni­ca­bil­ity that is cor­re­lated with prob­a­bil­ity but not iden­ti­cal?

• Charlie Steiner 19 Jun 2019 21:14 UTC

  4 points

  in reply to: TAG's comment on: Is “phys­i­cal non­de­ter­minism” a mean­ingful con­cept?

  “At the most ba­sic level of de­scrip­tion, things—the quan­tum fields or branes or what­ever—just do what they do. They don’t do it non­de­ter­minis­ti­cally, but they also don’t do it de­ter­minis­ti­cally”

  How do you know? If that claim isn’t based on a model, what is it based on?

  I’m happy to re­ply that the mes­sage of my com­ment as a whole ap­plies to this part—my claim about “what things do at a ba­sic level of de­scrip­tion” is a meta-model claim about what you can say about things at differ­ent lev­els of de­scrip­tion.

  It’s hu­man na­ture to in­ter­pret this as a claim that things have some essence of “just do­ing what they do” that is a com­peti­tor to the essences of de­ter­minism and non­de­ter­minism, but there is no such essence for the same rea­sons I’m already talk­ing about in the com­ment. Maybe I could have worded it more care­fully to pre­vent this read­ing, but I figure that would sac­ri­fice more clar­ity than it gained.

  The point is not about some “ba­sic na­ture of things,” the point is about some “ba­sic level of de­scrip­tion.” We might imag­ine some­one say­ing “I know there are some de­ter­minis­tic mod­els of atoms and some non­de­ter­minis­tic mod­els, but are the atoms re­ally de­ter­minis­tic or not?” Where this “re­ally” seems to mean some athe­o­retic di­rect un­der­stand­ing of the na­ture of atoms. My point, in short, is that athe­o­retic un­der­stand­ing is fruitless (“It’s just one damn thing af­ter an­other”) and the in­stinct that says it’s de­sir­able is mis­lead­ing.

  “much bet­ter ex­pla­na­tion of wet­ness would be in terms of sur­face ten­sion and in­ter­molec­u­lar forces and so on”

  Why? Be­cause they are real prop­er­ties?

  Be­cause they’re part of a de­tailed model of the world that helps tell a “func­tional and causal story” about the phe­nomenon. If I was go­ing to bad­mouth one set of essences just to prop up an­other, I would have said so :P

  You can ex­plain away some prop­er­ties in terms of oth­ers, but there is, go­ing to be some resi­due

  My point is that this resi­due is never go­ing to be the “Real Prop­er­ties,” they’re just go­ing to be the same the­ory-laden prop­er­ties as always.

  What makes a the­ory of ev­ery­thing a the­ory of ev­ery­thing is not that it pro­vides a fi­nal an­swer for which prop­er­ties are the real prop­er­ties that atoms have in some athe­o­retic di­rect way. It’s that it pro­vides a use­ful frame­work in which we can un­der­stand all (liter­ally all) sorts of stuff.

• Charlie Steiner 19 Jun 2019 20:47 UTC

  4 points

  AF

  in reply to: Stuart_Armstrong's comment on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  Now that I think about it, it’s a pretty big PR prob­lem if I have to start ev­ery ex­pla­na­tion of my value learn­ing scheme with “hu­mans don’t have ac­tual prefer­ences so the AI is just go­ing to try to learn some­thing ad­e­quate.” Maybe I should figure out a sys­tem of jar­gon such that I can say, in jar­gon, that the AI is learn­ing peo­ples’ ac­tual prefer­ences, and it will cor­re­spond to what laypeo­ple ac­tu­ally want from value learn­ing.

  I’m not sure whether such jar­gon would make ac­tual tech­ni­cal think­ing harder, though.

• Charlie Steiner 18 Jun 2019 11:01 UTC

  9 points

  AF

  on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  This is re­ally similar to some stuff I’ve been think­ing about, so I’ll be writ­ing up a longer com­ment with more com­pare/​con­trast later.

  But one thing re­ally stood out to me—I think one can go farther in grap­pling with and tak­ing ad­van­tage of “where $U_H$ lives.” $U_H$ doesn’t live in­side the hu­man, it lives in the AI’s model of the hu­man. Hu­mans aren’t ideal­ized agents, they’re clusters of atoms, which means they don’t have prefer­ences ex­cept af­ter the sort of coarse-grain­ing pro­ce­dure you de­scribe, and this coarse-grain­ing pro­ce­dure lives with a par­tic­u­lar model of the hu­man—it’s not in­her­ent in the atoms.

  This means that once you’ve speci­fied a value learn­ing pro­ce­dure and hu­man model, there is no resi­d­ual “ac­tual prefer­ences” the AI can check it­self against. The challenge was never to ac­cess our “ac­tual prefer­ences,” it was always to make a best effort to model hu­mans as they want to be mod­eled. This is deeply coun­ter­in­tu­itive (“What do you mean, the AI isn’t go­ing to learn what hu­mans’ ac­tual prefer­ences are?!”), but also liber­at­ing and mo­ti­vat­ing.

• Charlie Steiner 16 Jun 2019 21:31 UTC

  6 points

  on: Is “phys­i­cal non­de­ter­minism” a mean­ingful con­cept?

  In­ter­est­ing ques­tion! My an­swer is ba­si­cally a long warn­ing about es­sen­tial­ism: this ques­tion might seem like it’s step­ping down from the realm of hu­man mod­els to the realm of ac­tual things, to ask about the essence of those things. But I think bet­ter an­swers are go­ing to come from step­ping up from the realm of mod­els to the realm of meta-mod­els, to ask about the prop­er­ties of mod­els.

  At the most ba­sic level of de­scrip­tion, things—the quan­tum fields or branes or what­ever—just do what they do. They don’t do it non­de­ter­minis­ti­cally, but they also don’t do it de­ter­minis­ti­cally! Without re­course to hu­man mod­els, all us hu­mans can say is that the things just do what they do—mod­els are the things that make talk about cat­e­gories pos­si­ble in the first place.

  Any an­swer of the sort “no, things can always be ren­dered into a de­ter­minis­tic form by treat­ing ‘ran­dom’ re­sults as fixed con­stants” or “yes, there are perfectly valid classes of mod­els that in­clude non­de­ter­minism” is go­ing to be an an­swer about mod­els, within some meta-level frame­work. And that’s fine!

  This can seem un­satis­fy­ing be­cause it goes against our es­sen­tial­ist in­stinct—that the prop­er­ties in our mod­els should re­flect the real prop­er­ties that things have. If wa­ter is con­sid­ered wet, it’s be­cause wa­ter has the ba­sic prop­erty or essence of wet­ness (so the in­stinct goes).

  Note that this doesn’t ex­plain any of the me­chan­ics or physics of wet­ness. If you could look in­side some­one’s head as they were perform­ing this es­sen­tial­ist ma­neu­ver, they would start with a model (“wa­ter is wet”), then they would no­tice their model (“I model wa­ter as wet”), then they would jus­tify them­selves to them­selves, in a sort of re­as­sur­ing pat on the back (“I model wa­ter as wet be­cause wa­ter is re­ally, spe­cially wet”).

  I think that this line of self-re­as­sur­ing rea­son­ing is flawed, and a much bet­ter ex­pla­na­tion of wet­ness would be in terms of sur­face ten­sion and in­ter­molec­u­lar forces and so on—illu­mi­nat­ing the func­tional and causal story be­hind our model of the world, rather than be­liev­ing you’ve ex­plained wet­ness in terms of “real wet­ness”. Also see the story about Bleggs and Rubes.

  Long story short, any good ex­pla­na­tion for why we should or shoudn’t have non­de­ter­minism in a model is ei­ther go­ing to be about how to choose good mod­els, or it’s go­ing to be a causal and func­tional story that doesn’t pre­serve non­de­ter­minism (or de­ter­minism) as an essence.

  I think there’s an in­ter­est­ing ques­tion about physics in whether or not (and how) we should in­clude non­de­ter­minism as an op­tion in fun­da­men­tal the­o­ries. But first I just wanted to warn that the ques­tion “mod­els aside, are things re­ally non­de­ter­minis­tic” is not go­ing to have an in­ter­est­ing an­swer.

• Charlie Steiner 13 Jun 2019 1:17 UTC

  6 points

  on: What kind of thing is logic in an on­tolog­i­cal sense?

  https://​​www.less­wrong.com/​​posts/​​mHNzpX38HkZQrdYn3/​​philos­o­phy-of-num­bers-part-2

  Ba­si­cally, we have a men­tal model of logic the same way we have a men­tal model of ge­og­ra­phy. It’s use­ful to say that log­i­cal facts have refer­ents for the same in­ter­nal rea­son it’s use­ful to say that ge­o­graph­i­cal facts have refer­ents. But if you looked at a hu­man from out­side, the causal story be­hind log­i­cal facts vs. ge­o­graph­i­cal facts would be differ­ent.

• Charlie Steiner 10 Jun 2019 20:38 UTC

  10 points

  on: Logic, Bud­dhism, and the Dialetheia

  As a prac­tic­ing quan­tum me­chanic, I’d warn you against the claim that di­alethe­ism is used in quan­tum com­put­ers. Qu­bits are some­times de­scribed as tak­ing “both states at the same time,” but that’s not pre­cisely what’s go­ing on, and peo­ple who ac­tu­ally work on quan­tum com­put­ers use a more pre­cise un­der­stand­ing that doesn’t in­volve in­ter­pret­ing in­ter­me­di­ate qubits as truth val­ues.

  There are also two peo­ple I wanted to see in your post: Rus­sell and Gödel—math­e­mat­i­ci­ans rather than philoso­phers. Rus­sell’s type the­ory was one of the main at­tempts to elimi­nate para­doxes in math­e­mat­i­cal logic. Gödel showed how that doesn’t quite work, but also showed how things be­come a lot clearer if you con­sider prov­abil­ity as well as truth value.

• Charlie Steiner 10 Jun 2019 16:21 UTC

  19 points

  in reply to: c0rw1n's comment on: Learn­ing magic

  On Wed­nes­days at the Prince­ton Grad­u­ate Col­lege, var­i­ous peo­ple would come in to give talks. The speak­ers were of­ten in­ter­est­ing, and in the dis­cus­sions af­ter the talks we used to have a lot of fun. For in­stance, one guy in our school was very strongly anti-Catholic, so he passed out ques­tions in ad­vance for peo­ple to ask a re­li­gious speaker, and we gave the speaker a hard time.

  Another time some­body gave a talk about po­etry. He talked about the struc­ture of the poem and the emo­tions that come with it; he di­vided ev­ery­thing up into cer­tain kinds of classes. In the dis­cus­sion that came af­ter­wards, he said, “Isn’t that the same as in math­e­mat­ics, Dr. Eisen­hart?”

  Dr. Eisen­hart was the dean of the grad­u­ate school and a great pro­fes­sor of math­e­mat­ics. He was also very clever. He said, “I’d like to know what Dick Feyn­man thinks about it in refer­ence to the­o­ret­i­cal physics.” He was always putting me on in this kind of situ­a­tion.

  I got up and said, “Yes, it’s very closely re­lated. In the­o­ret­i­cal physics, the ana­log of the word is the math­e­mat­i­cal for­mula, the ana­log of the struc­ture of the poem is the in­ter­re­la­tion­ship of the the­o­ret­i­cal bling-bling with the so-and so”—and I went through the whole thing, mak­ing a perfect anal­ogy. The speaker’s eyes were beam­ing with hap­piness.

  Then I said, “It seems to me that no mat­ter what you say about po­etry, I could find a way of mak­ing up an ana­log with any sub­ject, just as I did for the­o­ret­i­cal physics. I don’t con­sider such analogs mean­ingful.”

  In the great big din­ing hall with stained-glass win­dows, where we always ate, in our steadily de­te­ri­o­rat­ing aca­demic gowns, Dean Eisen­hart would be­gin each din­ner by say­ing grace in Latin. After din­ner he would of­ten get up and make some an­nounce­ments. One night Dr. Eisen­hart got up and said, “Two weeks from now, a pro­fes­sor of psy­chol­ogy is com­ing to give a talk about hyp­no­sis. Now, this pro­fes­sor thought it would be much bet­ter if we had a real demon­stra­tion of hyp­no­sis in­stead of just talk­ing about it. There­fore he would like some peo­ple to vol­un­teer to be hyp­no­tized.

  I get all ex­cited: There’s no ques­tion but that I’ve got to find out about hyp­no­sis. This is go­ing to be ter­rific!

  Dean Eisen­hart went on to say that it would be good if three or four peo­ple would vol­un­teer so that the hyp­no­tist could try them out first to see which ones would be able to be hyp­no­tized, so he’d like to urge very much that we ap­ply for this. (He’s wast­ing all this time, for God’s sake!)

  Eisen­hart was down at one end of the hall, and I was way down at the other end, in the back. There were hun­dreds of guys there. I knew that ev­ery­body was go­ing to want to do this, and I was ter­rified that he wouldn’t see me be­cause I was so far back. I just had to get in on this demon­stra­tion!

  Fi­nally Eisen­hart said, “And so I would like to ask if there are go­ing to be any vol­un­teers …”

  I raised my hand and shot out of my seat, scream­ing as loud as I could, to make sure that he would hear me: “MEEEEEEEEEEE!”

  He heard me all right, be­cause there wasn’t an­other soul. My voice re­ver­ber­ated through­out the hall—it was very em­bar­rass­ing. Eisen­hart’s im­me­di­ate re­ac­tion was, “Yes, of course, I knew you would vol­un­teer, Mr. Feyn­man, but I was won­der­ing if there would be any­body else.”

  Fi­nally a few other guys vol­un­teered, and a week be­fore the demon­stra­tion the man came to prac­tice on us, to see if any of us would be good for hyp­no­sis. I knew about the phe­nomenon, but I didn’t know what it was like to be hyp­no­tized.

  He started to work on me and soon I got into a po­si­tion where he said, “You can’t open your eyes.”

  I said to my­self, “I bet I could open my eyes, but I don’t want to dis­turb the situ­a­tion: Let’s see how much fur­ther it goes.” It was an in­ter­est­ing situ­a­tion: You’re only slightly fogged out, and al­though you’ve lost a lit­tle bit, you’re pretty sure you could open your eyes. But of course, you’re not open­ing your eyes, so in a sense you can’t do it.

  He went through a lot of stuff and de­cided that I was pretty good.

  When the real demon­stra­tion came he had us walk on stage, and he hyp­no­tized us in front of the whole Prince­ton Grad­u­ate Col­lege. This time the effect was stronger; I guess I had learned how to be­come hyp­no­tized. The hyp­no­tist made var­i­ous demon­stra­tions, hav­ing me do things that I couldn’t nor­mally do, and at the end he said that af­ter I came out of hyp­no­sis, in­stead of re­turn­ing to my seat di­rectly, which was the nat­u­ral way to go, I would walk all the way around the room and go to my seat from the back.

  All through the demon­stra­tion I was vaguely aware of what was go­ing on, and co­op­er­at­ing with the things the hyp­no­tist said, but this time I de­cided, “Damn it, enough is enough! I’m gonna go straight to my seat.”

  When it was time to get up and go off the stage, I started to walk straight to my seat. But then an an­noy­ing feel­ing came over me: I felt so un­com­fortable that I couldn’t con­tinue. I walked all the way around the hall.

  I was hyp­no­tized in an­other situ­a­tion some time later by a woman. While I was hyp­no­tized she said, “I’m go­ing to light a match, blow it out, and im­me­di­ately touch the back of your hand with it. You will feel no pain.”

  I thought, “Baloney!” She took a match, lit it, blew it out, and touched it to the back of my hand. It felt slightly warm. My eyes were closed through­out all of this, but I was think­ing, “That’s easy. She lit one match, but touched a differ­ent match to my hand. There’s nothin’ to that; it’s a fake!”

  When I came out of the hyp­no­sis and looked at the back of my hand, I got the biggest sur­prise: There was a burn on the back of my hand. Soon a blister grew, and it never hurt at all, even when it broke.

  So I found hyp­no­sis to be a very in­ter­est­ing ex­pe­rience. All the time you’re say­ing to your­self, “I could do that, but I won’t”—which is just an­other way of say­ing that you can’t.

  • Surely You Must Be Jok­ing Mr Feynman

• Charlie Steiner 10 Jun 2019 16:04 UTC

  3 points

  on: Refer­ences that treat hu­man val­ues as units of se­lec­tion?

  You can’t talk sen­si­bly about what val­ues are right, or what we ‘should’ build into in­tel­li­gent agents.

  I agree that in our usual use of the word, it doesn’t make sense to talk about what (ter­mi­nal) val­ues are right.

  But you agree that (within a cer­tain level of ab­strac­tion and im­plied con­text) you can talk as if you should take cer­tain ac­tions? Like “you should try this dessert” is a sen­si­ble English sen­tence. So what about ac­tions that im­pact in­tel­li­gent agents?

  Like, sup­pose there was a pill you could take that would make you want to kill your fam­ily. Should you take it? No, prob­a­bly not. But now we’ve just ex­pressed a prefer­ence about the val­ues of an in­tel­li­gent agent (your­self).

  Mod­ify­ing your­self to want bad things is wrong in the same sense that the bad things are wrong in the first place: they are wrong with re­spect to your cur­rent val­ues, which are a thing we model you as hav­ing within a cer­tain level of ab­strac­tion.

• Charlie Steiner 7 Jun 2019 12:37 UTC

  4 points

  in reply to: Dagon's comment on: Ma­jor Up­date on Cost Disease

  We have to sep­a­rate col­leges from pub­lic K-12 ed­u­ca­tion. Col­leges are the place where you hear about in­creas­ing num­bers of non-teach­ing staff. K-12 ac­tu­ally has fewer ad­minis­tra­tors per stu­dent than 20 years ago (in most places).

• Charlie Steiner 5 Jun 2019 11:53 UTC

  9 points

  in reply to: shminux's comment on: How is Solomonoff in­duc­tion calcu­lated in prac­tice?

  Min­i­mum mes­sage length fit­ting uses an ap­prox­i­ma­tion of K-com­plex­ity and gets used some­times when peo­ple want to fit weird curves in a sort of prin­ci­pled way. But “real” Solomonoff in­duc­tion is about feed­ing liter­ally all of your sen­sory data into the al­gorithm to get pre­dic­tions for the fu­ture, not just fit­ting curves.

  So I guess I’d say that it’s pos­si­ble to ap­prox­i­mate K-com­plex­ity and use that in your prior for curve fit­ting, and peo­ple some­times do that. But that’s not nec­es­sar­ily go­ing to be your best es­ti­mate, be­cause your best es­ti­mate is go­ing to take into ac­count all of the data you’ve already seen, which be­comes im­pos­si­ble very quickly (even if you just want a con­trol­led ap­prox­i­ma­tion).

• Charlie Steiner 5 Jun 2019 11:44 UTC

  3 points

  in reply to: Pattern's comment on: How is Solomonoff in­duc­tion calcu­lated in prac­tice?

  Sure. The OP might more ac­cu­rately have asked “How is the Solomonoff prior calcu­lated?”

• Charlie Steiner 3 Jun 2019 6:29 UTC

  2 points

  in reply to: shminux's comment on: Selec­tion vs Control

  Yeah I think this is definitely a “stance” thing.

  Take the use of nat­u­ral se­lec­tion and hu­mans as ex­am­ples of op­ti­miza­tion and mesa-op­ti­miza­tion—the en­tire con­cept of “nat­u­ral se­lec­tion” is a hu­man-con­ve­nient way of de­scribing a pat­tern in the uni­verse. It’s ap­prox­i­mately an op­ti­mizer, but in or­der to get rid of that “ap­prox­i­mately” you have to rein­tro­duce epicy­cles un­til your model is as com­pli­cated as a model of the world again. Hu­mans aren’t op­ti­miz­ers ei­ther, that’s just a hu­man-con­ve­nient way of de­scribing hu­mans.

  More ab­stractly, the en­tire pro­cess of rec­og­niz­ing a mesa-op­ti­mizer—some­thing that mod­els the world and makes plans—is an act of stance-tak­ing. Or Quinean rad­i­cal trans­la­tion or what­ever. If a cat-rec­og­niz­ing neu­ral net learns an at­ten­tion mechanism that mod­els the world of cats and makes plans, it’s not go­ing to come with lit­tle la­bels on the neu­rons say­ing “these are my in­put-out­put in­ter­faces, this is my model of the world, this is my plan­ning al­gorithm.” It’s go­ing to be some in­scrutable lit­tle bit of lin­ear alge­bra with sus­pi­ciously com­pe­tent be­hav­ior.

  Not only could this com­pe­tent be­hav­ior be ex­plained ei­ther by op­ti­miza­tion or some va­ri­ety of “rote be­hav­ior,” but the neu­rons don’t care about these bound­aries and can oc­cupy a con­tinuum of pos­si­bil­ities be­tween any two cen­tral ex­am­ples. And worst of all, the same neu­rons might have mul­ti­ple differ­ent use­ful ways of think­ing about them, some of which are in terms of el­e­ments like “goals” and “search,” and oth­ers are in terms of the el­e­ments of rote be­hav­ior.

  In light of this, the prob­lem of mesa-op­ti­miz­ers is not “when will this bright line be crossed?” but “when will this sim­ple model of the AI’s be­hav­ior be pre­dictable use­ful?” Even though I think the first in­stinct is the op­po­site.

• Charlie Steiner 1 Jun 2019 9:21 UTC

  5 points

  AF

  on: Uncer­tainty ver­sus fuzzi­ness ver­sus ex­trap­o­la­tion desiderata

  Nice post. I sus­pect you’ll still have to keep em­pha­siz­ing that fuzzi­ness can’t play the role of un­cer­tainty in a hu­man-mod­el­ing scheme (like CIRL), and is in­stead a way of re­solv­ing hu­man be­hav­ior into a util­ity func­tion frame­work. As­sum­ing I read you cor­rectly.

  I think that there are some un­spo­ken com­mit­ments that the frame­work of fuzzi­ness makes for how to han­dle ex­trap­o­lat­ing ir­ra­tional hu­man be­hav­ior. If you rep­re­sent fuzzi­ness as a weight­ing over util­ity func­tions that gets ag­gre­gated lin­early (i.e. into an­other util­ity func­tion), this is use­ful for the AI mak­ing de­ci­sions but can’t be the same thing that you’re us­ing to model hu­man be­hav­ior, be­cause hu­mans are go­ing to take ac­tions that shouldn’t be mod­eled as util­ity max­i­miza­tion.

  To bridge this gap from hu­man be­hav­ior to util­ity func­tion, what I’m in­ter­pret­ing you as im­ply­ing is that you should rep­re­sent hu­man be­hav­ior in terms of a patch­work of util­ity func­tions. In the post you talk about fre­quen­cies in a simu­la­tion, where small per­tur­ba­tions might lead a hu­man to care about the to­tal or about the av­er­age. Rather than the AI cre­at­ing a con­text-de­pen­dent model of the hu­man, we’ve some­how taught it (this part might be non-ob­vi­ous) that these small per­tur­ba­tions don’t mat­ter, and should be “fuzzed over” to get a util­ity func­tion that’s a weighted com­bi­na­tion of the ones ex­hibited by the hu­man.

  But we could also imag­ine un­rol­ling this as a fre­quency over time, where an ir­ra­tional hu­man some­times takes the ac­tion that’s best for the to­tal and other times takes the ac­tion that’s best for the av­er­age. Should a fuzzy-val­ues AI rep­re­sent this as the hu­man act­ing ac­cord­ing to differ­ent util­ity func­tions at differ­ent times, and then fuzzing over those util­ity func­tions to de­cide what is best?

• Charlie Steiner 29 May 2019 9:35 UTC

  7 points

  AF

  on: A shift in ar­gu­ments for AI risk

  An al­ter­nate fram­ing could be about chang­ing group bound­aries rather than chang­ing de­mo­graph­ics in an iso­lated group.

  There were surely peo­ple in 2010 who thought that the main risk from AI was it be­ing used by bad peo­ple. The differ­ence might not be that these peo­ple have popped into ex­is­tence or only re­cently started talk­ing—it’s that they’re in­side the fence more than be­fore.

  And of course, re­al­ity is always com­pli­cated. One of the con­cerns in the “early LW” genre is value sta­bil­ity and self-trust un­der self-mod­ifi­ca­tion, which has noth­ing to do with sud­den growth. And one of the “re­cent” genre con­cerns is arms races, which are pred­i­cated on peo­ple ex­pect­ing sud­den ca­pa­bil­ity growth to give them a first mover ad­van­tage.

• Charlie Steiner 25 May 2019 4:41 UTC

  2 points

  on: Why the em­piri­cal re­sults of the Trav­el­ler’s Dilemma de­vi­ate strongly away from the Nash Equil­ibrium and seems to be close to the so­cial op­ti­mum?

  I would guess that peo­ple don’t ac­tu­ally com­pute the Nash equil­ibrium or ex­pect other peo­ple to.

  In­stead, they use the same heuris­tic rea­son­ing meth­ods that they evolved to learn, and which have served them well in so­cial situ­a­tions their en­tire life, and ex­pect other peo­ple to do the same.

  I think we should ex­pect these heuris­tics to be close to ra­tio­nal (not for the util­ities of hu­mans, but for the fit­ness of genes) in the an­ces­tral en­vi­ron­ment. But there’s no par­tic­u­lar rea­son to think they’re go­ing to be ra­tio­nal by any stan­dard in games cho­sen speci­fi­cally be­cause the Nash equil­ibrium is coun­ter­in­tu­itive to hu­mans.

• Charlie Steiner 23 May 2019 22:20 UTC

  4 points

  in reply to: Yuxi_Liu's comment on: Free will as an ap­pear­ance to others

  If I may toot my own horn: https://​​www.less­wrong.com/​​posts/​​yex7E6oHXYL93Evq6/​​book-re­view-con­scious­ness-explained

  I’ll ad­mit I’m not to­tally sure what Said Ach­miz means by his com­par­i­son, though :)

• Charlie Steiner 23 May 2019 8:48 UTC

  5 points

  in reply to: shminux's comment on: Does the Higgs-bo­son ex­ist?

  Sure, but he also says “Since my ex­pec­ta­tions some­times con­flict with my sub­se­quent ex­pe­riences, I need differ­ent names for the thin­gies that de­ter­mine my ex­per­i­men­tal pre­dic­tions and the thingy that de­ter­mines my ex­per­i­men­tal re­sults. I call the former thin­gies ‘be­liefs’, and the lat­ter thingy ‘re­al­ity’.”

  This key al­lows you to sub­sti­tute in to his pre­vi­ous para­graph, to ob­tain state­ments in terms of pre­dic­tions and ex­per­i­men­tal re­sults that would be Sabine-ap­proved.

  If we think of the philo­soph­i­cal camps as re­al­ism, in­stru­men­tal­ism /​ prag­ma­tism, and skep­ti­cism, the state of play seems to be less

  “I am a re­al­ist, you are a skep­tic, let’s ar­gue,”

  and more

  “I’m the true prag­ma­tist!” “No, I’m the true prag­ma­tist!”

  I have now skimmed the pre­vi­ous thread, where you also quoted what I just quoted, but said Eliezer was just as­sum­ing that there was some thingie out there be­ing re­al­ity. The al­ter­na­tive be­ing, pre­sum­ably, that our ob­ser­va­tions are not de­ter­mined by any­thing that acts like an ob­ject with prop­er­ties, and are in­stead brute facts.

  But the first sen­tence (“since my ex­pec­ta­tion some­times con­flict...”) is pre­cisely about how he’s not as­sum­ing an ex­ter­nal re­al­ity, but in­stead ad­vanc­ing it as a hy­poth­e­sis in or­der to ex­plain ob­ser­va­tions. Maybe he’s not do­ing it the way you’d like—and maybe I as a bi­ased reader will in­ter­pret that state­ment as a metaphor for some­thing I ex­pect, wheras you’d do the same but get a differ­ent re­sult.

• Charlie Steiner 23 May 2019 7:55 UTC

  4 points

  on: Free will as an ap­pear­ance to others

  This also works on your­self. If your best model of your­self is as an agent mak­ing choices based on their be­liefs, then you will seem to have free well to your­self.