Charlie Steiner(Charlie Steiner)

• Charlie Steiner 20 Feb 2020 22:59 UTC

  LW: 2 AF: 1

  AF

  in reply to: michaelcohen's comment on: Cu­ri­os­ity Killed the Cat and the Asymp­tot­i­cally Op­ti­mal Agent

  Maybe op­ti­mal­ity rel­a­tive to the best performer out of some class of al­gorithms that doesn’t in­clude “just pick the ab­solute best an­swer?” You ba­si­cally prove that in en­vi­ron­ments with traps, any­thing that would, ab­sent traps, be guaran­teed to find the ab­solute best an­swer will in­stead get trapped. So those aren’t ac­tu­ally very good perform­ers.

  I just can’t come up with any­thing too clever, though, be­cause the ob­vi­ous classes of al­gorithms, like “polyno­mial time,” in­clude the abil­ity to just pick the ab­solute best an­swer by luck.

• Charlie Steiner 20 Feb 2020 22:19 UTC

  3 points

  in reply to: adamShimi's comment on: Goal-di­rected = Model-based RL?

  The former (that is, model-based RL-> agent). The lat­ter (smart agent → model-based RL), I think, would be founded on a bit of a level er­ror. At bot­tom, there are only atoms and the void. Whether some­thing is “re­ally” an agent is a ques­tion of how well we can de­scribe this col­lec­tion of atoms in terms of an agent-shaped model. This is differ­ent from the ques­tion of what ab­strac­tions hu­mans used in the pro­cess of pro­gram­ming the AI; Like Ro­hin says, parts of the agent might be thought of as im­plicit in the pro­gram­ming, rather than ex­plicit.

  Sorry, I don’t know if I can di­rect you to any ex­plicit sources. If you check out pa­pers like Con­crete Prob­lems in AI Safety or oth­ers in that genre, though, you’ll see model-based RL used as a sim­plify­ing set of as­sump­tions that im­ply agency.

• Charlie Steiner 20 Feb 2020 20:41 UTC

  LW: 2 AF: 1

  AF

  on: Cu­ri­os­ity Killed the Cat and the Asymp­tot­i­cally Op­ti­mal Agent

  It seems like the up­shot is that even weak op­ti­mal­ity is too strong, since it has to try ev­ery­thing once. How does one make even weaker guaran­tees of good be­hav­ior that are use­ful in prov­ing things, with­out just de­fault­ing to ex­pected util­ity max­i­miza­tion?

• Charlie Steiner 20 Feb 2020 20:22 UTC

  4 points

  on: Goal-di­rected = Model-based RL?

  Yup, I’m pretty sure peo­ple are aware of this :) See also the model of an agent as some­thing with prefer­ences, be­liefs, available ac­tions, and a search+de­ci­sion al­gorithm that makes it take ac­tions it be­lieves will help its prefer­ences.

  But fu­ture AI re­search will re­quire some se­ri­ous gen­er­al­iza­tions that are left un-gen­er­al­ized in cur­rent meth­ods. A sim­ple grid­world prob­lem might treat the en­tire grid as a known POMDP and do search over pos­si­ble se­ries of ac­tions. Ob­vi­ously the real world isn’t a known POMDP, so sup­pose that we just call it an un­known POMDP and try to learn it through ob­ser­va­tion—now all of a sud­den, you can’t hand-spec­ify a cost func­tion in terms of the world model any­more, so that needs to be re-eval­u­ated as well.

• Charlie Steiner 20 Feb 2020 19:24 UTC

  17 points

  on: How do you sur­vive in the hu­man­i­ties?

  Ob­vi­ously I have much less in­for­ma­tion about your situ­a­tion than you, but it seems to me that you’re not in the right here, and you should be less ad­ver­sar­ial.

  E.g.:

  Yes­ter­day, a teacher was ex­plain­ing the com­po­si­tion of a liter­ary es­say, and she claimed that an es­say writer isn’t re­quired to provide jus­tifi­ca­tion for their claims. I asked, “Then why should I be­lieve any­thing the es­say says?” and she replied, “You’re free to de­cide whether you be­lieve it or not,” and I was just too ex­hausted from last week to ex­plain that’s not how be­liefs should work.

  But an es­say writer isn’t re­quired to provide jus­tifi­ca­tion for their claims. For ex­am­ple, your first sen­tence is a claim that you have started study­ing cre­ative writ­ing full time. Have you jus­tified this to me (ei­ther in some unattain­able ab­solute sense, or even just be­yond rea­son­able doubt)? No. Should you? Also no.

  When you run into an ab­surd claim like “An es­say writer isn’t re­quired to provide jus­tifi­ca­tions for their claims,” you should think se­ri­ously about how it might be true. I think you’re only go­ing to be satis­fied by un­der­stand­ing com­mu­ni­ca­tion on a more de­tailed level than your pro­fes­sors do, but you should do that, not just re­ject what they say.

  Back to the ob­ject level: Why should I be­lieve you when you claim that you’ve started study­ing cre­ative writ­ing? I do be­lieve you, of course—but prac­ti­cally speak­ing, why do I do that? Try to figure out an an­swer that gen­er­al­izes by be­ing based on the prac­ti­cal­ities of how hu­mans com­mu­ni­cate and in­fer things about the world. And then ap­ply that an­swer back to what sort of ev­i­dence an es­say writer needs to provide to their au­di­ence to do their job well.

  I also think you’re try­ing to use ar­gu­ments in ways that won’t work. Robert Noz­ick makes some clever com­ments about ar­gu­ments in best part of his book Philo­soph­i­cal Ex­pla­na­tions (the in­tro­duc­tion), some­thing like: The goal of most philoso­phers seems to be to find ar­gu­ments so com­pel­ling, that if a per­son were to dis­agree with the con­clu­sion af­ter read­ing the ar­gu­ment, their head would ex­plode.

  Longer quote:

  The ter­minol­ogy of philo­soph­i­cal art is co­er­cive: ar­gu­ments are pow­er­ful and best when they are knock­down, ar­gu­ments force you to a con­clu­sion, if you be­lieve the premises you have to or must be­lieve the con­clu­sion, some ar­gu­ments do not carry much punch, and so forth. A philo­soph­i­cal ar­gu­ment is an at­tempt to get some­one to be­lieve some­thing, whether he wants to be­lieve it or not. A suc­cess­ful philo­soph­i­cal ar­gu­ment, a strong ar­gu­ment, forces some­one to a be­lief.

  Though philos­o­phy is car­ried on as a co­er­cive ac­tivity, the penalty philoso­phers wield is, af­ter all, rather weak. If the other per­son is will­ing to bear the la­bel of “ir­ra­tional” or “hav­ing the worse ar­gu­ments,” he can skip away hap­pily main­tain­ing his pre­vi­ous be­lief. He will be trailed, of course, by the philoso­pher fu­ri­ously hurl­ing philo­soph­i­cal im­pre­ca­tions: “What do you mean, you’re will­ing to be ir­ra­tional? You shouldn’t be ir­ra­tional be­cause...” And al­though the philoso­pher is em­bar­rassed by his in­abil­ity to com­plete this sen­tence in a non­cir­cu­lar fash­ion—he can only pro­duce rea­sons for ac­cept­ing rea­sons—still, he is un­will­ing to let his ad­ver­sary go.

  Wouldn’t it be bet­ter if philo­soph­i­cal ar­gu­ments left the per­son no pos­si­ble an­swer at all, re­duc­ing him to im­po­tent silence? Even then, he might sit there silently, smil­ing, Bud­dhal­ike. Per­haps philoso­phers need ar­gu­ments so pow­er­ful they set up re­ver­ber­a­tions in the brain: if the per­son re­fuses to ac­cept the con­clu­sion, he dies. How’s that for a pow­er­ful ar­gu­ment. Yet, as with other phys­i­cal threats (“your money or your life”), he can choose defi­ance. A “perfect” philo­soph­i­cal ar­gu­ment would leave no choice.

  But the point of that chap­ter is that such ar­gu­ments don’t ex­ist. They’re an over­sim­plifi­ca­tion of how ar­gu­ments work. A fic­tion. Much like an es­say, an ar­gu­ment with a pro­fes­sor is an ex­er­cise in com­mu­ni­ca­tion, not in struc­tur­ing a co­er­cive ar­gu­ment.

  So if noth­ing else, I think tak­ing a more learn­ing-ori­ented ap­proach to­wards your pro­fes­sors might make you more likely to be able to con­vince them of things :)

• Charlie Steiner 18 Feb 2020 8:02 UTC

  LW: 4 AF: 2

  AF

  in reply to: Chris_Leong's comment on: Refer­ence Post: Triv­ial De­ci­sion Problem

  Reflec­tive mod­ifi­ca­tion flow: Sup­pose we have an EDT agent that can take an ac­tion to mod­ify its de­ci­sion the­ory. It will try to choose based on the av­er­age out­come con­di­tioned on tak­ing the differ­ent de­ci­sion. In some cir­cum­stances, EDT agents are do­ing well so it will ex­pect to do well by not chang­ing; in other cir­cum­stances, maybe it ex­pects to do bet­ter con­di­tional on self-mod­ify­ing to use the Coun­ter­fac­tual Per­spec­tive more.

  Evolu­tion­ary flow: If you put a mix­ture of EDT and FDT agents in an evolu­tion­ary com­pe­ti­tion where they’re play­ing some iter­ated game and high scor­ers get to re­pro­duce, what does the pop­u­la­tion look like at large times, for differ­ent games and start­ing pop­u­la­tions?

• Charlie Steiner 17 Feb 2020 9:58 UTC

  3 points

  on: On char­ac­ter­iz­ing heavy-tailedness

  It seems like if I’m try­ing to talk about a real-world case with finite sup­port, I’ll say some­thing like “it’s not ac­tu­ally a power law—but it’s well de­scribed by one over the rele­vant range of val­ues.” Mean­ing that I have some no­tion of “rele­vant” which is prob­a­bly de­rived from ac­tion-rele­vance, or rele­vance to my ob­ser­va­tions, or maybe com­pu­ta­tional com­plex­ity.

  If I can’t say that, then the other main op­tion is that I care more and more as the power law gets more ex­treme, and then as the pos­si­bil­ities reach their phys­i­cal limit I care most of all. But cases like this are so idiosyn­cratic that maybe there’s no point in try­ing to de­velop a unified lan­guage for them.

• Charlie Steiner 17 Feb 2020 7:46 UTC

  12 points

  on: A ‘Prac­tice of Ra­tion­al­ity’ Se­quence?

  I use­fully demon­strated ra­tio­nal­ity su­per­pow­ers yes­ter­day by bring­ing a power strip to a group pro­ject with limited power out­lets.

  Now, you could try to grind this abil­ity by play­ing im­prov games with the situ­a­tions around you, look­ing for af­for­dances, needs, and solu­tions. But this is only a sub-skill, and I think most of my util­ity comes from things that are more like mind­set tech­nol­ogy.

  A per­sonal anal­ogy: If I want to learn the notes of a tune on the flute, it works fine to just play it re­peat­edly—highly grind­able. If I want to make that tune sound bet­ter, this is harder to grind but still doable; it in­volves more skil­lful listen­ing to oth­ers, listen­ing to your­self, talk­ing, trial and er­ror. If I want to im­prove the skills I use to make tunes sound bet­ter, I can make lots of tunes sound bet­ter, but less of my skill is com­ing from grind­ing now, be­cause that ac­cu­mu­la­tion is slower than other meth­ods of learn­ing. And if I want to im­prove my abil­ity to learn the skills used in mak­ing tunes sound bet­ter...

  Well, first off, that’s a ra­tio­nal­ity-ad­ja­cent skill, in­nit? But sec­ond, grind­ing that is so slow, and so stochas­tic, that it’s hard to dis­t­in­guish from just liv­ing my life, but just hap­pen­ing to try to learn things, and ac­cept­ing that I might learn one-time things that ob­vi­ate a lot of grind­ing.

  So maybe the real grind­ing was bring­ing the power strip all along.

• Charlie Steiner 17 Feb 2020 5:08 UTC

  LW: 2 AF: 1

  AF

  on: The Catas­trophic Con­ver­gence Conjecture

  How much are you think­ing about sta­bil­ity un­der op­ti­miza­tion? Most ob­jec­tive catas­tro­phes are also hu­man catas­tro­phes. But if a pow­er­ful agent is try­ing to achieve some goal while avoid­ing ob­jec­tive catas­tro­phes, it seems like it’s still in­cen­tivized to de­throne hu­mans—to cause ba­si­cally the most hu­man-catas­trophic thing that’s not ob­jec­tive-catas­trophic.

• Charlie Steiner 17 Feb 2020 1:43 UTC

  LW: 2 AF: 1

  AF

  on: Refer­ence Post: Triv­ial De­ci­sion Problem

  I’m definitely satis­fied with this kind of con­tent.

  The names sug­gest you’re clas­sify­ing de­ci­sion pro­ce­dures by what kind of thoughts they have in spe­cial cases. But “sneak­ily” the point is this is rele­vant be­cause these are the kinds of thoughts they have all the time.

  I think the next place to go is to put this in the con­text of meth­ods of choos­ing de­ci­sion the­o­ries—the big ones be­ing re­flec­tive mod­ifi­ca­tion and evolu­tion­ary/​pop­u­la­tion level change. Pretty gen­er­ally it seems like the triv­ial per­spec­tive is un­sta­ble is un­der these, but there are some cir­cum­stances where it’s not.

• Charlie Steiner 10 Feb 2020 6:53 UTC

  LW: 11 AF: 5

  AF

  on: What can the prin­ci­pal-agent liter­a­ture tell us about AI risk?

  Thank you for putting all the time and thought­ful­ness into this post, even if the con­clu­sion is “nope, doesn’t pan out.” I’m grate­ful that it’s out here.

Gricean com­mu­ni­ca­tion and meta-preferences

• Charlie Steiner 10 Feb 2020 1:10 UTC

  19 points

  on: Did AI pi­o­neers not worry much about AI risks?

  I think it’s mostly (3). Not be­cause AI safety is an out­lier, but be­cause of how much work peo­ple had to do to come to grips with Mo­ravec’s para­dox.

  If you take some­one clever and throw them at the prob­lem of GAI, the first thing they’ll think of is some­thing do­ing log­i­cal rea­son­ing, able to fol­low nat­u­ral lan­guage com­mands. Their in­tu­ition will be based on giv­ing or­ders to a hu­man. It takes a lot of work to sup­plant that in­tu­ition with some­thing more mechanis­tic.

  Like, it seems ob­vi­ous to us now that build­ing some­thing that takes nat­u­ral lan­guage com­mands and ac­tu­ally does what we mean is a very hard prob­lem. But this is ex­actly a Mo­ravec’s para­dox situ­a­tion, be­cause know­ing what peo­ple mean is mostly effortless and un­con­scious to us.

• Charlie Steiner 8 Feb 2020 7:42 UTC

  3 points

  in reply to: leggi's comment on: The BODY al­ign­ment game. Take a break, have a quick play.

  Hey, thanks for this con­ge­nial re­ply to my fairly rude com­ment :)

  So, I bring up the mil­i­tary thing be­cause of a room­mate of mine, but if I google “mil­i­tary pos­ture tips,” I get this page, which ba­si­cally says that if you’re hunched for­ward, you need to stretch the mus­cles caus­ing that force, and ex­er­cise the mus­cles that nat­u­rally op­pose it. In short, get a stronger up­per and lower back! They also give spe­cific recom­men­da­tions (albeit mostly geared to­wards body-weight ex­er­cises easy for a home reader to do).

• Charlie Steiner 6 Feb 2020 10:40 UTC

  LW: 4 AF: 2

  AF

  on: Syn­the­siz­ing am­plifi­ca­tion and debate

  I re­ally love the level of de­tail in this sketch!

  I’m men­tally sub­sti­tut­ing $continu{e}_t$ for some ques­tion more like “should this de­bate con­tinue?”, be­cause I think the setup you de­scribe keeps go­ing un­til $Amp$ is satis­fied with an an­swer, which might be never for weak $M$. It’s also not ob­vi­ous to me that this re­ward sys­tem you de­scribe ac­tu­ally teaches agents to de­bate be­tween odd and even steps. If there’s a right an­swer that the judge might be con­vinced of, I think $M$ will be trained to give it no mat­ter the step par­ity, be­cause when that hap­pens it gets re­warded.

  Really, it feels like the state of the de­bate is more like the state of a RNN, and you’re go­ing to end up train­ing some­thing that can make use of that state to do a good job end­ing de­bates and mak­ing the hu­man re­sponse be similar to the model re­sponse.

• Charlie Steiner 6 Feb 2020 9:26 UTC

  5 points

  on: Po­ten­tial Re­search Topic: Vingean Reflec­tion, Value Align­ment and Aspiration

  Thanks! This is an in­ter­est­ing recom­men­da­tion.

  I was definitely struck by the re­sem­blance be­tween her no­tion of “nor­ma­tive de­pen­dence” and the ideas be­hind the CIRL frame­work. And I think that the fix to the AI rea­son­ing about some­thing more in­tel­li­gent is more or less the same thing hu­mans do, which is we ab­stract away the plan­ning and re­place it with some “power” to do some­thing. Like if I imag­ine play­ing Mag­nus Car­lsen in chess, I don’t simu­late a chess game at all, I com­pare an imag­i­nary chess-win­ning power I at­tribute to us in my ab­stracted men­tal rep­re­sen­ta­tion.

  But as for the philo­soph­i­cal prob­lems she men­tions in the in­ter­view, I felt like they fell into pretty stan­dard or­tho­dox philo­soph­i­cal failure modes. For the sake of clar­ity, I guess I should say I mean the ob­ses­sion with his­tory, and the de­fault as­sump­tion that ques­tions have one right an­swer—things I think are boon­dog­gles have to be ad­dressed just be­cause they’re his­tor­i­cal, and there’s too much worry about what hu­mans “re­ally” are like as op­posed to con­sid­er­a­tion of mod­els of hu­mans.

• Charlie Steiner 6 Feb 2020 6:01 UTC

  LW: 7 AF: 4

  AF

  on: Wri­teup: Progress on AI Safety via Debate

  You have an en­tire copy of the post in the com­ment­ing guidelines, fyi :)

  What’s of­ten go­ing on in un­re­solv­able de­bates among hu­mans is that there is a vague defi­ni­tion baked into the ques­tion, such that there is no “re­ally” right an­swer (or too many right an­swers).

  E.g. “Are viruses al­ive?”

  To the ex­tent that we’ve dealt with the ques­tion of whether viruses are al­ive, it’s been by un­der­stand­ing the com­pli­ca­tions and let­ting go of the need for the cat­e­gor­i­cal think­ing that gen­er­ated the ques­tion in the first place. Allow­ing this as an op­tion seems like it brings back down the com­plex­ity class of things you can re­solve de­bates on (though if you count “it’s a tie” as a re­s­olu­tion, you might re­tain the abil­ity to ask ques­tions in PSPACE but just have lots of un­in­for­ma­tive ties and only up­date your own wor­ld­view when it’s su­per easy).

  For ques­tions of value, though, this ap­proach might not even always work, be­cause the ques­tion might be “is it right to take ac­tion A or ac­tion B,” and even if you step back from the cat­e­gory “right” be­cause it’s too vague, you still have to choose be­tween ac­tion A or B. But you still have the origi­nal is­sue that the ques­tion has too few /​ too many right an­swers. Any thoughts on ways to make de­bate do work on this sort of tricky prob­lem?

• Charlie Steiner 5 Feb 2020 5:15 UTC

  3 points

  on: The BODY al­ign­ment game. Take a break, have a quick play.

  Weird ques­tion, why bother men­tally map­ping two differ­ent ends of the same bone (ster­num)? In fact, why all this trivia about bone knobs in the first place? If I want good pos­ture, I’d be bet­ter off learn­ing the les­sons of the mil­i­tary, and if I want to re­lax, why bone knobs?

• Charlie Steiner 3 Feb 2020 4:48 UTC

  3 points

  on: Money isn’t real. When you donate money to a char­ity, how does it ac­tu­ally help?

  It’s fine, peo­ple are only 1 layer of un­re­al­ity re­moved from money, so they can in­ter­act via grav­ity, which “leaks” into the 4th di­men­sion (ex­plain­ing why it’s so much weaker than the elec­tro­mag­netic force).

• Charlie Steiner 3 Feb 2020 2:09 UTC

  LW: 4 AF: 2

  AF

  on: Towards de­con­fus­ing values

  When you say the hu­man de­ci­sion pro­ce­dure causes hu­man val­ues, what I hear is that the hu­man de­ci­sion pro­ce­dure (and its sur­round­ing way of de­scribing the world) is more on­tolog­i­cally ba­sic than hu­man val­ues (and their sur­round­ing way if de­scribing the world).

  Our de­ci­sion pro­ce­dure is “the rea­son for our val­ues” in the same way that the mo­tion of elec­tric charge in your com­puter is the rea­son it plays videogames (even though “the elec­tric charge is mov­ing” and “it’s play­ing a game” might be de­scribing the same phys­i­cal event). The ar­row be­tween them isn’t the most typ­i­cal causal ar­row be­tween two peers in a sin­gu­lar way of de­scribing the world, it’s an ar­row of re­duc­tion/​emer­gence, be­tween things at differ­ent lev­els of ab­strac­tion.