Stuart_Armstrong

• Stuart_Armstrong12 Dec 2018 14:16 UTC

  2 points

  in reply to: rohinmshah's comment on: Figur­ing out what Alice wants: non-hu­man Alice

  In the ex­am­ple in https://​​www.less­wrong.com/​​posts/​​rcXaY3FgoobMkH2jc/​​figur­ing-out-what-al­ice-wants-part-ii , I give ex­am­ples of two al­gorithms with the same out­puts but where we would at­tribute differ­ent prefer­ences to them. This sidesteps the im­pos­si­bil­ity re­sult, since it al­lows us to con­sider ex­tra in­for­ma­tion, namely the in­ter­nal struc­ture of the al­gorithm, in a way rele­vant to value-com­put­ing.

• Stuart_Armstrong12 Dec 2018 13:48 UTC

  2 points

  in reply to: rohinmshah's comment on: As­sum­ing we’ve solved X, could we do Y...

  I think it de­pends on the in­di­vi­d­ual. Cer­tainly, be­fore re­al­is­ing the points above, I would oc­ca­sion­ally men­tally do the “as­sume hu­man val­ues solved” in my mind, in an un­ri­gor­ous and men­tally mis­lead­ing way.

• Stuart_Armstrong12 Dec 2018 0:41 UTC

  2 points

  in reply to: jessica.liu.taylor's comment on: Figur­ing out what Alice wants: non-hu­man Alice

  What do you mean by “you ac­tu­ally have Y val­ues”? What are you defin­ing val­ues to be?

• Stuart_Armstrong12 Dec 2018 0:19 UTC

  2 points

  in reply to: jessica.liu.taylor's comment on: Figur­ing out what Alice wants: non-hu­man Alice

  Be­cause once we have these pa­ram­e­ters, we can learn the val­ues of any given hu­man. In con­trast, it we learn the val­ues of a given hu­man, we don’t get to learn the val­ues of any other one.

  I’d ar­gue fur­ther: these pa­ram­e­ters form part of a defi­ni­tion of hu­man val­ues. We can’t just “learn hu­man val­ues”, as these don’t ex­ist in the world. Whereas “learn what hu­mans model each other’s val­ues (and ra­tio­nal­ity) to be” is some­thing that makes sense in the world.

• Stuart_Armstrong12 Dec 2018 0:07 UTC

  3 points

  in reply to: avturchin's comment on: Bounded ra­tio­nal­ity abounds in mod­els, not ex­plic­itly defined

  If we want to ap­ply it to hu­mans, some­thing much more com­pli­cated than that, which uses some mea­sure of how com­plex hu­mans see ac­tions, takes into ac­count how and when we search for al­ter­nate solu­tions. There’s a rea­son most mod­els don’t use bounded ra­tio­nal­ity; it ain’t sim­ple.

• Stuart_Armstrong11 Dec 2018 23:13 UTC

  3 points

  in reply to: TheWakalix's comment on: A hun­dred Shakespeares

  Cor­rected, thanks!

A hun­dred Shakespeares

• Stuart_Armstrong11 Dec 2018 22:42 UTC

  3 points

  in reply to: DanielFilan's comment on: Bounded ra­tio­nal­ity abounds in mod­els, not ex­plic­itly defined

  I agree it’s part of the story, but only a part. And real hu­mans don’t act as if there was a set of ac­tions of size n, and they could con­sider all of them with equal ease. Some­times hu­mans have much smaller ac­tion sets, some­times they can pro­duce com­pletely un­ex­pected ac­tions, and most of the time we have a pretty small set of ob­vi­ous ac­tions and a much larger set of po­ten­tial ac­tions we might be able to think up at the cost of some effort.

Bounded ra­tio­nal­ity abounds in mod­els, not ex­plic­itly defined

Figur­ing out what Alice wants: non-hu­man Alice

As­sum­ing we’ve solved X, could we do Y...

Why we need a *the­ory* of hu­man values

• Stuart_Armstrong5 Dec 2018 15:56 UTC

  2 points

  in reply to: Donald Hobson's comment on: For­mal Open Prob­lem in De­ci­sion Theory

  Thanks for in­tro­duc­ing me to the box topol­ogy—see­ing it defined so ex­plic­itly, and see­ing what prop­er­ties it fails, cleared up a few of my in­tu­itions.

• Stuart_Armstrong4 Dec 2018 14:54 UTC

  2 points

  in reply to: John_Maxwell_IV's comment on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  A chess tree search al­gorithm would never hit upon kil­ling other pro­cesses. An evolu­tion­ary chess-play­ing al­gorithm might learn to do that. It’s not clear whether goal-di­rected is rele­vant to that dis­tinc­tion.

• Stuart_Armstrong3 Dec 2018 20:56 UTC

  7 points

  in reply to: Donald Hobson's comment on: For­mal Open Prob­lem in De­ci­sion Theory

  Hum, $I^{\omega }$ should be com­pact by Ty­chonoff’s the­o­rem (see also the Hilbert Cube, which is home­o­mor­phic to $I^{\omega }$).

  For your proof, I think that $G$ is not open in the product topol­ogy. The product topol­ogy is the coars­est topol­ogy where all the pro­jec­tion maps are con­tin­u­ous.

  To make all the pro­jec­tion maps con­tin­u­ous we need all sets in $S$ to be open, where we define $\sigma \in S$ iff there ex­ists an $i$, such that $p_i\left(\sigma \right)$ is open in $[0,1]$ and $\sigma =\{x=(x_1,x_2,\dots )|x_i\in p_i(\sigma ),0\le x_j\le 1\text{for}i\ne j\}$.

  Let $S^{\prime }$ be the set of finite in­ter­sec­tion of these sets. For any ${\sigma }^{\prime }\in S^{\prime }$, there ex­ists a finite set $N_{{\sigma }^{\prime }}\subset N$ such that if $x\in {\sigma }^{\prime }$ and $y_i=x_i$ for $i\in N_{{\sigma }^{\prime }}$, then $y\in {\sigma }^{\prime }$ as well.

  If we take $S^{\prime \prime }$ to be the ar­bi­trary union of $S^{\prime }$, this con­di­tion will be pre­served. Thus $G$ is not con­tained in the ar­bi­trary unions and finite in­ter­sec­tions of $S$, so it seems it is not an open sent.

  Also, $I^{\omega }$ is sec­ond-countable. From the wikipe­dia ar­ti­cle on sec­ond-countable:

  Any countable product of a sec­ond-countable space is sec­ond-countable

• Stuart_Armstrong3 Dec 2018 20:43 UTC

  9 points

  on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  Note that this starts from the as­sump­tion of goal-di­rected be­hav­ior and de­rives that the AI will be an EU max­i­mizer along with the other con­ver­gent in­stru­men­tal sub­goals.

  The re­sult is ac­tu­ally stronger than that, I think: if the AI is goal-di­rected at least in part, then that part will (tend to) purge the non-goal di­rected be­havi­ours and then fol­low the EU path.

  I won­der if we could get the­o­rems as to what kinds of min­i­mal goal di­rected be­havi­our will re­sult in the agent be­com­ing a com­pletely goal-di­rected agent.

• Stuart_Armstrong1 Dec 2018 18:00 UTC

  11 points

  on: Is Science Slow­ing Down?

  But still? A hun­dred Shake­speares?

  I’d wa­ger there are thou­sands of Shake­speare-equiv­a­lents around to­day. The is­sue is that Shake­speare was not only tal­ented, he was suc­cess­ful—wildly pop­u­lar, and able to live off his writ­ing. He was a su­per­star of the­atre. And we can only have a limited amount of su­per­stars, no mat­ter how large the pop­u­la­tion grows. So if we took only his first few plays (be­fore he got the fame feed­back loop and money), and gave them to some­one who had, some­how, never heard of Shake­speare, I’d wa­ger they would find many other au­thors at least as good.

  This is a mild point in favour of ex­pla­na­tion 1, but it’s not that the num­ber of de­voted re­searchers is limited, it’s that the slots at the top of the re­search lad­der are limited. In this view, any very tal­ented in­di­vi­d­ual who was also a su­per­star, will pro­duce a huge amount of re­search. The num­ber of very tal­ented in­di­vi­d­u­als has gone up, but the num­ber of su­per­star slots has not.

• Stuart_Armstrong28 Nov 2018 11:45 UTC

  2 points

  in reply to: gworley's comment on: Web of con­no­ta­tions: Bleggs, Rubes, ther­mostats and beliefs

  But I also sense a priv­ileg­ing of a par­tic­u­lar wor­ld­view, namely a hu­man one, that may ar­tifi­cially limit the sorts of use­ful cat­e­gories we are will­ing to con­sider.

  This is de­liber­ate—a lot what I’m try­ing to do is figure out hu­man val­ues, so the hu­man wor­ld­views and in­ter­pre­ta­tions will gen­er­ally be the most rele­vant.

Hu­mans can be as­signed any val­ues what­so­ever…

• Stuart_Armstrong17 Oct 2018 20:42 UTC

  4 points

  on: Policy Beats Morality

  poli­ti­ci­ans like be­ing able to ac­tu­ally change pub­lic behavior

  But the ways in which they want to/​can change them is strongly in­fluenced by moral prefer­ences among vot­ers, donor, and civil ser­vants. Why did they shift re­cy­cling or bring in clean air/​wa­ter acts, rather than bring­ing in any of a mil­lion other policy changes they could have?