riceissa

[Question] Is it harder to be­come a MIRI math­e­mat­i­cian in 2019 com­pared to in 2013?

• riceissa 28 Oct 2019 22:54 UTC

  6 points

  on: [AN #62] Are ad­ver­sar­ial ex­am­ples caused by real but im­per­cep­ti­ble fea­tures?

  Based on the Oc­to­ber 2019 up­date, it looks like Ought is now us­ing “fac­tored cog­ni­tion” as an um­brella term that in­cludes both fac­tored gen­er­a­tion (which used to be called fac­tored cog­ni­tion) and fac­tored eval­u­a­tion.

  (Com­ment­ing here be­cause as far as I know this post is one of the main places that dis­cusses this dis­tinc­tion.)

• riceissa 26 Oct 2019 22:29 UTC

  5 points

  on: Jacy Reese (born Jacy An­this)?

  It’s in­ter­est­ing this minor fact that, to sev­eral peo­ple in­clud­ing me, has seemed like an ob­vi­ous omis­sion, doesn’t meet Wikipe­dia’s stan­dards for in­clu­sion. But if Wikipe­dia had less strict stan­dards it would be very hard to keep out false in­for­ma­tion.

  Eliezer Yud­kowsky has made similar dis­tinc­tions when talk­ing about sci­en­tific vs le­gal vs ra­tio­nal ev­i­dence (see this wiki page) and sci­ence vs prob­a­bil­ity the­ory.

  I think there is an in­ter­est­ing ques­tion of “what ought to count as ev­i­dence, if we want to pro­duce the best on­line en­cy­clo­pe­dia we can, given the flawed hu­mans we have to write it?” My own view is that Wikipe­dia’s stan­dards for ev­i­dence have be­come too strict in cases like this.

• riceissa 26 Oct 2019 1:54 UTC

  1 point

  in reply to: John_Maxwell's comment on: De­liber­a­tion as a method to find the “ac­tual prefer­ences” of humans

  Thanks, I think I agree (but want to think about this more). I might edit the post in the fu­ture to in­cor­po­rate this change.

• riceissa 26 Oct 2019 1:51 UTC

  1 point

  in reply to: Charlie Steiner's comment on: De­liber­a­tion as a method to find the “ac­tual prefer­ences” of humans

  I agree with this, and didn’t mean to im­ply any­thing against it in the post.

• riceissa 26 Oct 2019 1:42 UTC

  8 points

  on: Two ex­pla­na­tions for vari­a­tion in hu­man abilities

  Re­gard­ing your foot­note, liter­acy rates de­pend on the defi­ni­tion of liter­acy used. Un­der min­i­mal defi­ni­tions, “pretty close to 100 per­cent of the pop­u­la­tion is ca­pa­ble of read­ing” is true, but un­der stric­ter defi­ni­tions, “maybe 20 or 30 per­cent” seems closer to the mark.

  https://​​en.wikipe­dia.org/​​wiki/​​Liter­acy#United_States

  https://​​en.wikipe­dia.org/​​wiki/​​Func­tional_illiter­acy#Prevalence

  https://​​en.wikipe­dia.org/​​wiki/​​Liter­acy_in_the_United_States

  “Cur­rent liter­acy data are gen­er­ally col­lected through pop­u­la­tion cen­suses or house­hold sur­veys in which the re­spon­dent or head of the house­hold de­clares whether they can read and write with un­der­stand­ing a short, sim­ple state­ment about one’s ev­ery­day life in any writ­ten lan­guage. Some sur­veys re­quire re­spon­dents to take a quick test in which they are asked to read a sim­ple pas­sage or write a sen­tence, yet clearly liter­acy is a far more com­plex is­sue that re­quires more in­for­ma­tion.” http://​​uis.un­esco.org/​​en/​​topic/​​literacy

  I’m not sure why you are so op­ti­mistic about peo­ple learn­ing calcu­lus.

• riceissa 24 Oct 2019 0:07 UTC

  5 points

  in reply to: habryka's comment on: AI Align­ment Open Thread Oc­to­ber 2019

  Thanks!

  My sense (guess­ing from your post his­tory) is that you are mostly con­fused about com­ments.

  I am more con­fused about posts than com­ments. For posts, only my com­par­i­son of de­ci­sion the­o­ries post is cur­rently cross-posted to AF, but I ac­tu­ally think my post about de­liber­a­tion, ques­tion about iter­a­tive ap­proaches to al­ign­ment (along with Ro­hin’s an­swer), and ques­tion about co­or­di­na­tion on AI progress pro­jects are more rele­vant to AF (ei­ther be­cause they make new claims or be­cause they en­courage oth­ers to do so). If I see that a par­tic­u­lar post hasn’t been cross-posted to AF, I’m won­der­ing if I should be think­ing more like “ev­ery sin­gle mod­er­a­tor has looked at the post, and be­lieves it doesn’t be­long on AF” or more like “ei­ther the mod­er­a­tors are busy, or some­thing about the post ti­tle caused them to not look at the post, and it sort of fell through the cracks”.

• riceissa 23 Oct 2019 22:39 UTC

  1 point

  on: AI Align­ment Open Thread Oc­to­ber 2019

  [Meta] I’m not a full mem­ber on Align­ment Fo­rum, but I’ve had some of my LW con­tent cross-posted to AF. How­ever, this cross-post­ing seems hap­haz­ard, and does not cor­re­spond to my in­tu­itive feel­ing of which of my posts/​com­ments “should” end up on AF. I would like for one of the fol­low­ing to hap­pen:

  • More in­sight into the mechanism that de­cides what gets cross-posted, so I feel less an­noyed at the ar­bi­trary-seem­ing na­ture of it.

  • More con­trol over what gets cross-posted (if this re­quires ap­ply­ing for full mem­ber­ship, I would be will­ing to do that).

  • Have all my AF cross-post­ing be un­done so that read­ers don’t get a mis­lead­ing im­pres­sion of my AI al­ign­ment con­tent. (I would like to avoid peo­ple vis­it­ing my AF pro­file, read­ing con­tent there, and con­clud­ing some­thing about my AI al­ign­ment out­put based on that.)

• riceissa 23 Oct 2019 21:40 UTC

  3 points

  in reply to: An1lam's comment on: An1lam’s Short Form Feed

  If you’ve saved some of your fa­vorite links, please share!

  • I like CheCheDaWaff’s com­ments on r/​Anki; see here for a de­cent place to start. In par­tic­u­lar, for proofs, I’ve shifted to­ward adding “prove this the­o­rem” cards rather than try­ing to break the proof into many small pieces. (The lat­ter ad­heres more to the spaced rep­e­ti­tion philos­o­phy, but I found it just doesn’t re­ally work.)

  • Richard Reitz has a Google doc with a bunch of stuff.

  • I like this fo­rum com­ment (as a data point, and as mo­ti­va­tion to try to avoid similar failures).

  • I like https://​​es­ha­pard.github.io

  • Master How To Learn also has some in­sights but most posts are low-qual­ity.

  One thing I should men­tion is that a lot of the above links aren’t writ­ten well. See this Quora an­swer for a view I ba­si­cally agree with.

  I couldn’t stop think­ing about it

  I agree that think­ing about this is pretty ad­dict­ing. :) I think this kind of mo­ti­va­tion helps me to find and read a bunch on­line and to make oc­ca­sional com­ments (such as the grand­par­ent) and brain dumps, but I find it’s not quite enough to get me to in­vest the time to write a com­pre­hen­sive post about ev­ery­thing I’ve learned.

• riceissa 23 Oct 2019 4:10 UTC

  5 points

  in reply to: An1lam's comment on: An1lam’s Short Form Feed

  I would be sur­prised if Gw­ern hasn’t already thought about the claim go­ing to make

  I briefly looked at gw­ern’s pub­lic database sev­eral months ago, and got the im­pres­sion that he isn’t us­ing Anki in the in­cre­men­tal read­ing/​learn­ing way that you (and Michael Niel­sen) de­scribe. In­stead, he seems to just add a bunch of ran­dom facts. This isn’t to say gw­ern hasn’t thought about this, but just that if he has, he doesn’t seem to be mak­ing use of this in­sight.

  In the Pla­tonic graph of this do­main’s knowl­edge on­tol­ogy, how cen­tral is this node?

  I feel like the cen­ter of­ten shifts as I learn more about a topic (be­cause I de­velop new in­ter­ests within it). The ques­tions I ask my­self are more like “How em­bar­rassed would I be if some­one asked me this and I didn’t know the an­swer?” and “How much does know­ing this help me learn more about the topic or re­lated top­ics?” (Th­ese aren’t ideal phras­ings of the ques­tions my gut is ask­ing.)

  know­ing that I’ll re­mem­ber at least the stuff I’ve Anki-ized has a sur­pris­ingly strong mo­ti­va­tional im­pact on me on a gut level

  In my ex­pe­rience, I of­ten still for­get things I’ve en­tered into Anki ei­ther be­cause the card was poorly made or be­cause I didn’t add enough “sur­round­ing cards” to ce­ment the knowl­edge. So I’ve shifted away from this to think­ing some­thing more like “at least Anki will make it very ob­vi­ous if I didn’t in­ter­nal­ize some­thing well, and will give me an op­por­tu­nity in the fu­ture to come back to this topic to un­der­stand it bet­ter in­stead of just hav­ing it fade with­out de­tec­tion”.

  there’s O(5) ac­tual blog posts about it

  I’m con­fused about what you mean by this. (One guess I have is big-O no­ta­tion, but big-O no­ta­tion is not sen­si­tive to con­stants, so I’m not sure what the 5 is do­ing, and big-O no­ta­tion is also about asymp­totic be­hav­ior of a func­tion and I’m not sure what in­put you’re con­sid­er­ing.)

  I think there are few well-re­searched and com­pre­hen­sive blog posts, but I’ve found that there is a lot of ad­di­tional wis­dom the spaced rep­e­ti­tion com­mu­nity has ac­cu­mu­lated, which is mostly writ­ten down in ran­dom Red­dit com­ments and smaller blog posts. I feel like I’ve benefited some­what from read­ing this wis­dom (but have benefited more from just try­ing a bunch of things my­self). For my­self, I’ve con­sid­ered writ­ing up what I’ve learned about us­ing Anki, but it hasn’t been a pri­or­ity be­cause (1) other top­ics seem more im­por­tant to work on and write about; (2) most new­com­ers can­not dis­t­in­guish been good and bad ad­vice, so I an­ti­ci­pate hav­ing low im­pact by writ­ing about Anki; (3) I’ve only been ex­per­i­ment­ing in­for­mally and per­son­ally, and it’s difficult to tell how well my les­sons gen­er­al­ize to oth­ers.

De­liber­a­tion as a method to find the “ac­tual prefer­ences” of humans

• riceissa 22 Oct 2019 4:29 UTC

  9 points

  on: Ra­tion­al­ity Ex­er­cises Prize of Septem­ber 2019 ($1,000)

  Were the win­ners ever an­nounced? If I’m count­ing cor­rectly, it has now been over four weeks since Septem­ber 20, so the win­ners should have been an­nounced around two weeks ago. (I checked for new posts by Ben, this post, and the com­ments on this post.)

• riceissa 21 Oct 2019 20:28 UTC

  1 point

  in reply to: capybaralet's comment on: AI Safety “Suc­cess Sto­ries”

  I think I was imag­in­ing that the pivotal tool AI is de­vel­oped by highly com­pe­tent and safety-con­scious hu­mans who use it to perform a pivotal act (or se­ries of pivotal acts) that effec­tively pre­cludes the kind of is­sues men­tioned in Wei’s quote there.

  Even if you make this as­sump­tion, it seems like the re­li­ance on hu­man safety does not go down. I think you’re think­ing about some­thing more like “how likely it is that lack of hu­man safety be­comes a prob­lem” rather than “re­li­ance on hu­man safety”.

• riceissa 21 Oct 2019 1:22 UTC

  7 points

  on: We tend to for­get com­pli­cated things

  I think you are de­scribing over­learn­ing and chunk­ing (once con­cepts be­come chun­ked they “feel easy”, and one re­li­able way to chunk ideas is to over­learn them).

  Some re­lated links:

  • A com­ment by Qiaochu Yuan

  • Bryan Ca­plan’s “Liber­tar­i­anism as Mo­ral Over­learn­ing”

  • Aug­ment­ing Long-term Me­mory (search “chunk”)

• riceissa 19 Oct 2019 2:17 UTC

  1 point

  on: Hu­mans can be as­signed any val­ues what­so­ever…

  I’m cu­ri­ous what you think of my com­ment here, which sug­gests that Kol­mogorov com­plex­ity might be enough af­ter all, as long as we are will­ing to change our no­tion of com­pat­i­bil­ity.

  (I’m also cu­ri­ous what you think of Daniel’s post, al­though to a lesser ex­tent.)

• riceissa 18 Oct 2019 1:32 UTC

  4 points

  in reply to: capybaralet's comment on: AI Safety “Suc­cess Sto­ries”

  I think pivotal tool story has low re­li­ance on hu­man safety (al­though I’m con­fused by that row in gen­eral).

  From the Task-di­rected AGI page on Ar­bital:

  The ob­vi­ous dis­ad­van­tage of a Task AGI is moral haz­ard—it may tempt the users in ways that a Sovereign would not. A Sovereign has moral haz­ard chiefly dur­ing the de­vel­op­ment phase, when the pro­gram­mers and users are per­haps not yet in a po­si­tion of spe­cial rel­a­tive power. A Task AGI has on­go­ing moral haz­ard as it is used.

  (My un­der­stand­ing is that task AGI = ge­nie = Pivotal Tool.)

  Wei Dai gives some ex­am­ples of what could go wrong in this post:

  For ex­am­ple, such AIs could give hu­mans so much power so quickly or put them in such novel situ­a­tions that their moral de­vel­op­ment can’t keep up, and their value sys­tems no longer ap­ply or give es­sen­tially ran­dom an­swers. AIs could give us new op­tions that are ir­re­sistible to some parts of our mo­ti­va­tional sys­tems, like more pow­er­ful ver­sions of video game and so­cial me­dia ad­dic­tion. In the course of try­ing to figure out what we most want or like, they could in effect be search­ing for ad­ver­sar­ial ex­am­ples on our value func­tions. At our own re­quest or in a sincere at­tempt to help us, they could gen­er­ate philo­soph­i­cal or moral ar­gu­ments that are wrong but ex­tremely per­sua­sive.

  The un­der­ly­ing prob­lem seems to be that when hu­mans are in con­trol over long-term out­comes, we are rely­ing more on the hu­mans to have good judg­ment, and this be­comes in­creas­ingly a prob­lem the more task-shaped the AI be­comes.

  I’m cu­ri­ous what your own think­ing is (e.g. how would you fill out that row?).

• riceissa 18 Oct 2019 1:11 UTC

  2 points

  in reply to: capybaralet's comment on: AI Safety “Suc­cess Sto­ries”

  Or does it also in­clude a story about how AI is de­ployed (and by who, etc.)?

  The “Con­trol­led ac­cess” row seems to im­ply that at least part of how the AI is de­ployed is part of each suc­cess story (with some other parts left to be filled in later). I agree that hav­ing more de­tails for each story would be nice.

  Some­what re­lated to this is that I’ve found it slightly con­fus­ing that each suc­cess story is named af­ter the kind of AI that is pre­sent in that story. So when one says “Sovereign Sin­gle­ton”, this could mean ei­ther the AI it­self or the AI to­gether with all the other as­sump­tions (e.g. hard take­off) for how hav­ing that kind of AI leads to a “win”.

  What links here?

  • De­liber­a­tion as a method to find the “ac­tual prefer­ences” of humans by riceissa (22 Oct 2019 9:23 UTC; 24 points)

• riceissa 9 Oct 2019 21:14 UTC

  8 points

  on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  I thought about this more and re-read the A&M pa­per, and I now have a differ­ent line of think­ing com­pared to my pre­vi­ous com­ments.

  I still think A&M’s No Free Lunch the­o­rem goes through, but now I think A&M are prov­ing the wrong the­o­rem. A&M try to find the sim­plest (plan­ner, re­ward) de­com­po­si­tion that is com­pat­i­ble with the hu­man policy, but it seems like we in­stead ad­di­tion­ally want com­pat­i­bil­ity with all the ev­i­dence we have ob­served, in­clud­ing sen­sory data of hu­mans say­ing things like “if I was more ra­tio­nal, I would be ex­er­cis­ing right now in­stead of watch­ing TV” and “no re­ally, my re­ward func­tion is not empty”. The im­por­tant point is that such sen­sory data gives us in­for­ma­tion not just about the hu­man policy, but also about the de­com­po­si­tion. Forc­ing com­pat­i­bil­ity with this sen­sory data seems to rule out de­gen­er­ate pairs. This makes me feel like Oc­cam’s Ra­zor would work for in­fer­ring prefer­ences up to a cer­tain point (i.e. as long as the situ­a­tions are all “in-dis­tri­bu­tion”).

  If we are try­ing to find the (plan­ner, re­ward) de­com­po­si­tion of non-hu­man minds: I think if we were ran­domly handed a mind from all of mind de­sign space, then A&M’s No Free Lunch the­o­rem would ap­ply, be­cause the sim­plest ex­pla­na­tion re­ally is that the mind has a de­gen­er­ate de­com­po­si­tion. But if we were ran­domly handed an alien mind from our uni­verse, then we would be able to use all the facts we have learned about our uni­verse, in­clud­ing how the aliens likely evolved, any state­ments they seem to be mak­ing about what they value, and so on.

  Does this line of think­ing also ap­ply to the case of sci­ence? I think not, be­cause we wouldn’t be able to use our ob­ser­va­tions to get in­for­ma­tion about the de­com­po­si­tion. Un­like the case of val­ues, the nat­u­ral world isn’t mak­ing state­ments like “ac­tu­ally, the laws are empty and all the com­plex­ity is in the ini­tial con­di­tions”. I still don’t think the No Free Lunch the­o­rem works for sci­ence ei­ther, be­cause of my pre­vi­ous com­ments.

  What links here?

  • riceissa's comment on Hu­mans can be as­signed any val­ues what­so­ever… by Stuart_Armstrong (19 Oct 2019 2:17 UTC; 1 point)

• riceissa 9 Oct 2019 7:50 UTC

  5 points

  in reply to: Ben Pace's comment on: List of re­solved con­fu­sions about IDA

  Seems odd to have the ideal­is­tic goal get to be the stan­dard name, and the dime-a-dozen failure mode be a longer name that is more con­fus­ing.

  I agree this is con­fus­ing.

  Is there a rea­son why the stan­dard terms are not be­ing used to re­fer to the stan­dard, short-term re­sults?

  As far as I know, Paul hasn’t ex­plained his choice in de­tail. One rea­son he does men­tion, in this com­ment, is that in the con­text of strat­egy-steal­ing, prefer­ences like “help me stay in con­trol and be well-in­formed” do not make sense when in­ter­preted as prefer­ences-as-elic­ited, since the cur­rent user has no way to know if they are in con­trol or well-in­formed.

  In the post Wei con­trasts “cur­rent” and “ac­tual” prefer­ences. “Stated” vs “re­flec­tive” prefer­ences also seem like nice al­ter­na­tives too.

  I think cur­rent=elic­ited=stated, but ac­tual≈re­flec­tive (be­cause there is the pos­si­bil­ity that un­der­go­ing re­flec­tion isn’t a good way to find out our ac­tual prefer­ences, or as Paul says ‘There’s a hy­poth­e­sis that “what I’d say af­ter some par­tic­u­lar ideal­ized pro­cess of re­flec­tion” is a rea­son­able way to cap­ture “ac­tual pre­fer­ences,” but I think that’s up for de­bate—e.g. it could fail if me-on-re­flec­tion is self­ish and has val­ues op­posed to cur­rent-me, and cer­tainly it could fail for any par­tic­u­lar pro­cess of re­flec­tion and so it might just hap­pen to be the case that there is no pro­cess of re­flec­tion that satis­fies it.’)

• riceissa 9 Oct 2019 6:41 UTC

  1 point

  in reply to: Ben Pace's comment on: List of re­solved con­fu­sions about IDA

  I think Paul calls that “prefer­ences-as-elic­ited”, so if we’re talk­ing about act-based agents, it would be “short-term prefer­ences-as-elic­ited” (see this com­ment).