Nick_Beckstead

• Nick_Beckstead 11 May 2012 3:56 UTC

  66 points

  in reply to: Wei Dai’s comment on: Thoughts on the Sin­gu­lar­ity In­sti­tute (SI)

  While I have sympathy with the complaint that SI’s critics are inarticulate and often say wrong things, Eliezer’s comment does seem to be indicative of the mistake Holden and Wei Dai are describing. Most extant presentations of SIAI’s views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals. This makes it harder to make high quality objections. I think it would be more appropriate to react to poor critical engagement more along the lines of “We haven’t gotten great critics. That probably means that we need to work on our arguments and their presentation,” and less along the lines of “We haven’t gotten great critics. That probably means that there’s something wrong with the rest of the world.”

  What links here?

  • A Schol­arly AI Risk Wiki by lukeprog (25 May 2012 20:53 UTC; 28 points)
  • lukeprog's comment on Thoughts on the Sin­gu­lar­ity In­sti­tute (SI) by HoldenKarnofsky (11 May 2012 23:11 UTC; 26 points)
  • lukeprog's comment on Thoughts on the Sin­gu­lar­ity In­sti­tute (SI) by HoldenKarnofsky (11 May 2012 8:19 UTC; 7 points)

Com­mon sense as a prior

A Pro­posed Ad­just­ment to the Astro­nom­i­cal Waste Argument

My daily re­flec­tion routine

• Nick_Beckstead 20 Aug 2011 1:10 UTC

  29 points

  on: Why We Can’t Take Ex­pected Value Es­ti­mates Liter­ally (Even When They’re Un­bi­ased)

  Here’s what I think is true and important about this post: some people will try to explicitly estimate expected values in ways that don’t track the real expected values, and when they do this, they’ll make bad decisions. We should avoid these mistakes, which may be easy to fall into, and we can avoid some of them by using regressions of the kind described above in the case of charity cost-effectiveness estimates. As Toby points out, this is common ground between GiveWell and GWWC. Let me list a what I take to be a few points of disagreement.

  I think that after making an appropriate attempt to gather evidence, the result of doing the best expected value calculation that you can is by far the most important input into a large scale philanthropic decision. We should think about the result of the calculation makes sense, we should worry if it is wildly counterintuitive, and we should try hard to avoid mistakes. But the result of this calculation will matter more than most kinds of informal reasoning, especially if the differences in expected value are great. I think this will be true for people who are competent with thinking in terms of subjective probabilities and expected values, which will rule out a lot of people, but will include a lot of the people who would consider whether to make important philanthropic decisions on the basic of expected value calculations.

  I think this argument unfairly tangles up making decisions explicitly on the basis of expected value calculations with Pascal’s Mugging. It’s not too hard to choose a bounded utility function that doesn’t tell you to pay the mugger, and there are independent (though not clearly decisive) reasons to use a bounded utility function for decision-making, even when the probabilities are stable. Since the unbounded utility function assumption can shoulder the blame, the invocation of Pascal’s Mugging doesn’t seem all that telling. (Also, for reasons Wei Dai gestures at I don’t accept Holden’s conjecture that making regression adjustments will get us out of the Pascal’s Mugging problem, even if we have unbounded utility functions.)

  Though I agree that intuition can be a valuable tool when trying to sanity check an expected value calculation, I am hesitant to rely too heavily on it. Things like scope insensitivity and ambiguity aversion could easily make me unreasonably queasy about relying a perfectly reasonable expected value calculation.

  Finally, I classify several of the arguments in this post as “perfect world” arguments because they involve thinking a lot about what would happen if everyone behaved in a certain kind of way. I don’t want to rest too much weight on these arguments because my behavior doesn’t causally or acausally affect the way enough people would behave in order for these arguments to be directly relevant to my decisions. Even if I accepted perfect world arguments, some of these arguments appear not to work. For example, if all donors were rational altruists, and that was common knowledge, then charities that were effective would have a strong incentive to provide evidence of their effectiveness. If some charity refused to share information, that would be very strong evidence that the charity was not effective. So it doesn’t seem to be true, as Holden claims, that if everyone was totally reliant on explicit expected value calculations, we’d all give to charities about which we have very little information. (Deciding not to be totally transparent is not such good evidence now, since donors are far from being rational altruists.)

  Though I have expressed mostly disagreement, I think Holden’s post is very good and I’m glad that he made it.

• Nick_Beckstead 29 Jul 2013 10:11 UTC

  28 points

  on: Why I’m Skep­ti­cal About Un­proven Causes (And You Should Be Too)

  A few comments and requests for clarification:

  1. Are you only talking about donations? Do you think it would also be a mistake to work on speculative causes? (That seems much different in that there are many more opportunities for learning from working on a cause than from donating to it.) I think you are on much stronger ground with claims about donations, though I think it can make sense for other people with the right kind of information to fund other opportunities. E.g., think about early GiveWell. I wouldn’t want to buy into something which said giving to early GiveWell was a bad idea, even for people who had a lot of information about the project. I think some people may be in a similar position for funding some early EA orgs, and they shouldn’t be discouraged from funding them.

  2. What counts as a “speculative cause”? Meta-research? Political advocacy? Education for talented youth? Climate change? Prioritization work? Funding early GiveWell? Funding 80K? Anyone who says the thing they are doing is somehow improving very long-run outcomes? Anything that hasn’t been recommended by GiveWell? Anything that hasn’t been proven to work with RCT-quality evidence? Asking for a definition is probably unfair, but maybe a few illustrative examples of non-speculative causes and speculative causes would be helpful.

  3. If what you care about is the far future, why does AMF get to count as a “non-speculative” giving opportunity? We have very little idea how much AMF improves very long-run outcomes in comparison with other things, little effort has gone into studying this, and it seems many of your arguments that we are bad at understanding long-run effects apply.

  On this last point, I think it may make sense to say something like, “We are very bad at saying how much various changes (saving a life in the US, saving a life in Africa, boosting GDP in the US by X, boosting GDP in an African country by X, etc.) improve long-run outcomes. But we know that AMF stands out for creating these changes in a proven, cost-effective way. And I think the intermediate changes AMF is making are good for long-run outcomes, and not clearly less important than other intermediate changes we could be making. So that makes it the best donation target even by far future standards.” I just think this is a highly tentative position, and one that could easily change with further thought and research, and should be acknowledged as such.

• Nick_Beckstead 15 Mar 2013 23:56 UTC

  21 points

  on: Bayesian Ad­just­ment Does Not Defeat Ex­is­ten­tial Risk Charity

  Thank you for writing this post. I feel that additional discussion of these ideas is valuable, and that this post adds to the discussion.

  Note about my comment below: Though I’ve spoken with Holden about these issues in the past, what I say here is what I think, and shouldn’t be interpreted as his opinion.

  I don’t think Holden’s arguments are intended to show that existential risk is not a promising cause. To the contrary, global catastrophic risk reduction is one of GiveWell Labs’ priority causes. I think his arguments are only intended to show that one can’t appeal to speculative explicit expected value calculations to convincingly argue that targeted existential risk reduction is the best area to focus on. This perspective is much more plausible than the view that these arguments show that existential risk is not the best cause to investigate.

  I believe that Holden’s position becomes more plausible with the following two refinements:

  • Define the prior over good accomplished in terms of “lives saved, together with all the ripple effects of saving the lives.” By “ripple effects,” I mean all the indirect effects of the action, including speeding up development, reducing existential risk, or having other lasting impacts on the distant future.

  • Define the prior in terms of expected good accomplished, relative to “idealized probabilities,” where idealized probabilities are the probabilities we’d have given the available evidence at the time of the intervention, were we to construct our views in a way that avoided procedural errors (such as the influence of various biases, calculation errors, formulating the problem incorrectly).

  When you do the first thing, it makes the adjustment play out rather differently. For instance, I believe the following would not be true:

  As we’ve seen, Karnofsky’s toy examples use extreme priors, and these priors would entail a substantial adjustment to EV estimates for existential risk charities. This adjustment would in turn be sufficient to alter existential risk charities from good ideas to bad ideas.

  The reason is that if there is a decent probability of humanity having a large and important influence on the far future, ripple effects could be quite large. If that’s true, targeted existential risk reduction—meaning efforts to reduce existential risk which focus on it directly—would not necessarily have many orders of magnitude greater effects on the far future than activities which do not focus on existential risk directly.

  For similar reasons, I believe that Carl Shulman’s “Charity Doomsday Argument” would not go through if one follows the first suggestion. If ordinary actions can shape the far future as well, Holden’s framework doesn’t suggest that humanity will have a cramped future.

  If we adopt the second suggestion, defining the prior over expected good accomplished, pointing to specific examples of highly successful interventions in the past does not clearly refute a narrow prior probability distribution. We have to establish, in addition, that given what people knew at the time, these interventions had highly outsized expected returns. This is somewhat analogous to the way in which pointing to specific stocks which had much higher returns than other stocks does not refute the efficient markets hypothesis; one has to show that, in the past, those stocks were knowably underpriced. A normal or log-normal prior over expected returns may be refuted still, but a refutation would be more subtle.

  A couple of other points seem relevant as well, if one takes the above on board. First, as the “friend in a foreign country” example illustrates, a very low prior probability in a claim does not necessarily mean that the claim is unbelievable in practice. I believe that every time someone reads a newspaper, they can justifiably attain high credence in specific hypotheses, which, prior to reading the newspaper, had extremely low prior probabilities. Something similar may be true when specific novel scientific hypotheses, such as the ideal gas law, are discovered. So it seems that even if one adopts a fairly extreme prior, it wouldn’t have to be impossible to convince you that humanity would have a very large influence on the far future, or that something would actually reduce existential risk.

  Finally, I’d like to comment on this idea:

  Increased economic growth could have effects not just on timing, but on safety itself. For example, economic growth could increase existential risk by speeding up dangerous technologies more quickly than society can handle them safely, or it could decrease existential risk by promoting some sort of stability. It could also have various small but permanent effects on the future. Still, it would seem to be a fairly major coincidence if the policy of saving people’s lives in the Third World were also the policy that maximized safety. One would at least expect to see more effect from interventions targeted specifically at speeding up economic growth. An approach to foreign aid aimed at maximizing growth effects rather than near-term lives or DALYs saved would probably look quite different. Even then, it’s hard to see how economic growth could be the policy that maximized safety unless our model of what causes safety were so broken as to be useless.

  There is a spectrum of strategies for shaping the far future that ranges from the very targeted (e.g., stop that asteroid from hitting the Earth) to very broad (e.g., create economic growth, help the poor, provide education programs for talented youth), with options like “tell powerful people about the importance of shaping the far future” in between. The limiting case of breadth might be just optimizing for proximate benefits or for speeding up development. I suspect that global health is probably not the best place on this spectrum to be, but I don’t find that totally obvious. I think it’s a very interesting question where on this spectrum we should prefer to be, other things being equal. My sense is that many people on LessWrong think that we should be on the highly targeted end of this spectrum. I am highly uncertain about this issue, and I’d be interested in seeing stronger arguments for or against this view.

  What links here?

  • JonahS's comment on Ro­bust­ness of Cost-Effec­tive­ness Es­ti­mates and Philanthropy by JonahS (24 May 2013 16:56 UTC; 2 points)

Re­view of stud­ies says you can de­crease mo­ti­vated cog­ni­tion through self-affirmation

• Nick_Beckstead 1 Feb 2012 3:07 UTC

  19 points

  on: Ex­is­ten­tial risk for non-consequentialists

  Most philosophers who are non-consequentialists are pluralists that think consequences are very important, so they can still use standard arguments to support the idea that reducing x-risk is important. A lot of non-consequentialism is about moral prohibitions rather than prescriptions, so I suspect most of it would have little to say about altruistic considerations. And of course a lot of it is loose, vague, and indeterminate, so it would be hard to draw out any comparative claims anyway.

• Nick_Beckstead 2 Dec 2013 18:02 UTC

  18 points

  on: A cri­tique of effec­tive altruism

  I’d like to see more critical discussion of effective altruism of the type in this post. I particularly enjoyed the idea of “pretending to actually try.” People doing sloppy thinking and then making up EA-sounding justifications for their actions is a big issue.

  As Will McAskill said in a Facebook comment, I do think that a lot of smart people in the EA movement are aware of the issues you’re bringing up and have chosen to focus on other things. Big picture, I find claims like “your thing has problem X so you need to spend more resources on fixing X” more compelling when you point to things we’ve been spending time on and say that we should have done less of those things and more of the thing you think we should have been doing. E.g., I currently spend a lot of my time on research, advocacy, and trying to help improve 80,000 Hours and I’d be pretty hesitant to switch to writing blogposts criticizing mistakes that people in the EA community commonly make, though I’ve considered doing so and agree this would be help address some of the issues you’ve identified. But I would welcome more of that kind of thing.

  I disagree with your perspective that the effective altruism movement has underinvested in research into population ethics. I wrote a PhD thesis which heavily featured population ethics and aimed at drawing out big-picture takeaways for issues like existential risk. I wouldn’t say I settled all the issues, but I think we’d make more progress as movement if we did less philosophy and more basic fact-finding of the kind that goes into GiveWell shallow cause overviews.

  Disclosure: I am a Trustee for the Centre for Effective Altruism and I formerly worked at GiveWell as a summer research analyst.

• Nick_Beckstead 11 Aug 2013 7:46 UTC

  18 points

  in reply to: JonahS’s comment on: Com­mon sense as a prior

  The intrinsic interest of the question of interpretation of quantum mechanics

  The question of what quantum mechanics means has been considered one of the universe’s great mysteries. As such, people interested in physics have been highly motivated to understand it. So I think that the question is privileged relative to other questions that physicists would have opinions on — it’s not an arbitrary question outside of the domain of their research accomplishments.

  My understanding is that the interpretation of QM is (1) not regarded as a very central question in physics, being seen more as a “philosophy” question and being worked on to a reasonable extent by philosophers of physics and physicists who see it as a hobby horse, (2) is not something that physics expertise—having good physical intuition, strong math skills, detailed knowledge of how to apply QM on concrete problems—is as relevant for as many other questions physicists work on, and (3) is not something about which there is an extremely enormous amount to say. These are some of the main reasons I feel I can update at all from the expert distribution of physicists on this question. I would hardly update at all from physicist opinions on, say, quantum gravity vs. string theory, and I think it would basically be crazy for me to update substantially in one direction or the other if I had comparable experience on that question.

  [ETA: As evidence of (1), I might point to the prevalence of the “shut up and calculate” mentality which seems to have been reasonably popular in physics for a while. I’d also point to the fact that Copenhagen is popular but really, really, really, really not good. And I feel that this last claim is not just Nick Beckstead’s idiosyncratic opinion, but the opinion of every philosopher of physics I have ever spoken with about this issue.]

  What links here?

  • Nick_Beckstead's comment on Com­mon sense as a prior by Nick_Beckstead (11 Aug 2013 19:12 UTC; 13 points)

• Nick_Beckstead 12 Jun 2012 14:14 UTC

  18 points

  on: Re­ply to Holden on ‘Tool AI’

  There are two ways to read Holden’s claim about what happens if 100 experts check the proposed FAI safety proof. On one reading, Holden is saying that if 100 experts check it and say, “Yes, I am highly confident that this is in fact safe,” then activating the AI kills us all with 90% probability. On the other reading, Holden is saying that even if 100 experts do their best to find errors and say, “No, I couldn’t identify any way in which this will kill us, though that doesn’t mean it won’t kill us,” then activating the AI kills us all with 90% probability. I think the first reading is very implausible. I don’t believe the second reading, but I don’t think it’s obviously wrong. I think the second reading is the more charitable and relevant one.

• Nick_Beckstead 2 Dec 2013 16:49 UTC

  17 points

  in reply to: MichaelVassar’s comment on: A cri­tique of effec­tive altruism

  I would love to hear about your qualms with the EA movement if you ever want to have a conversation about the issue.

  Edited: When I first read this, I thought you were saying you hadn’t brought these problems up with me, but re-reading it it sounds like you tried to raise these criticisms with me. This post has a Vassar-y feel to it but this is mostly criticism I wouldn’t say I’d heard from you, and I would have guessed your criticisms would be different. In any case, I would still be interested in hearing more from you about your criticisms of EA.

• Nick_Beckstead 7 Feb 2012 17:35 UTC

  17 points

  on: The Hu­man’s Hid­den Utility Func­tion (Maybe)

  What’s the evidence that this is the “leading theory of choice in the human brain”? (I am not saying I have evidence that it isn’t, but it’s important for this post that some large relevant section of the scientific community thinks this theory is awesome.)

• Nick_Beckstead 11 Aug 2013 7:16 UTC

  16 points

  in reply to: lukeprog’s comment on: Com­mon sense as a prior

  I feel the main disanalogy with the Muslim theology case is that elite common sense does not regard top Muslim philosophers as having much comparative expertise on the question of whether Allah exists, but they would regard top physicists as having very strong comparative expertise on the interpretation of quantum mechanics. By this I mean that elite common sense would generally substantially defer to the opinions of the physicists but not the Muslim philosophers. This disanalogy is sufficiently important to me that I find the overall argument by analogy highly non-compelling.

  I note that there are some meaningful respects in which elite common sense would regard the Muslim philosophers as epistemic authorities. They would recognize their authority as people who know about what the famous arguments for Allah’s existence and nature are, what the famous objections and replies are, and what has been said about intricacies of related metaphysical questions, for example.

• Nick_Beckstead 1 Jun 2013 11:40 UTC

  16 points

  in reply to: lukeprog’s comment on: A dis­claimer about my effec­tive philan­thropy posts (in con­nec­tion with as­tro­nom­i­cal waste)

  In philosophy, a common saying is “It’s best not to try to do philosophy all at once.” Very much in the spirit of this comment and Jonah’s point #3.

• Nick_Beckstead 9 Jul 2013 20:46 UTC

  15 points

  on: “Can we know what to do about AI?”: An Introduction

  Some possible sources of examples, many I haven’t checked and many lacking precise details:

  • CFCs?

  • Asteroid tracking (if they weren’t planning on asteroids in the next ten years, seems like natural risks with average return periods > 10 years could be a good source of examples, though I feel this maybe isn’t what you are asking for because it is so statistical rather than scenario-based)

  • Legislation where you don’t expect the legislation to have effects for more than 10 years. Some international treaties/​conventions might be examples

  • Religion examples. In 1833 William Miller predicted the return of Jesus Christ in 1843 and people took specific actions. [Edited: original date written was 1943, which was a typo]

  • Space exploration examples? Maybe these things required far advance planning. Voyager may be an example?

  • Futures markets? How far do they go out?

  • Simon-Ehrlich wager

  • Sleeper agents? I have heard stories that some Russian agents were sleepers in the US for greater than 10 years, waiting to strike at the right time if war broke out. May want to check on it.

  • Eugenics people

  • People conducting very long longitudinal studies

  There are some mundane examples that fit your description but I don’t think are what you are asking for. E.g., people working on buildings that take more than 10 years to complete, people buying something for a young daughter and saving it for when she gets married (I guess this used to be a thing), people getting mortgages, people saving for retirement, people studying anatomy when 16 in hopes of becoming a doctor, making a trust fund for your kid.

• Nick_Beckstead 15 May 2012 2:23 UTC

  14 points

  in reply to: PhilGoetz’s comment on: Thoughts on the Sin­gu­lar­ity In­sti­tute (SI)

  I would be interested to see if you could link to posts where you made versions of these objections.

  What links here?

  • Holden’s Ob­jec­tion 1: Friendli­ness is dangerous by PhilGoetz (18 May 2012 0:48 UTC; 13 points)

• Nick_Beckstead 11 Aug 2013 19:12 UTC

  13 points

  in reply to: Eliezer Yudkowsky’s comment on: Com­mon sense as a prior

  I haven’t fully put together my thoughts on this, but it seems like a bad test to “break someone’s trust in a sane world” for a number of reasons:

  • this is a case where all the views are pretty much empirically indistinguishable, so it isn’t an area where physicists really care all that much

  • since the views are empirically indistinguishable, it is probably a low-stakes question, so the argument doesn’t transfer well to breaking our trust in a sane world in high-stakes cases; it makes sense to assume people would apply more rationality in cases where more rationality pays off

  • as I said in another comment, MWI seems like a case where physics expertise is not really what matters, so this doesn’t really show that the scientific method as applied by physicists is broken; it seems it at most it shows that physics aren’t good at questions that are essentially philosophical; it would be much more persuasive if you showed that e.g., quantum gravity was obviously better than string theory and only 18% of physicists working in the relevant area thought so

  [Edited to add a missing “not”]

• Nick_Beckstead 6 Jun 2013 17:40 UTC

  12 points

  on: Tiling Agents for Self-Mod­ify­ing AI (OPFAI #2)

  I am very glad to see MIRI taking steps to list open problems and explain why those problems are important for making machine intelligence benefit humanity.

  I’m also struggling to see why this Lob problem is a reasonable problem to worry about right now (even within the space of possible AI problems). Basically, I’m skeptical that this difficulty or something similar to it will arise in practice. I’m not sure if you disagree, since you are saying you don’t think this difficulty will “block AI.” And if it isn’t going to arise in practice (or something similar to it), I’m not sure why this should be high on the priority list of general AI issues to think about it (edited to add: or why working on this problem now should be expected to help machine intelligence develop in a way that benefits humanity).

  Some major questions I have are:

  • What are some plausible concrete examples of self-modifications where Lob issues might cause you to stumble? I promise not to interpret your answer as “Eliezer says this is probably going to happen.”

  • Do you think that people building AGI in the future will stumble over Lob issues if MIRI doesn’t work on those issues? If so, why?

  Part of where I’m coming from on the first question is that Lobian issues only seem relevant to me if you want to argue that one set of fundamental epistemic standards is better than another, not for proving that other types of software and hardware alterations (such as building better arms, building faster computers, finding more efficient ways to compress your data, finding more efficient search algorithms, or even finding better mid-level statistical techniques) would result in more expected utility. But I would guess that once you have an agent operating with a minimally decent fundamental epistemic standards, you just can’t prove that altering the agent’s fundamental epistemic standards would result in an improvement. My intuition is that you can only do that when you have an inconsistent agent, and in that situation it’s unclear to me how Lobian issues apply.

  Part of where I’m coming from on the second question is that evolutionary processes made humans who seem capable of overcoming putative Lobian obstacles to self-modification. See my other comment for more detail. The other part has to do with basic questions about whether people will adequately prepare for AI by default.

  What links here?

  • Nick_Beckstead's comment on Tiling Agents for Self-Mod­ify­ing AI (OPFAI #2) by Eliezer Yudkowsky (7 Jun 2013 12:49 UTC; 3 points)