Thanks, Toby, I highly appreciate your perspective. I’m honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn’t talk about most of the flaws I listed, except something like my #1: “The study of the positive is at a much earlier stage of development.” I’m curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity’s flaws imply that in the early stages there is seemingly no way to ensure that the process doesn’t go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of “permanently destroy humanity’s future potential”.)
Finally, I wonder if you can clear up something I’ve always wondered about the Long Reflection—was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)
I’d say my views have evolved, but are quite continuous with my original views. More a sharpening and changes of emphasis than a disagreement with the stated view in The Precipice. One reason I didn’t express things more like your (2)–(8) is that I hadn’t explicitly thought about many of them. I’ve been appreciating your writing on this here and your podcast interview. I’m not 100% convinced of your position (e.g. I’m not even quite sure how to make ‘humans are bad at philosophy’ into a meaningful statement), but I’m very sympathetic to it and see your evolving views on these matters as some of the most interesting writing about humanity’s deeper predicament.
One reason I’m optimistic is because of humanity’s progress over the last three thousand years (often in very difficult circumstances where the number of people with the ability and freedom to think about these things was often tiny and disconnected). My claim is basically that if you remove the traps where we could get stuck and then give the process long enough, we should get there. This is a reason why I think it could take a long time (I’d think ideally thousands of years, though I’ve calculated millions of years as an upper bound for how long we have before we start losing appreciable fractions of the future).
You could well be right about the existence of various traps during the Long Reflection. I still think my ordering is roughly right — put out the current fires and develop good fire defenses before settling down to think things through. It is partly because I don’t think we should set up a formal structure for the process that I’m not as concerned about the initial setup dooming it. e.g. we didn’t have a formal structure for philosophical progress over the last three thousand years. I’d expect traps to be slow moving enough that one can learn about them and avoid them during the process, though this still does mean that the structure of First eliminate all traps Second do Long Reflection is a simplification.
I did envisage it as being pre-ASI and think that Will did too. This is partly because the risk we undergo in getting to ASI doesn’t have to be borne. In an ideally managed world we’d instead develop lots of narrower and safer tools to help (think Wikipedia and theorem provers and Mathematica). In my view, in the Long Reflection we delay potentially existential technologies/choices until we can be sure the risk is low or that it can’t be avoided or work out how to make a version with low risk. Then we can adopt those that we’ve assessed as worth adopting. (Some people like Robin Hanson argue against a straw man that the Long Reflection would ban all new technology/choices until proven safe or until the entire reflection is completed, but neither Will nor I have advanced such a proposal. Instead the focus is on radical technologies and choices, such as ASI or a diaspora of humanity throughout the galaxy.)
Thanks, Toby, I highly appreciate your perspective. I’m honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn’t talk about most of the flaws I listed, except something like my #1: “The study of the positive is at a much earlier stage of development.” I’m curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity’s flaws imply that in the early stages there is seemingly no way to ensure that the process doesn’t go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of “permanently destroy humanity’s future potential”.)
Finally, I wonder if you can clear up something I’ve always wondered about the Long Reflection—was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)
I’d say my views have evolved, but are quite continuous with my original views. More a sharpening and changes of emphasis than a disagreement with the stated view in The Precipice. One reason I didn’t express things more like your (2)–(8) is that I hadn’t explicitly thought about many of them. I’ve been appreciating your writing on this here and your podcast interview. I’m not 100% convinced of your position (e.g. I’m not even quite sure how to make ‘humans are bad at philosophy’ into a meaningful statement), but I’m very sympathetic to it and see your evolving views on these matters as some of the most interesting writing about humanity’s deeper predicament.
One reason I’m optimistic is because of humanity’s progress over the last three thousand years (often in very difficult circumstances where the number of people with the ability and freedom to think about these things was often tiny and disconnected). My claim is basically that if you remove the traps where we could get stuck and then give the process long enough, we should get there. This is a reason why I think it could take a long time (I’d think ideally thousands of years, though I’ve calculated millions of years as an upper bound for how long we have before we start losing appreciable fractions of the future).
You could well be right about the existence of various traps during the Long Reflection. I still think my ordering is roughly right — put out the current fires and develop good fire defenses before settling down to think things through. It is partly because I don’t think we should set up a formal structure for the process that I’m not as concerned about the initial setup dooming it. e.g. we didn’t have a formal structure for philosophical progress over the last three thousand years. I’d expect traps to be slow moving enough that one can learn about them and avoid them during the process, though this still does mean that the structure of First eliminate all traps Second do Long Reflection is a simplification.
I did envisage it as being pre-ASI and think that Will did too. This is partly because the risk we undergo in getting to ASI doesn’t have to be borne. In an ideally managed world we’d instead develop lots of narrower and safer tools to help (think Wikipedia and theorem provers and Mathematica). In my view, in the Long Reflection we delay potentially existential technologies/choices until we can be sure the risk is low or that it can’t be avoided or work out how to make a version with low risk. Then we can adopt those that we’ve assessed as worth adopting. (Some people like Robin Hanson argue against a straw man that the Long Reflection would ban all new technology/choices until proven safe or until the entire reflection is completed, but neither Will nor I have advanced such a proposal. Instead the focus is on radical technologies and choices, such as ASI or a diaspora of humanity throughout the galaxy.)