I think the discussion wouldn’t have to be like “here’s a crazy plan”.
I think there could have been something more like: “Important fact to understand about the situation: Even if superintelligence comes within the next 10 years, it’s pretty likely that sub-ASI systems will have had a huge impact on the world by then — changing the world in a few-year period more than any technology ever has changed the world in a few-year period. It’s hard to predict what this would look like [easy calls, hard calls, etc]. Some possible implications could be: [long list: …, automated alignment research, AI-enabled coordination, people being a lot more awake to the risks of ASI, lots of people being in relationships with AIs and being supportive of AI rights, not-egregiously-misaligned AIs that are almost as good at bio/cyber/etc as the superintelligences...]. Some of these things could be helpful, some could be harmful. Through making us more uncertain about the situation, this lowers our confidence that everyone will die. In particular, some chance that X, Y, Z turns out really helpful. But obviously, if we see humanity as an agent, it would be a dumb plan for humanity to just assume that this crazy, hard-to-predict mess will save the whole situation.”
I.e. it could be presented as an important thing to understand about the strategic situation rather than as a proposed plan.
Modulo the sentence “this lowers our confidence that everyone will die”, since I don’t think it’s what they believe, or what I believe, though it’s messy. My guess is this period is also majorly responsible for increasing risks by creating tons of economic momentum that then make it hard to stop when you get to really risky AI, and so my best guess is the overall technological diffusion will make things riskier instead of less, though I don’t have a strong take either way.
Having the economic incentives plus other things explained, and being like “and yep, this seems like it might make things worse or better, it makes it harder to be confident about how things go, though the core difficulty remains”, would be good.
Through making us more uncertain about the situation, this lowers our confidence that everyone will die.
This seems to rely on the assumption that “there’s nowhere to go but up”: that we’re pretty certain of doom, so wildcards in the future can only make us less certain. But I don’t think that works. Wildcards in the future can also increase s-risks, and there’s no limit how bad things can get, potentially much worse than extinction.
I think the discussion wouldn’t have to be like “here’s a crazy plan”.
I think there could have been something more like: “Important fact to understand about the situation: Even if superintelligence comes within the next 10 years, it’s pretty likely that sub-ASI systems will have had a huge impact on the world by then — changing the world in a few-year period more than any technology ever has changed the world in a few-year period. It’s hard to predict what this would look like [easy calls, hard calls, etc]. Some possible implications could be: [long list: …, automated alignment research, AI-enabled coordination, people being a lot more awake to the risks of ASI, lots of people being in relationships with AIs and being supportive of AI rights, not-egregiously-misaligned AIs that are almost as good at bio/cyber/etc as the superintelligences...]. Some of these things could be helpful, some could be harmful. Through making us more uncertain about the situation, this lowers our confidence that everyone will die. In particular, some chance that X, Y, Z turns out really helpful. But obviously, if we see humanity as an agent, it would be a dumb plan for humanity to just assume that this crazy, hard-to-predict mess will save the whole situation.”
I.e. it could be presented as an important thing to understand about the strategic situation rather than as a proposed plan.
I agree that a section like this would be good!
Modulo the sentence “this lowers our confidence that everyone will die”, since I don’t think it’s what they believe, or what I believe, though it’s messy. My guess is this period is also majorly responsible for increasing risks by creating tons of economic momentum that then make it hard to stop when you get to really risky AI, and so my best guess is the overall technological diffusion will make things riskier instead of less, though I don’t have a strong take either way.
Having the economic incentives plus other things explained, and being like “and yep, this seems like it might make things worse or better, it makes it harder to be confident about how things go, though the core difficulty remains”, would be good.
This seems to rely on the assumption that “there’s nowhere to go but up”: that we’re pretty certain of doom, so wildcards in the future can only make us less certain. But I don’t think that works. Wildcards in the future can also increase s-risks, and there’s no limit how bad things can get, potentially much worse than extinction.