Plugging the concept of viatopia again because it’s a nice explanatory partner to a Long Reflection or Long Self-Correction. A viatopia is a world that will, with near-certainty, perform a Long Correction and then implement the desired future. Or, a Long Correction is what you would do if you found yourself in a viatopia when your current sentient inhabitants are not yet ethically developed enough to commit to the project of engineering the future.
(I’d also be very happy to see someone coin another term that means what is gestured at by viatopia.)
It’s not a strict definition, because a society undergoing the Long Reflection might be more insecure than what is necessary to qualify as a viatopia.
I’d like to see people having the public conversation: do we need to achieve existential security before we slow everything down and do a Long Reflection? And more generally, at what level of technology would the world be comfortable stopping?
It’s possible that AI Pause advocates will need to argue for a concrete technological vision, something like: “With the level of AI we have today in the year 2031, just applying the current models will continue to produce breakthroughs in medicine, manufacturing, etc. enough to cure cancer and bring the world economic baseline up to an American middle-class outcome. We are committed to continuing to fund this progress while still restricting frontier AI development.”
I think that having that conversation at least lets people imagine a concrete world state from which a Long Reflection can happen. And then there’s an easier way to point at what is being talked about: it’s the answer to the question “and then what?”
Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Yet almost no one has articulated a positive vision for what comes after superintelligence. Few people are even asking, “What if we succeed?” Even fewer have tried to answer.
Same post also says it’s meant to be a generalization of Long Reflection:
Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
How can humanity have the greatest chance of achieving its potential? I think that at the highest level we should adopt a strategy proceeding in three phases:2
Reaching Existential Security
The Long Reflection
Achieving Our Potential
On this view, the first great task for humanity is to reach a place of safety—a place where existential risk is low and stays low. I call this existential security.
It has two strands. Most obviously, we need to preserve humanity’s potential, extracting ourselves from immediate danger so we don’t fail before we’ve got our house in order. This includes direct work on the most pressing existential risks and risk factors, as well as near-term changes to our norms and institutions.
But we also need to protect humanity’s potential—to establish lasting safeguards that will defend humanity from dangers over the longterm future, so that it becomes almost impossible to fail.3 Where preserving our potential is akin to fighting the latest fire, protecting our potential is making changes to ensure that fire will never again pose a serious threat.4 This will involve major changes to our norms and institutions (giving humanity the prudence and patience we need), as well as ways of increasing our general resilience to catastrophe. This needn’t require foreseeing all future risks right now. It is enough if we can set humanity firmly on a course where we will be taking the new risks seriously: managing them successfully right from their onset or sidestepping them entirely.
Note that existential security doesn’t require the risk to be brought down to zero. That would be an impossible target, and attempts to achieve it may well be counter-productive. What humanity needs to do is bring this century’s risk down to a very low level, then keep gradually reducing it from there as the centuries go on. In this way, even though there may always remain some risk in each century, the total risk over our entire future can be kept small.5 We could view this as a form of existential sustainability. Futures in which accumulated existential risk is allowed to climb toward 100 percent are unsustainable. So we need to set a strict risk budget over our entire future, parceling out this non-renewable resource with great care over the generations to come.
Ultimately, existential security is about reducing total existential risk by as many percentage points as possible. Preserving our potential is helping lower the portion of the total risk that we face in the next few decades, while protecting our potential is helping lower the portion that comes over the longer run. We can work on these strands in parallel, devoting some of our efforts to reducing imminent risks and some to building the capacities, institutions, wisdom and will to ensure that future risks are minimal.6
A key insight motivating existential security is that there appear to be no major obstacles to humanity lasting an extremely long time, if only that were a key global priority. As we saw in Chapter 3, we have ample time to protect ourselves against natural risks: even if it took us millennia to resolve the threats from asteroids, supervolcanism and supernovae, we would incur less than one percentage point of total risk.
The greater risk (and tighter deadline) stems from the anthropogenic threats. But being of humanity’s own making, they are also within our control. Were we sufficiently patient, prudent and coordinated, we could simply stop imposing such risks upon ourselves. We would factor in the hidden costs of carbon emissions (or nuclear weapons) and realize they are not a good deal. We would adopt a more mature attitude to the most radical new technologies—devoting at least as much of humanity’s brilliance to forethought and governance as to technological development.
In the past, the survival of humanity didn’t require much conscious effort: our past was brief enough to evade the natural threats and our power too limited to produce anthropogenic threats. But now our longterm survival requires a deliberate choice to survive. As more and more people come to realize this, we can make this choice. There will be great challenges in getting people to look far enough ahead and to see beyond the parochial conflicts of the day. But the logic is clear and the moral arguments powerful. It can be done.
If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.7
I just learned about what a viatopia was from your linked article, but I certainly agree that there should be more discussion on this. In general, I think that focusing too much on terminal values can lead to internal disagreement on matters which don’t bear much significance in the present, especially when most of us already agree on certain things that are desirable in the medium-term (such as the ones you’ve listed e.g. curing cancer).
And more generally, at what level of technology would the world be comfortable stopping?
I’m not entirely sure what year I think would be optimal, but since frontier LLMs see upgrades every few months, I feel as though we only scratch the surface of what we can do with our current models before the next generation is already out. As such, I think that even our current LLMs + improved knowledge on how to make use of them (gathered over years) could already offer significant help toward achieving those medium-term goals.
In any case, it would be nice if viatopia had more attention, as it seems like a promising way to do “one thing at a time” and potentially reduce the confusion (and risk of error) of trying to develop full-length plans from the outset.
(I know that those conversations have happened here and in plenty of living rooms. I’d be very interested in links to anything one degree more public, if anyone is aware of them!)
Plugging the concept of viatopia again because it’s a nice explanatory partner to a Long Reflection or Long Self-Correction. A viatopia is a world that will, with near-certainty, perform a Long Correction and then implement the desired future. Or, a Long Correction is what you would do if you found yourself in a viatopia when your current sentient inhabitants are not yet ethically developed enough to commit to the project of engineering the future.
(I’d also be very happy to see someone coin another term that means what is gestured at by viatopia.)
It’s not a strict definition, because a society undergoing the Long Reflection might be more insecure than what is necessary to qualify as a viatopia.
I’d like to see people having the public conversation: do we need to achieve existential security before we slow everything down and do a Long Reflection? And more generally, at what level of technology would the world be comfortable stopping?
It’s possible that AI Pause advocates will need to argue for a concrete technological vision, something like: “With the level of AI we have today in the year 2031, just applying the current models will continue to produce breakthroughs in medicine, manufacturing, etc. enough to cure cancer and bring the world economic baseline up to an American middle-class outcome. We are committed to continuing to fund this progress while still restricting frontier AI development.”
I think that having that conversation at least lets people imagine a concrete world state from which a Long Reflection can happen. And then there’s an easier way to point at what is being talked about: it’s the answer to the question “and then what?”
Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Same post also says it’s meant to be a generalization of Long Reflection:
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
@Toby_Ord @wdmacaskill in case they want to weigh in on this.
I just learned about what a viatopia was from your linked article, but I certainly agree that there should be more discussion on this. In general, I think that focusing too much on terminal values can lead to internal disagreement on matters which don’t bear much significance in the present, especially when most of us already agree on certain things that are desirable in the medium-term (such as the ones you’ve listed e.g. curing cancer).
I’m not entirely sure what year I think would be optimal, but since frontier LLMs see upgrades every few months, I feel as though we only scratch the surface of what we can do with our current models before the next generation is already out. As such, I think that even our current LLMs + improved knowledge on how to make use of them (gathered over years) could already offer significant help toward achieving those medium-term goals.
In any case, it would be nice if viatopia had more attention, as it seems like a promising way to do “one thing at a time” and potentially reduce the confusion (and risk of error) of trying to develop full-length plans from the outset.
(I know that those conversations have happened here and in plenty of living rooms. I’d be very interested in links to anything one degree more public, if anyone is aware of them!)