I agree with most of the points presented in the post to various extents, but I don’t think these arguments actually support the post’s conclusion.
If you take only one thing from this post, take this:any theory of change that falls to one of these competitive pressures is completely useless. The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.
Doesn’t this plan (effectively an outright ASI ban) fall to competitive pressures between labs and nations just as easily as the others?
In fact, I’d say it’s a lot more brittle. On a long enough timeline this plan would fail if any secret ASI project exists somewhere in the world, which due to the very competitive pressures outlined in the post, is almost certain to exist, with the bonus that the winners in this scenario would be guaranteed to be a group willing to start a secret ASI project presumably against international law.
In the meantime, the three filters that this post mention all involve a smaller number of actors. In the first filter, only a few countries can start a global nuclear war, and in the other two filters (in the case where they occur) only one AI or group of people will decide the fate of humanity.
Also, a tangential observation: some of the arguments presented in the post seem to suggest that on the margin, it would be better to race faster in order to create aligned ASI before either your own government or their nuke-happy adversaries realize what’s going on. I don’t know what to make of this observation.
If governments are situationally aware, then they will be aware of the risk of secret AI projects and act on it, given that such projects are tantamount to declaring war (see filter 1). No one is going to allow the possibility of a small actor running such a project. The equilibrium can only either be actually solving the common knowledge problem for real (however costly this might be), or existential war. Though I suppose the alternative is, if the verification is not good enough, but the actors involved think it is, then there’s room for a secret project to happen.
I agree, though I think the risk of a secret ASI project being successful with limited resources probably increases significantly the longer the timeline, based on a basic fragile-world extrapolation. The group of actors with capability to reach ASI will expand from major powers to middle powers and eventually small nations (some of which may be nuclear pariah states) and private groups. All it takes is one group to pull off a successful secret project (or possibly a not-so-secret one if it’s run by a country with enough nukes) to break the equilibrium.
Assuming the above, then an AI pause is doable, but not an indefinite ASI ban. Eventually the risk of secret projects is large enough that the world as a whole needs to resume AI development to stay ahead of one if it existed.
FWIW, I think this is a problem for decades in the future. I agree that an ASI ban doesn’t solve the problem indefinitely, but we’ll have extra decades to figure out what to do.
I agree with most of the points presented in the post to various extents, but I don’t think these arguments actually support the post’s conclusion.
Doesn’t this plan (effectively an outright ASI ban) fall to competitive pressures between labs and nations just as easily as the others?
In fact, I’d say it’s a lot more brittle. On a long enough timeline this plan would fail if any secret ASI project exists somewhere in the world, which due to the very competitive pressures outlined in the post, is almost certain to exist, with the bonus that the winners in this scenario would be guaranteed to be a group willing to start a secret ASI project presumably against international law.
In the meantime, the three filters that this post mention all involve a smaller number of actors. In the first filter, only a few countries can start a global nuclear war, and in the other two filters (in the case where they occur) only one AI or group of people will decide the fate of humanity.
Also, a tangential observation: some of the arguments presented in the post seem to suggest that on the margin, it would be better to race faster in order to create aligned ASI before either your own government or their nuke-happy adversaries realize what’s going on. I don’t know what to make of this observation.
If governments are situationally aware, then they will be aware of the risk of secret AI projects and act on it, given that such projects are tantamount to declaring war (see filter 1). No one is going to allow the possibility of a small actor running such a project. The equilibrium can only either be actually solving the common knowledge problem for real (however costly this might be), or existential war. Though I suppose the alternative is, if the verification is not good enough, but the actors involved think it is, then there’s room for a secret project to happen.
I agree, though I think the risk of a secret ASI project being successful with limited resources probably increases significantly the longer the timeline, based on a basic fragile-world extrapolation. The group of actors with capability to reach ASI will expand from major powers to middle powers and eventually small nations (some of which may be nuclear pariah states) and private groups. All it takes is one group to pull off a successful secret project (or possibly a not-so-secret one if it’s run by a country with enough nukes) to break the equilibrium.
Assuming the above, then an AI pause is doable, but not an indefinite ASI ban. Eventually the risk of secret projects is large enough that the world as a whole needs to resume AI development to stay ahead of one if it existed.
FWIW, I think this is a problem for decades in the future. I agree that an ASI ban doesn’t solve the problem indefinitely, but we’ll have extra decades to figure out what to do.