Camp A for all the reasons you listed. I think the only safe path forward is one of earnest intent for mutual cooperation rather than control. Not holding my breath though.
I would have labelled Camp A as the control camp (control of the winner over everyone else) and Camp B as the mutual cooperation camp (since ending the race is the fruit of cooperation between nations). If we decide to keep racing for superintelligence, what does the finish line look like, in your view?
I think @rife is talking either about mutual cooperation betwen safety advocates and capabilities researchers, or mutual cooperation between humans and AIs.
Cooperation between humans and AIs rather than an attempt to control AIs. I think the race is going to happen regardless of who drops out of it. If those who are in the lead eventually land on mutual alignment, then we stand a chance. We’re not going to outsmart the AIs nor will we stay on control of them, nor should we.
I’m mostly “camp D[oomer]: we can’t stop, and we can’t control it, and this is bad, and so the future will be bad”; but the small portion of my hope that things go well relies on cooperation being solved better than it ever has been before. otherwise, I think faster and faster brain-sized-meme evolution is not going to go well for anything we care about. achieving successful, stable, defensible inter-meme cooperation seems … like a tall order, to put it mildly. the poem on my user page is about this, as are my pinned comments. to the degree camp A[ccelerate safety] can possibly make camp D wrong, I’m on board with helping figure out how to solve alignment in time, and I think camp B[an it]/B[uy time] are useful and important.
but like, let’s get into the weeds and have a long discussion in the comments here, if you’re open to it: how do we achieve a defensibly-cooperative future that doesn’t end up wiping out most humans and AIs? how do we prevent an AI that is willing to play defect-against-all, that doesn’t want anything like what current humans and AI want, from beating all of us combined? which research paths are you excited about to make cooperation durable?
Instead, we argue that we need a solution for preserving humanity and improving the future despite not having an easy solution of allowing gradual disempowerment coupled with single-objective beneficial AI...
The first question, one that is central to some discussions of long-term AI risk, is how can humanity stay in control after creating smarter-than-human AI?
But given the question, the answer is overdetermined. We don’t stay in control, certainly not indefinitely. If we build smarter than human AI, which is certainly not a good idea right now, at best we must figure out how we are ceding control. If nothing else, power-seeking AI will be a default, and will be disempowering—even if it’s not directly an existential threat. Even if we solve the problem of treachery robustly, and build an infantilizing vision of superintelligent personal assistants, over long enough time scales, it’s implausible that we not only build that race of more intelligent systems, but do not then cede any power. (And if we did, somehow, the implications of keeping systems that are increasingly intelligent in permanent bondage seems at best morally dubious.)
So, if we (implausibly) happen to be in a world of alignment-by-default, or (even more implausibly) find a solution to intent alignment and agree to create a super-nanny for humanity, what world would we want? Perhaps we use this power to collectively evolve past humanity—or perhaps the visions of pushing for transhumanism before ASI to allow someone, some group to stay in control are realized. Either way, what then for the humans?
Camp A for all the reasons you listed. I think the only safe path forward is one of earnest intent for mutual cooperation rather than control. Not holding my breath though.
I would have labelled Camp A as the control camp (control of the winner over everyone else) and Camp B as the mutual cooperation camp (since ending the race is the fruit of cooperation between nations). If we decide to keep racing for superintelligence, what does the finish line look like, in your view?
I think @rife is talking either about mutual cooperation betwen safety advocates and capabilities researchers, or mutual cooperation between humans and AIs.
Cooperation between humans and AIs rather than an attempt to control AIs. I think the race is going to happen regardless of who drops out of it. If those who are in the lead eventually land on mutual alignment, then we stand a chance. We’re not going to outsmart the AIs nor will we stay on control of them, nor should we.
I’m mostly “camp D[oomer]: we can’t stop, and we can’t control it, and this is bad, and so the future will be bad”; but the small portion of my hope that things go well relies on cooperation being solved better than it ever has been before. otherwise, I think faster and faster brain-sized-meme evolution is not going to go well for anything we care about. achieving successful, stable, defensible inter-meme cooperation seems … like a tall order, to put it mildly. the poem on my user page is about this, as are my pinned comments. to the degree camp A[ccelerate safety] can possibly make camp D wrong, I’m on board with helping figure out how to solve alignment in time, and I think camp B[an it]/B[uy time] are useful and important.
but like, let’s get into the weeds and have a long discussion in the comments here, if you’re open to it: how do we achieve a defensibly-cooperative future that doesn’t end up wiping out most humans and AIs? how do we prevent an AI that is willing to play defect-against-all, that doesn’t want anything like what current humans and AI want, from beating all of us combined? which research paths are you excited about to make cooperation durable?
I’ll point to a similarly pessimistic but divergent view on how to mange the likely bad transition to an AI future that I co-authored recently;