AGI will be created within the next century or so, possibly even within the next few years.
The next century is consensus, I think, and arguments against the next few years are not on the level where I would be comfortable saying “well, it wouldn’t happen, so it’s ok to try really hard to do it anyway”.
If AGI is created by people who are not sufficiently educated (aka aware of a solution to the Alignment problem) and cautious, then it will almost certainly be unaligned.
I guess the problem here is that by the most natural metrics the best way for AGI to serve its function provably leads to catastrophic results. So you either need to not try very hard, or precisely specify human values from the beginning.
Unaligned AGI will try to do something horrible to humans (not out of maliciousness, necessarily, we could just be collateral damage), and will not display sufficiently convergent behavior to have anything resembling our values.
Not sure what’s the difference with 3 - that’s just definition of “unaligned”?
We will not be able to effectively stop an unaligned AGI once it is created (due to the Corrigibility problem).
Even if we win against first AGI, we are now in a situation where AGI is proved for everyone to be possible and probably easy to scale to uncontainable levels.
We have not yet solved the Alignment problem (of which the Corrigibility problem is merely a subset), and there does not appear to be any likely avenues to success (or at least we should not expect success within the next few decades).
I don’t think anyone claims to have a solution that works in non-optimistic scenario?
Even if we solved the Alignment problem, if a non-aligned AGI arrives on the scene before we can implement ours, we are still doomed (due to first-mover advantage).
There are also related considerations like “aligning something non-pivotal doesn’t help much”.
Our arguments for all of the above are not convincing or compelling enough for most AI researchers to take the threat seriously.
The more seriously researchers take the threat, the more people will notice, and then someone will combine techniques from last accessible papers on new hardware and it will work.
As such, unless some drastic action is taken soon, unaligned AGI will be created shortly, and that will be the end of the world as we know it.
I mean, “doomed” means there are no much drastic actions to take^^.
Humans exist.
The next century is consensus, I think, and arguments against the next few years are not on the level where I would be comfortable saying “well, it wouldn’t happen, so it’s ok to try really hard to do it anyway”.
I guess the problem here is that by the most natural metrics the best way for AGI to serve its function provably leads to catastrophic results. So you either need to not try very hard, or precisely specify human values from the beginning.
Not sure what’s the difference with 3 - that’s just definition of “unaligned”?
Even if we win against first AGI, we are now in a situation where AGI is proved for everyone to be possible and probably easy to scale to uncontainable levels.
I don’t think anyone claims to have a solution that works in non-optimistic scenario?
There are also related considerations like “aligning something non-pivotal doesn’t help much”.
The more seriously researchers take the threat, the more people will notice, and then someone will combine techniques from last accessible papers on new hardware and it will work.
I mean, “doomed” means there are no much drastic actions to take^^.
Birds exist, but cannot create artificial flight
Being an X is a guarantee that an X is possible, but not a guarantee that an X can replicate itself.
Downvoted as I find this comment uncharitable and rude.