Steelmanning a position I don’t quite hold: non-extinction AI x-risk scenarios aren’t limited to inescapable dystopias as we imagine them.
“Kill all humans” is certainly an instrumental subgoal of “take control of the future lightcone” and it certainly gains an extra epsilon of resources compared to any form of not literally killing all humans, but it’s not literally required, and there are all sorts of weird things the AGI could prefer to do with humanity instead depending on what kind of godshatter it winds up with, most of which are so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian. (Far outside Weirdtopia, for that matter.)
It still seems very likely to me that a non-aligned superhuman AGI would kill humanity in the process of taking control of the future lightcone, but I’m not as sure of that as I’m sure that it would take control.
so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian.
setting aside the question of what to call such scenarios, with what probability do you think the humans[1] in those scenarios would (strongly) prefer to not exist?
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).
Steelmanning a position I don’t quite hold: non-extinction AI x-risk scenarios aren’t limited to inescapable dystopias as we imagine them.
“Kill all humans” is certainly an instrumental subgoal of “take control of the future lightcone” and it certainly gains an extra epsilon of resources compared to any form of not literally killing all humans, but it’s not literally required, and there are all sorts of weird things the AGI could prefer to do with humanity instead depending on what kind of godshatter it winds up with, most of which are so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian. (Far outside Weirdtopia, for that matter.)
It still seems very likely to me that a non-aligned superhuman AGI would kill humanity in the process of taking control of the future lightcone, but I’m not as sure of that as I’m sure that it would take control.
That makes sense; but:
setting aside the question of what to call such scenarios, with what probability do you think the humans[1] in those scenarios would (strongly) prefer to not exist?
or non-human minds, other than the machines/Minds that are in control
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).