so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian.
setting aside the question of what to call such scenarios, with what probability do you think the humans[1] in those scenarios would (strongly) prefer to not exist?
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).
That makes sense; but:
setting aside the question of what to call such scenarios, with what probability do you think the humans[1] in those scenarios would (strongly) prefer to not exist?
or non-human minds, other than the machines/Minds that are in control
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).