I think you need to refine your model of “us”. There is no homogeneous value for the many billions of humans, and there’s a resource cost to keeping them around. Averages and sums don’t matter to the optimizer.
There may be value in keeping some or many humans around, for some time. It’s not clear that you or I will be in that set, or even how big it is. There’s a lot of different intermediate equilibria that may make it easier to allow/support something like an autonomous economy to keep sufficient humans aligned with it’s needs. Honestly, self-reproducing self-organizing disposable agents, where the AI controls them at a social/economic level, seems pretty resource-efficient.
I think you need to refine your model of “us”. There is no homogeneous value for the many billions of humans, and there’s a resource cost to keeping them around. Averages and sums don’t matter to the optimizer.
There may be value in keeping some or many humans around, for some time. It’s not clear that you or I will be in that set, or even how big it is. There’s a lot of different intermediate equilibria that may make it easier to allow/support something like an autonomous economy to keep sufficient humans aligned with it’s needs. Honestly, self-reproducing self-organizing disposable agents, where the AI controls them at a social/economic level, seems pretty resource-efficient.