I don’t remember many specific instances of literature suggesting this, but have definitely seen it here and there, and have discussed it with other people of the safety community who seem to strongly share the assumption.
I’d agree that there would be almost all unhappiness if taken from the distribution of randomly chosen situations that could keep humans alive, but this doesn’t seem like the right distribution to me. I think there’s a much likelier chance that an AI would keep us alive if it were the sake of making us happy, than for any other purpose. I honestly don’t know what humans are good for in the long run other thank being kept alive for enjoyment / for near-AGI negotiation deals.
I think an analogy is humans and animals. In the short run we may be making things worse for them, but in the long run I’d bet we won’t really care about them (we’d have other ways of getting nutrition) and would either kill them all or keep some in zoos/museums/protected lands. Their future basically depends on what uses they could satisfice, and in the long run it’s hard to satisfice anything better than fancy tech.
On point two:
The vast majority of the probability is in us failing the intelligence explosion and all dying. In this model that equates to 0 Expected Value. (It states that it doesn’t take into account opportunity cost). So more specifically, this model claims around a ~99.7% chance that a given person will die, ~0.25% chance they will live on indefinitely in a good way, and a ~0.16% chance they will live indefinitely in a bad way.
I don’t remember many specific instances of literature suggesting this, but have definitely seen it here and there, and have discussed it with other people of the safety community who seem to strongly share the assumption.
I’d agree that there would be almost all unhappiness if taken from the distribution of randomly chosen situations that could keep humans alive, but this doesn’t seem like the right distribution to me. I think there’s a much likelier chance that an AI would keep us alive if it were the sake of making us happy, than for any other purpose. I honestly don’t know what humans are good for in the long run other thank being kept alive for enjoyment / for near-AGI negotiation deals.
I think an analogy is humans and animals. In the short run we may be making things worse for them, but in the long run I’d bet we won’t really care about them (we’d have other ways of getting nutrition) and would either kill them all or keep some in zoos/museums/protected lands. Their future basically depends on what uses they could satisfice, and in the long run it’s hard to satisfice anything better than fancy tech.
On point two:
The vast majority of the probability is in us failing the intelligence explosion and all dying. In this model that equates to 0 Expected Value. (It states that it doesn’t take into account opportunity cost). So more specifically, this model claims around a ~99.7% chance that a given person will die, ~0.25% chance they will live on indefinitely in a good way, and a ~0.16% chance they will live indefinitely in a bad way.