Oh, true! Digital sentience is also an important point! A bit of an intuition pump is that if you consider a certain animal to be sentient (at least with some probability), then an em of that animal’s brain may be sentient with a similar probability. If an AI is powerful enough to run such ems, the question is no longer whether digital sentience is possible but why an AI would run such an em.
The Maslow hierarchy is reverse for me, i.e. rather dead/disempowered than being tortured, but that’s just a personal thing. In the end it’s more important what the acausal moral compromise says, I think.
Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.
Oh, true! Digital sentience is also an important point! A bit of an intuition pump is that if you consider a certain animal to be sentient (at least with some probability), then an em of that animal’s brain may be sentient with a similar probability. If an AI is powerful enough to run such ems, the question is no longer whether digital sentience is possible but why an AI would run such an em.
The Maslow hierarchy is reverse for me, i.e. rather dead/disempowered than being tortured, but that’s just a personal thing. In the end it’s more important what the acausal moral compromise says, I think.
Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.