Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.
Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.