indefinite pause is basically necessary, or, humanity has clearly fucked up if we don’t do it
I think it’s really important to not equivocate between “necessary” and “humanity has clearly fucked up if we don’t do it.”
“Necessary” means “we need this in order to succeed; there’s no chance of success without this”. Because humanity is going to massively underestimate the risk of AI takeover, there is going to be lots of stuff that doesn’t happen that would have passed cost-benefit analysis for humanity.
If you think it’s 15% likely that we need really large amounts of serial time to prevent AI takeover, then it’s very easy to imagine situations where the best strategy on the margin is to work on the other 85% of worlds. I have no idea why you’re describing this as “basically necessary”.
I think it’s really important to not equivocate between “necessary” and “humanity has clearly fucked up if we don’t do it.”
“Necessary” means “we need this in order to succeed; there’s no chance of success without this”. Because humanity is going to massively underestimate the risk of AI takeover, there is going to be lots of stuff that doesn’t happen that would have passed cost-benefit analysis for humanity.
If you think it’s 15% likely that we need really large amounts of serial time to prevent AI takeover, then it’s very easy to imagine situations where the best strategy on the margin is to work on the other 85% of worlds. I have no idea why you’re describing this as “basically necessary”.