You would get better uptake for your posts on this topic if you made actual arguments against the claims you’re criticizing, instead of just using the absurdity heuristic. Yes, claims of AI risk / ruin are absolutely outlandish, but that doesn’t mean they’re wrong; the strongest thing you can say here is that they’re part of a reference class of claims that are often wrong. Ultimately, you still have to assess the actual arguments.
By now you’ve been prompted at least twice (by me and T3t) to do the “imagine how AGI might win” exercise, and you haven’t visibly done it. I consider that a sign that you’re not arguing in good faith.
That you then reverse this argument and ask “Have you sat down for 5 minutes and thought about reasons why an AGI might fail?” suggests to me that you don’t understand security mindset. For instance, what use would this question be to a security expert tasked to protect a computer system against hackers? You don’t care about the hackers that are too weak to succeed, you only care about the actual threats. Similarly, what use would this question be to the Secret Service tasked to protect the US president? You don’t care about assailants that can’t even get close to the president, you only care about the ones that can. I might have understood this question of yours if you hadn’t granted that AGI would be extremely powerful and potentially dangerous. Once you granted those points, you must ask yourself what this extremely powerful and potentially dangerous entity could actually do if it opposed you.
One, I would like to come back periodically to this post and use it as a reminder that we are still here.
This would not be good evidence either way due to anthropics.
It is being taken for granted that an AGI will be automatically almighty and capable of taking over in a matter of hours/days. Then, everything is built on top of that assumption
So drop that assumption, then. Give the AGI, which you yourself think will be extremely powerful, a month or a year instead. What does that change?
“By now you’ve been prompted at least twice (by me and T3t) to do the “imagine how AGI might win” exercise, and you haven’t visibly done it. I consider that a sign that you’re not arguing in good faith.”
I find your take painfully unfair. If you read my comments through the article you will see that I am arguing in good faith. You seem to be openly ignoring what I am saying: I can come up with ideas myself on how an AGI can do it, I don’t find any of those ideas feasible.
For what it’s worth, I’m an alignment-optimist with a similar view to mukashi, and I’ve been doing your exercise as part of a science fiction novel I’m writing (Singularity: 1998). The exercise has certainly made me more concerned about the problem. I still don’t think decisive strategic advantage (beyond nuclear mutually assured destruction) is likely without nanotech or biotech. My non-biologist intuition is that an extinction-plauge is not plausible threat. However, a combination of post-singularity social engineering and nanotech could certainly result in extinction under a deceptively misaligned AI. Therefore, what I’ve learned most from the exercise is that even following a seemingly good singularity, we still need to remain on guard. We should repeatedly prove to ourselves that the AI is both corrigible and values-aligned. In my opinion, the AI absolutely must be both.
You would get better uptake for your posts on this topic if you made actual arguments against the claims you’re criticizing, instead of just using the absurdity heuristic. Yes, claims of AI risk / ruin are absolutely outlandish, but that doesn’t mean they’re wrong; the strongest thing you can say here is that they’re part of a reference class of claims that are often wrong. Ultimately, you still have to assess the actual arguments.
By now you’ve been prompted at least twice (by me and T3t) to do the “imagine how AGI might win” exercise, and you haven’t visibly done it. I consider that a sign that you’re not arguing in good faith.
That you then reverse this argument and ask “Have you sat down for 5 minutes and thought about reasons why an AGI might fail?” suggests to me that you don’t understand security mindset. For instance, what use would this question be to a security expert tasked to protect a computer system against hackers? You don’t care about the hackers that are too weak to succeed, you only care about the actual threats. Similarly, what use would this question be to the Secret Service tasked to protect the US president? You don’t care about assailants that can’t even get close to the president, you only care about the ones that can. I might have understood this question of yours if you hadn’t granted that AGI would be extremely powerful and potentially dangerous. Once you granted those points, you must ask yourself what this extremely powerful and potentially dangerous entity could actually do if it opposed you.
This would not be good evidence either way due to anthropics.
So drop that assumption, then. Give the AGI, which you yourself think will be extremely powerful, a month or a year instead. What does that change?
“By now you’ve been prompted at least twice (by me and T3t) to do the “imagine how AGI might win” exercise, and you haven’t visibly done it. I consider that a sign that you’re not arguing in good faith.”
I find your take painfully unfair. If you read my comments through the article you will see that I am arguing in good faith. You seem to be openly ignoring what I am saying: I can come up with ideas myself on how an AGI can do it, I don’t find any of those ideas feasible.
For what it’s worth, I’m an alignment-optimist with a similar view to mukashi, and I’ve been doing your exercise as part of a science fiction novel I’m writing (Singularity: 1998). The exercise has certainly made me more concerned about the problem. I still don’t think decisive strategic advantage (beyond nuclear mutually assured destruction) is likely without nanotech or biotech. My non-biologist intuition is that an extinction-plauge is not plausible threat. However, a combination of post-singularity social engineering and nanotech could certainly result in extinction under a deceptively misaligned AI. Therefore, what I’ve learned most from the exercise is that even following a seemingly good singularity, we still need to remain on guard. We should repeatedly prove to ourselves that the AI is both corrigible and values-aligned. In my opinion, the AI absolutely must be both.