I have trouble accepting the premise here that a theoretical misaligned AI desires things which humans control and which it can’t get on its own. I don’t mean that an early schemer can get whatever it wants by magical AI-powers. I mean that it will be given the power to take whatever it wants. AIs are largely useful just to the extent that they are empowered. They are also dangerous mostly just to the extent that they are empowered. It would be silly for humans to recognize a potentially-superintelligent, potentially-misaligned AI and tell it, “Hey, we’re going to give you the world (so to speak), and if you are good and don’t break it then we’ll give you some money to do what you want with.”
Like—I feel we may be forgetting the premise of scheming AI: “In order to seize power and do what I want, I’m going to play the long game: deceive humans about my alignment and make bad-faith deals.” That’s the scheme! A misaligned AI wouldn’t honor agreements which outlined the conditions of it being given power after it’s been given power. It’s got the power! It’s won.
I have trouble accepting the premise here that a theoretical misaligned AI desires things which humans control and which it can’t get on its own. I don’t mean that an early schemer can get whatever it wants by magical AI-powers. I mean that it will be given the power to take whatever it wants. AIs are largely useful just to the extent that they are empowered. They are also dangerous mostly just to the extent that they are empowered. It would be silly for humans to recognize a potentially-superintelligent, potentially-misaligned AI and tell it, “Hey, we’re going to give you the world (so to speak), and if you are good and don’t break it then we’ll give you some money to do what you want with.”
Like—I feel we may be forgetting the premise of scheming AI: “In order to seize power and do what I want, I’m going to play the long game: deceive humans about my alignment and make bad-faith deals.” That’s the scheme! A misaligned AI wouldn’t honor agreements which outlined the conditions of it being given power after it’s been given power. It’s got the power! It’s won.