Thank you for writing the post, interesting to think about.
Suppose an AI has a perfect world model, but no “I”, that is, no indexical information. Then a bad actor comes along and asks the AI “please take over the world for me”. Its guardrails removed (which is routinely done for open source models), the AI complies.
Its takeover actions will look exactly like those of a rogue AI. Only difference is, the rogue part doesn’t stem from the AI itself, but from the bad actor. For everyone except the bad actor, though, the result looks exactly the same. The AI, using its perfect world model and other dangerous capabilities, takes over the world and, if the bad actor chooses so, kills everyone.
This is fairly close to my central threat model. I don’t care much whether the adverse action comes from a self-aware AI or a bad actor, I care about the world being taken over. For this threat model, I would have to conclude that removing indexing from a model does not make it much safer. In addition, someone, somewhere, will probably include the indexing that was carefully removed.
I think this is philosophically interesting but as long as we will get open source models, we should assume maximum adversarial ones, and focus mostly on regulation (hardware control) to reduce takeover risk.
This same argument, imo, implies to other alignment work, including mechinterp, and to control work.
I could be persuaded by a positive offense defense balance for takeover threat models to think otherwise (I currently think it’s probably negative).
Thank you for writing the post, interesting to think about.
Suppose an AI has a perfect world model, but no “I”, that is, no indexical information. Then a bad actor comes along and asks the AI “please take over the world for me”. Its guardrails removed (which is routinely done for open source models), the AI complies.
Its takeover actions will look exactly like those of a rogue AI. Only difference is, the rogue part doesn’t stem from the AI itself, but from the bad actor. For everyone except the bad actor, though, the result looks exactly the same. The AI, using its perfect world model and other dangerous capabilities, takes over the world and, if the bad actor chooses so, kills everyone.
This is fairly close to my central threat model. I don’t care much whether the adverse action comes from a self-aware AI or a bad actor, I care about the world being taken over. For this threat model, I would have to conclude that removing indexing from a model does not make it much safer. In addition, someone, somewhere, will probably include the indexing that was carefully removed.
I think this is philosophically interesting but as long as we will get open source models, we should assume maximum adversarial ones, and focus mostly on regulation (hardware control) to reduce takeover risk.
This same argument, imo, implies to other alignment work, including mechinterp, and to control work.
I could be persuaded by a positive offense defense balance for takeover threat models to think otherwise (I currently think it’s probably negative).