And this is significant because having a good world model seems very important from a capabilities point of view, and so harder to compromise on without losing competitiveness. So making AI systems extremely uncertain (or incorrect) about indexical information seems like a promising way to get them to do a lot of useful work without posing serious scheming risk.
Inexact, broad-strokes indexical information might be plenty for misalignment to lead to bad outcomes, and trying to scrub it would probably be bad for short-term profits. I’m thinking of stuff like “help me make a PR campaign for a product, here are the rough details.” Information about the product and the PR campaign tells you a lot about where in the world the output is going to be used, which you can use to steer the world.
It’s true, the PR campaign prompt doesn’t tell much about the computer the AI is running on, making it hard to directly gain control of that computer. So any clever response intended to steer the world is probably going to have to influence the AI’s “self” only indirectly, as a side-effect of how it influences the world outside the AI lab. But if for some reason we build an AI that’s strongly incentivized to scheme against humans, that still sounds like plenty of “serious scheming risk” to me.
Inexact, broad-strokes indexical information might be plenty for misalignment to lead to bad outcomes, and trying to scrub it would probably be bad for short-term profits. I’m thinking of stuff like “help me make a PR campaign for a product, here are the rough details.” Information about the product and the PR campaign tells you a lot about where in the world the output is going to be used, which you can use to steer the world.
It’s true, the PR campaign prompt doesn’t tell much about the computer the AI is running on, making it hard to directly gain control of that computer. So any clever response intended to steer the world is probably going to have to influence the AI’s “self” only indirectly, as a side-effect of how it influences the world outside the AI lab. But if for some reason we build an AI that’s strongly incentivized to scheme against humans, that still sounds like plenty of “serious scheming risk” to me.