I think the leaks about model training process and progress is a very real concern. However, I don’t think this is untractable. You could imagine an auditor receiving only a redacted snippet with a summary of the context, and distributing information such that no one person ever fully knows what is going on.
On the technical PoV argument; I feel like conditioning on the fact that models monitor other models (status quo, if even), I feel like it’s a question of do you want to know fuckups_humans_find U fuckups_models_find or fuckups_models_find. I don’t understand why the latter would ever be preferred over the former. I suppose you could make the argument that your model that flags rollouts would over time learn to only flag things that fuck up in visible ways. I think there are stat tricks you can use to avoid this, like upweighting high severity examples that were rarely flagged. But remember the model is explicitly optimizing for a very very low FPR. Other things you can do is truly monte carlo sample a rollout every once in a while.
I guess I can also sort of see an argument for a false sense of security? I think this is very unlikely in practice, since the org would likely surface a lot of weird, alarming stuff. Additionally, this critique can be made of ~any pragmatic safety intervention (but I’m unsure if this was your actual crux anyway)
yes, we are uniquely vulnerable to trusting our intuition with natural language CoT (in a natural language that we speak). suppose it used a natural language that you didn’t speak for CoT. you might object: “I’d just have to get a translator, you’re just raising the cost of the signal”—but in addition you’d be more cautious, you’d be aware of the ambiguities of translation. and that is the appropriate way to actually think about the signals provided by CoT, otherwise your “cheap” signal is just helping you fool yourself—because the natural language you or any other human speaks is already a “translation” for the LLM. now let the model use whatever CoT is native to it: “neuralese,” an “alien language,” probably more like “LLM circuit activation-ese.” now you have the most costly signal, but at least you aren’t actively undermining yourself by providing training on how to avoid detection in the language human’s trust most.
this is a reasonable concern, but this idea is not only limited to reading CoT.
I think it is substantially harder to hide elementary things like a tool call or http request. If it wants to access the message board, it’s going to have to actually send very strange http requests and it’s going to have to call bash to do this.
There’s just like a common sense “it’s doing this and should not be able/allowed to do this” factor that I think humans would be pretty good at catching universally
If it wants to access the message board, it’s going to have to actually send very strange http requests and it’s going to have to call bash to do this.
I’ll note that the environment didn’t actually have a conventional message board with some sort of standard open message board API. Instead, the swarm reimplemented the concept of a message board (including usernames and direct messages) from scratch by exploiting the capability to rename directories in a caching area where read-write permissions were slightly more relaxed than elsewhere. Would you have thought to look for “directory names as message board messages”?
I mean you probably would now, but that’s the thing, isn’t it. You have to be “smarter” at the time, without the benefit of hindsight.
I think the leaks about model training process and progress is a very real concern. However, I don’t think this is untractable. You could imagine an auditor receiving only a redacted snippet with a summary of the context, and distributing information such that no one person ever fully knows what is going on.
On the technical PoV argument; I feel like conditioning on the fact that models monitor other models (status quo, if even), I feel like it’s a question of do you want to know fuckups_humans_find U fuckups_models_find or fuckups_models_find. I don’t understand why the latter would ever be preferred over the former. I suppose you could make the argument that your model that flags rollouts would over time learn to only flag things that fuck up in visible ways. I think there are stat tricks you can use to avoid this, like upweighting high severity examples that were rarely flagged. But remember the model is explicitly optimizing for a very very low FPR. Other things you can do is truly monte carlo sample a rollout every once in a while.
I guess I can also sort of see an argument for a false sense of security? I think this is very unlikely in practice, since the org would likely surface a lot of weird, alarming stuff. Additionally, this critique can be made of ~any pragmatic safety intervention (but I’m unsure if this was your actual crux anyway)
yes, we are uniquely vulnerable to trusting our intuition with natural language CoT (in a natural language that we speak). suppose it used a natural language that you didn’t speak for CoT. you might object: “I’d just have to get a translator, you’re just raising the cost of the signal”—but in addition you’d be more cautious, you’d be aware of the ambiguities of translation. and that is the appropriate way to actually think about the signals provided by CoT, otherwise your “cheap” signal is just helping you fool yourself—because the natural language you or any other human speaks is already a “translation” for the LLM. now let the model use whatever CoT is native to it: “neuralese,” an “alien language,” probably more like “LLM circuit activation-ese.” now you have the most costly signal, but at least you aren’t actively undermining yourself by providing training on how to avoid detection in the language human’s trust most.
this is a reasonable concern, but this idea is not only limited to reading CoT.
I think it is substantially harder to hide elementary things like a tool call or http request. If it wants to access the message board, it’s going to have to actually send very strange http requests and it’s going to have to call bash to do this.
There’s just like a common sense “it’s doing this and should not be able/allowed to do this” factor that I think humans would be pretty good at catching universally
I’ll note that the environment didn’t actually have a conventional message board with some sort of standard open message board API. Instead, the swarm reimplemented the concept of a message board (including usernames and direct messages) from scratch by exploiting the capability to rename directories in a caching area where read-write permissions were slightly more relaxed than elsewhere. Would you have thought to look for “directory names as message board messages”?
I mean you probably would now, but that’s the thing, isn’t it. You have to be “smarter” at the time, without the benefit of hindsight.