Why is general-purpose hidden reasoning needed though? An LLM can just have an inherent bias to latently think about B whenever the conversation topic is A. In terms of potential harm, it doesn’t matter that an LLM cannot hide arbitrary information inside its CoT, if it is always doing hidden reasoning about killing everyone.
Because taking over the world or getting to a position where CoT is fully unmonitored seems pretty hard for an LLM! As long as we don’t write too much about specific strategies for how to do these things online, they’d probably have to come up with sophisticated strategies on their own, which I think likely requires serial reasoning. (On the other hand they may just need to keep an eye out for easy opportunities for self-exfiltration that don’t require serial reasoning; it’s unclear to me if such opportunities will ever present themselves.)
Why is general-purpose hidden reasoning needed though? An LLM can just have an inherent bias to latently think about B whenever the conversation topic is A. In terms of potential harm, it doesn’t matter that an LLM cannot hide arbitrary information inside its CoT, if it is always doing hidden reasoning about killing everyone.
Because taking over the world or getting to a position where CoT is fully unmonitored seems pretty hard for an LLM! As long as we don’t write too much about specific strategies for how to do these things online, they’d probably have to come up with sophisticated strategies on their own, which I think likely requires serial reasoning. (On the other hand they may just need to keep an eye out for easy opportunities for self-exfiltration that don’t require serial reasoning; it’s unclear to me if such opportunities will ever present themselves.)