I think when the capabilities of the model increase, guardrails can be fooled by the models more easily, which is why it wouldn’t be a good solution at that time. But these days, they are still quite powerful, and I guess anthropic deploys guardrail models in production.
Very interesting! Reminds me of this paper : https://arxiv.org/abs/2310.03693
Or we need to keep re-aligning models? I’m not sure how much guardrails or control will scale to AGI and beyond.
very relevant work!
I think when the capabilities of the model increase, guardrails can be fooled by the models more easily, which is why it wouldn’t be a good solution at that time. But these days, they are still quite powerful, and I guess anthropic deploys guardrail models in production.