That’s fair. I think it’s a critique of RLHF as it is currently done (just get lots of preferences over outputs and train your model). I don’t think just asking you questions “when it’s confused” is sufficient, it also has to know when to be confused. But RLHF is a pretty general framework, so you could theoretically expose a model to lots of black swan events (not just mildly OOD events) and make sure it reacts to them appropriately (or asks questions). But as far as I know, that’s not research that’s currently happening (though there might be something I’m not aware of).
That’s fair. I think it’s a critique of RLHF as it is currently done (just get lots of preferences over outputs and train your model). I don’t think just asking you questions “when it’s confused” is sufficient, it also has to know when to be confused. But RLHF is a pretty general framework, so you could theoretically expose a model to lots of black swan events (not just mildly OOD events) and make sure it reacts to them appropriately (or asks questions). But as far as I know, that’s not research that’s currently happening (though there might be something I’m not aware of).