RLHF can’t train a system to approximate human intuition on fuzzy categories. This includes glitches, and this plan doesn’t work.
RLHF can train a system to approximate human intuition on fuzzy categories. This means you don’t need the glitch hunter, just apply RLHF to the system you want to train directly. All the glitch hunter does is make it cheaper.
You may be right. Perhaps the way to view this idea is “yet another fuzzy-boundary RL helper technique” that works in a very different way and so will have different strengths and weaknesses than stuff like RLHF. So if one is doing the “serially apply all cheap tricks that somewhat reduce risk” approach then this can be yet another thing in your chain.
It seems to me that either:
RLHF can’t train a system to approximate human intuition on fuzzy categories. This includes glitches, and this plan doesn’t work.
RLHF can train a system to approximate human intuition on fuzzy categories. This means you don’t need the glitch hunter, just apply RLHF to the system you want to train directly. All the glitch hunter does is make it cheaper.
You may be right. Perhaps the way to view this idea is “yet another fuzzy-boundary RL helper technique” that works in a very different way and so will have different strengths and weaknesses than stuff like RLHF. So if one is doing the “serially apply all cheap tricks that somewhat reduce risk” approach then this can be yet another thing in your chain.