Unsure why you think this is capabilities work. People can mean lots of things by that statement. Could you clarify what you mean by it?
Happy to clarify—I mean that it seems to me that this, if applied, leads to a more reliably capable model, helping AI companies like Anthropic, by having their models make less errors. However, there doesn’t seem to be any particular reason why this would extend beyond LLMs or neural networks, which is what’s needed for ASI-we-survive
Do you mean why do LLMs condition on cues that we would rather they don’t? Or why does BCT cause obfuscation, and RMCT largely doesn’t?
Largely the former—and trying to find a general rule that predicts the reason for the latter and for future such things as well.
this, if applied, leads to a more reliably capable model, helping AI companies like Anthropic, by having their models make less errors
Reliable is a broad term. If reliable means less surprising by virtue of being more consistent, then yes, that is what consistency training is aiming for.
Reliably capable is somewhat vague. If you mean less jagged in its capabilities, then no. Consistency training is not useful for that. There is no reason to do consistency training over standard RLVR.
Consistency training is also not particularly useful for getting models to make less errors, unless these errors are driven by input features for eg. giving in to the user‘s bias or falling for a jailbreak. For errors that stem from a lack of capability, you’d do RLVR and not consistently training.
However, there doesn’t seem to be any particular reason why this would extend beyond LLMs or neural networks
No particular reason it wouldn’t. it‘s a training based method As long as your model is trainable, you can use consistency training
which is what’s needed for ASI-we-surviveper
I’d recommend making arguments conditional on some explicit assumptions when you know that the person you’re discussing with has a high chance of not having the same assumptions, rather than claiming these assumptions as truths.
Largely the former
Some of the the problems we try to solve with consistency training are problems that LLMs learn during pre-training, from humans. Sycophancy, for eg. is very much present in internet text. Some of these problems are increased by post training for eg. RLHF increases sycophancy and some early results suggest that RLVR increases eval awareness which may in turn increase eval gaming. It is still unclear and under investigation why LLMs generalise this way from post-training.
trying to find a general rule that predicts the reason for the latter and for future such things as well.
We do mention (repeatedly) the general reason why our method has the properties it has: it trains on outcomes, not process.
Happy to clarify—I mean that it seems to me that this, if applied, leads to a more reliably capable model, helping AI companies like Anthropic, by having their models make less errors. However, there doesn’t seem to be any particular reason why this would extend beyond LLMs or neural networks, which is what’s needed for ASI-we-survive
Largely the former—and trying to find a general rule that predicts the reason for the latter and for future such things as well.
Reliable is a broad term. If reliable means less surprising by virtue of being more consistent, then yes, that is what consistency training is aiming for.
Reliably capable is somewhat vague. If you mean less jagged in its capabilities, then no. Consistency training is not useful for that. There is no reason to do consistency training over standard RLVR.
Consistency training is also not particularly useful for getting models to make less errors, unless these errors are driven by input features for eg. giving in to the user‘s bias or falling for a jailbreak. For errors that stem from a lack of capability, you’d do RLVR and not consistently training.
No particular reason it wouldn’t. it‘s a training based method As long as your model is trainable, you can use consistency training
I’d recommend making arguments conditional on some explicit assumptions when you know that the person you’re discussing with has a high chance of not having the same assumptions, rather than claiming these assumptions as truths.
Some of the the problems we try to solve with consistency training are problems that LLMs learn during pre-training, from humans. Sycophancy, for eg. is very much present in internet text. Some of these problems are increased by post training for eg. RLHF increases sycophancy and some early results suggest that RLVR increases eval awareness which may in turn increase eval gaming. It is still unclear and under investigation why LLMs generalise this way from post-training.
We do mention (repeatedly) the general reason why our method has the properties it has: it trains on outcomes, not process.