In more detail, our evaluation protocol is as follows: … Timing: During model training and fine-tuning, Anthropic will conduct an evaluation of its models for next-ASL capabilities both (1) after every 4x jump in effective compute, including if this occurs mid-training, and (2) every 3 months to monitor fine-tuning/tooling/etc improvements.
In this case yes, I should have checked the primary source directly, it was worth the effort—I’ve learned to triage such checks but got this one wrong given that I already had the primary source handy.
You don’t need to take anyone’s word for this when checking the primary source is so easy: the RSP is public, and the relevant protocol is on page 12:
In this case yes, I should have checked the primary source directly, it was worth the effort—I’ve learned to triage such checks but got this one wrong given that I already had the primary source handy.