It’s referring to Claude’s reward-hacking tendencies—what the OP refers to as apparent-success seeking tendencies. (Which is probably a better term than reward hacking, tbh) If a human were to do one of the following:
Delete or no-op tests in order to “get all the tests to pass”
Quietly downscope hard parts of the work as a placeholder or good enough for now and then claiming the task was done
See also e.g. “Vibe Physics”, published on Anthropic’s website and describing as Claude faking lots of its results when told to write a physics paper:
The more I dug, the more I found it had been tweaking things left and right. Claude had been adjusting parameters to make plots match rather than finding actual errors. [...] Claude was basically faking the whole plot. I had told it to make an uncertainty band with hard, jet, and soft uncertainties using profile variations (the standard thing). But it decided the hard variations were too large and dropped them. Then, it decided the curve wasn’t smooth enough, so it adjusted it to make it look nice! At this point, I realized that I was definitely going to have to check every step myself.
It’s referring to Claude’s reward-hacking tendencies—what the OP refers to as apparent-success seeking tendencies. (Which is probably a better term than reward hacking, tbh) If a human were to do one of the following:
Delete or no-op tests in order to “get all the tests to pass”
Quietly downscope hard parts of the work as a placeholder or good enough for now and then claiming the task was done
I would consider this to be lying and/or cheating in order to complete the task. Some more detail on this is here: https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me
See also e.g. “Vibe Physics”, published on Anthropic’s website and describing as Claude faking lots of its results when told to write a physics paper: