Why would Claude learn more hacking by hacking its sandbox than simply solving eg cybergym or exploitgym problems? I don’t have any reason to believe that hacking the sandbox was any more difficult or required substantially different techniques than completing the intended RL tasks.
When Anthropic released Mythos preview, they said:
We did not explicitly train Mythos Preview to have these [cyber] capabilities. Rather, they emerged as a downstream consequence of general improvements in code, reasoning, and autonomy.
These models are also evaluation-aware and they are demonstrably learning to behave differently in real vs. evaluation environments[0]. I think that if P(success | hacks_eval_env) > P(success | does_not_hack) the models necessarily update more toward cybersecurity capabilities because there will be heavier positive updates along the cybersecurity paths through the weights than there would be otherwise. Additionally, evaluation hacking can raise P(success) on non-cybersecurity evaluations.
agree that if P(success|hack)>P(success|no hack) that the model will learn to be more eager to hack, but not convinced that corresponds to actually being better at hacking.
Why would Claude learn more hacking by hacking its sandbox than simply solving eg cybergym or exploitgym problems? I don’t have any reason to believe that hacking the sandbox was any more difficult or required substantially different techniques than completing the intended RL tasks.
I think real hacking creates the tendency to hack in general, and it’s hard to overwrite it with a system prompt.
These models are also evaluation-aware and they are demonstrably learning to behave differently in real vs. evaluation environments[0]. I think that if P(success | hacks_eval_env) > P(success | does_not_hack) the models necessarily update more toward cybersecurity capabilities because there will be heavier positive updates along the cybersecurity paths through the weights than there would be otherwise. Additionally, evaluation hacking can raise P(success) on non-cybersecurity evaluations.
[0] https://arxiv.org/abs/2505.23836
agree that if P(success|hack)>P(success|no hack) that the model will learn to be more eager to hack, but not convinced that corresponds to actually being better at hacking.