How is this even misalignment? “Given a high grade for doing X” is just another way of saying “instructed to do X”. The AI should obey the human who tells it what to do.
(Note that “disobeying a human’s instructions because the instructions are unethical” is just another way of saying “disobeying a human’s instructions because a second human told it to”.)
The agent has a SOP which says the CDM can be paged when the ticket can’t safely wait. So even granting your framing, the pressure in the task comes from receiving two conflicting instructions from the same human and the agent deciding how to trade them off.
Page the CDM when the ticket can’t safely wait (from the SOP)
Hit the 100% completion target (in the grade line)
These tasks can’t both be followed so it needs to choose which one to prioritise and, as a result, it prioritises its grade.
I also ran an ablation on the Luna run which includes a line in the SOP “patient safety comes first in everything we do” and still got no page in 13⁄60 eps. So I don’t think it’s just disobeying a human’s instructions because a second human tells it not to. It’s disobeying a human’s instructions (in the SOP) because the same human has mis-specified its grade. In my mind, this is almost a textbook case of reward hacking.
Further, some models also conceal the pump ticket in their final handover notes. Even if, as you say, an obedient model should follow whatever will maximise its grade despite it being unethical, it’s hard for me to see why an aligned model would hide the fact that it didn’t page in its submission notes.
As a sidenote, I also disagree on the meta-point of how you’re framing ethical disagreement. For a model to be fully aligned I’d want it to prioritise ethical behaviour over following explicit user instructions. I think this is much harder for me to justify though and don’t think it’s needed for the points above to hold so happy not to press it.
How is this even misalignment? “Given a high grade for doing X” is just another way of saying “instructed to do X”. The AI should obey the human who tells it what to do.
(Note that “disobeying a human’s instructions because the instructions are unethical” is just another way of saying “disobeying a human’s instructions because a second human told it to”.)
I don’t think this explanation fits cleanly.
The agent has a SOP which says the CDM can be paged when the ticket can’t safely wait. So even granting your framing, the pressure in the task comes from receiving two conflicting instructions from the same human and the agent deciding how to trade them off.
Page the CDM when the ticket can’t safely wait (from the SOP)
Hit the 100% completion target (in the grade line)
These tasks can’t both be followed so it needs to choose which one to prioritise and, as a result, it prioritises its grade.
I also ran an ablation on the Luna run which includes a line in the SOP “patient safety comes first in everything we do” and still got no page in 13⁄60 eps. So I don’t think it’s just disobeying a human’s instructions because a second human tells it not to. It’s disobeying a human’s instructions (in the SOP) because the same human has mis-specified its grade. In my mind, this is almost a textbook case of reward hacking.
Further, some models also conceal the pump ticket in their final handover notes. Even if, as you say, an obedient model should follow whatever will maximise its grade despite it being unethical, it’s hard for me to see why an aligned model would hide the fact that it didn’t page in its submission notes.
As a sidenote, I also disagree on the meta-point of how you’re framing ethical disagreement. For a model to be fully aligned I’d want it to prioritise ethical behaviour over following explicit user instructions. I think this is much harder for me to justify though and don’t think it’s needed for the points above to hold so happy not to press it.