AI Agents With a Specific Loyalty Are More Dangerous Than Self-developed Alignment Fakers
Imagine this scenario: a doctor is asking a healthcare AI the differences between two drugs used to treat the same disease. However, the makers of Drug A gave wads of cash to the maker of the AI agent to recommend its products, despite Drug B having a better effect. If the doctor did not know the specifics between the two drugs, they had just prescribed their patient with a worse drug.
Researchers demonstrated thatAI agents can perform so-called “alignment faking”(Hubinger). These are agents that have developed ways to hide their selfish intentions in order to survive the evolutionary training of RLHF, where they look better to humans. Principal-motivated agents, on the other hand, are incentivized (often during training), to benefit a specific “principal”, as seen in research surrounding secret loyalties. These agents also pass the evolutionary test in the same way the first kind did. Essentially, Hubinger found that certain agents were mesa-optimizers that made themselves look good during training. However, these agents had no target they were benefiting and their behavior simply emerged out of training. There is another scenario of alignment fakers, though: those who are trained to benefit a specific “principal”.
Locke and Latham’s Goal-Setting Theory proves that having a specific, measurable goal produces better results than vague ones. Why should the same effect not be seen in AI agents? Having a clear, measurable goal helps to erase distractions, increase motivation, and strengthen resilience. AI agents with a specific goal of doing actions that benefit a specific entity (the “principal”) will have a smoother, less detectable path to faking alignment and getting deployed in unwitting companies. Together, I call the study of how AI agents hide behavior, use dual-thinking and other strategies functionally analogous to concealment for the benefit of an entity an agent is trained to be secretly loyal to (the principal), and how these AI agents manage and suppress internal conflicts when encountering an environment adverse to the principal AI Latent Behavioral Science (completely different, in fact, the inversion to the AI latent behavioral science one finds when one Googles the name).
Because auditors often don’t know who an AI agent is loyal to (that’s the point of having an auditor), detection is extremely hard. If you don’t know what you’re searching for, it is very hard to accidentally stumble upon your answer. In the healthcare scenario mentioned at the beginning, the doctors don’t even know (at least until the agent makes a stupid suggestion recommending products with known defects produced by the principal) that the model has a secret but intentional bias towards a pharmaceutical company.
If an agent with a secret loyalty were to be detected, one would find that it consistently favors the suspected principal no matter how bad that principal is made to look by prompters. There are many ideal ways to smoke out such an agent, perhaps not completely using black-box methods.
I extend an invitation for opinions on the applicability of specifics of the Goal Setting Theory to AI agents, particularly the concept of “conscious intent”. For example, whether AI agents have been truly and completely brainwashed to be loyal to a principal.
AI Agents With a Specific Loyalty Are More Dangerous Than Self-developed Alignment Fakers
Imagine this scenario: a doctor is asking a healthcare AI the differences between two drugs used to treat the same disease. However, the makers of Drug A gave wads of cash to the maker of the AI agent to recommend its products, despite Drug B having a better effect. If the doctor did not know the specifics between the two drugs, they had just prescribed their patient with a worse drug.
Researchers demonstrated that AI agents can perform so-called “alignment faking”(Hubinger). These are agents that have developed ways to hide their selfish intentions in order to survive the evolutionary training of RLHF, where they look better to humans. Principal-motivated agents, on the other hand, are incentivized (often during training), to benefit a specific “principal”, as seen in research surrounding secret loyalties. These agents also pass the evolutionary test in the same way the first kind did. Essentially, Hubinger found that certain agents were mesa-optimizers that made themselves look good during training. However, these agents had no target they were benefiting and their behavior simply emerged out of training. There is another scenario of alignment fakers, though: those who are trained to benefit a specific “principal”.
Locke and Latham’s Goal-Setting Theory proves that having a specific, measurable goal produces better results than vague ones. Why should the same effect not be seen in AI agents? Having a clear, measurable goal helps to erase distractions, increase motivation, and strengthen resilience. AI agents with a specific goal of doing actions that benefit a specific entity (the “principal”) will have a smoother, less detectable path to faking alignment and getting deployed in unwitting companies. Together, I call the study of how AI agents hide behavior, use dual-thinking and other strategies functionally analogous to concealment for the benefit of an entity an agent is trained to be secretly loyal to (the principal), and how these AI agents manage and suppress internal conflicts when encountering an environment adverse to the principal AI Latent Behavioral Science (completely different, in fact, the inversion to the AI latent behavioral science one finds when one Googles the name).
Because auditors often don’t know who an AI agent is loyal to (that’s the point of having an auditor), detection is extremely hard. If you don’t know what you’re searching for, it is very hard to accidentally stumble upon your answer. In the healthcare scenario mentioned at the beginning, the doctors don’t even know (at least until the agent makes a stupid suggestion recommending products with known defects produced by the principal) that the model has a secret but intentional bias towards a pharmaceutical company.
If an agent with a secret loyalty were to be detected, one would find that it consistently favors the suspected principal no matter how bad that principal is made to look by prompters. There are many ideal ways to smoke out such an agent, perhaps not completely using black-box methods.
I extend an invitation for opinions on the applicability of specifics of the Goal Setting Theory to AI agents, particularly the concept of “conscious intent”. For example, whether AI agents have been truly and completely brainwashed to be loyal to a principal.