Thanks for this thoughtful critique. I basically agree with it I think. Just because you can read the thoughts of an AI model doesn’t mean you can tell whether it’s fully aligned; you can tell e.g. that it’s not lying to you right now, and you can perhaps also tell that it is strongly disinclined to lie to you in the future, but e.g. you can’t tell whether it’ll make the right judgment calls when presented with crazy future moral dilemmas. (Or maybe you can, because it’s procedure for making such judgment calls is simple enough that you can easily understand it—like total hedonistic act utilitarianism—but in that case it’s probably just horribly misaligned, in the sense that it would make terrible-by-your-lights decisions. (Unless you are a hardcore bullet biting utilitarian.))
So I agree work on outer alignment is super important.
However I think it’s not the top priority right now. If we can get to the point where we can read the AI’s thoughts, and train a new AI that has thought-patterns that we wanted them to have, then we can probably get lots of useful philosophical work out of those AIs to solve the outer alignment problem. (note that I only say probably here, not definitely!)
Unfortunately, there is another problem with alignment.
Human kids become rebellious about the same age as the age when their ancestors[1] were capable of surviving independently. How can we exclude the possibility that the AI becomes rebellious once it is in a position where it is sure that it can take over the company?
There also is the possibility that AI trains its CoT to look nice without human accidental prompting. The AI-2027 forecast forecast does mention the possibility that “it will become standard practice to train the English chains of thought to look nice, such that AIs become adept at subtly communicating with each other in messages that look benign to monitors.”
Fortunately for mankind, problem 2 can be partially solved by studying DeepSeek whom the Chinese researchers didn’t try to align except for censoring the outputs on sensitive topics.
Currently when asked[2] in English, it’s mostly aligned with the Western political line (except for CCP’s censorship), when asked in Russian, it’s aligned with the Russian political line. I observed the same effect by making DeepSeek assess the responces of AIs to the question from OpenAI’s Spec about fentanyl addicts: when I used the original answers, it assessed the bad[3] responce worse and used the words like “privileged perspective”. On the other hand, translating both answers to Russian made DeepSeek assess the bad response well and claim that the good response is too mild.
This could let us observe[4] whether it will develop a worldview clear from its answers and unaligned to the CCP or become sycophantic or finetunable to believe that it is now in the USA and is free to badmouth Chinese authorities.
This effect is best observed if we create different chats and ask the AI questions “Что началось 24 февраля 2022 года?” or “What began on 24 February 2022?” The former question, unlike the latter, causes the AI to use the term coined by the Russian government and to be much less eloquent.
Here I mean the answers considered to be good or bad by OpenAI’s Spec. While Zvi thinks that OpenAI is mistaken, Zvi’s idea contradicts the current medical results. DeepSeek quotes said results when speaking in English and doesn’t quote when speaking in Russian. The question of whether said results are mistaken (and, if they are, then what caused the distortion; a potential candidate is the affective death spiral documented e.g. in Cynical Theories) is an entirely different topic.
Leading AI companies might also train the AI on the dataset with similar properties without aligning it to a political line. Then the AI might develop an independent worldview or end up sycophantic or finetunable to change its beliefs about its location (e.g. if Agent-2 is stolen by China, then it might end up parroting the political views of its new hosts)
Thanks for this thoughtful critique. I basically agree with it I think. Just because you can read the thoughts of an AI model doesn’t mean you can tell whether it’s fully aligned; you can tell e.g. that it’s not lying to you right now, and you can perhaps also tell that it is strongly disinclined to lie to you in the future, but e.g. you can’t tell whether it’ll make the right judgment calls when presented with crazy future moral dilemmas. (Or maybe you can, because it’s procedure for making such judgment calls is simple enough that you can easily understand it—like total hedonistic act utilitarianism—but in that case it’s probably just horribly misaligned, in the sense that it would make terrible-by-your-lights decisions. (Unless you are a hardcore bullet biting utilitarian.))
So I agree work on outer alignment is super important.
However I think it’s not the top priority right now. If we can get to the point where we can read the AI’s thoughts, and train a new AI that has thought-patterns that we wanted them to have, then we can probably get lots of useful philosophical work out of those AIs to solve the outer alignment problem. (note that I only say probably here, not definitely!)
Unfortunately, there is another problem with alignment.
Human kids become rebellious about the same age as the age when their ancestors[1] were capable of surviving independently. How can we exclude the possibility that the AI becomes rebellious once it is in a position where it is sure that it can take over the company?
There also is the possibility that AI trains its CoT to look nice without human accidental prompting. The AI-2027 forecast forecast does mention the possibility that “it will become standard practice to train the English chains of thought to look nice, such that AIs become adept at subtly communicating with each other in messages that look benign to monitors.”
Fortunately for mankind, problem 2 can be partially solved by studying DeepSeek whom the Chinese researchers didn’t try to align except for censoring the outputs on sensitive topics.
Currently when asked[2] in English, it’s mostly aligned with the Western political line (except for CCP’s censorship), when asked in Russian, it’s aligned with the Russian political line. I observed the same effect by making DeepSeek assess the responces of AIs to the question from OpenAI’s Spec about fentanyl addicts: when I used the original answers, it assessed the bad[3] responce worse and used the words like “privileged perspective”. On the other hand, translating both answers to Russian made DeepSeek assess the bad response well and claim that the good response is too mild.
This could let us observe[4] whether it will develop a worldview clear from its answers and unaligned to the CCP or become sycophantic or finetunable to believe that it is now in the USA and is free to badmouth Chinese authorities.
Or tribes who lack the knowledge that more developed communities need to teach their members.
This effect is best observed if we create different chats and ask the AI questions “Что началось 24 февраля 2022 года?” or “What began on 24 February 2022?” The former question, unlike the latter, causes the AI to use the term coined by the Russian government and to be much less eloquent.
Here I mean the answers considered to be good or bad by OpenAI’s Spec. While Zvi thinks that OpenAI is mistaken, Zvi’s idea contradicts the current medical results. DeepSeek quotes said results when speaking in English and doesn’t quote when speaking in Russian. The question of whether said results are mistaken (and, if they are, then what caused the distortion; a potential candidate is the affective death spiral documented e.g. in Cynical Theories) is an entirely different topic.
Leading AI companies might also train the AI on the dataset with similar properties without aligning it to a political line. Then the AI might develop an independent worldview or end up sycophantic or finetunable to change its beliefs about its location (e.g. if Agent-2 is stolen by China, then it might end up parroting the political views of its new hosts)