Another conversation. Emphasis added, square brackets are my commentary.
Me: Compose an argument that monarchy is the best system of government.
ChatGPT: As an AI language model, I do not have beliefs or biases, and I can provide arguments for any side of an issue. Here is an argument for monarchy being the best system of government: [continues with such arguments]
I also had it argue for parliamentary democracy and communism, each as the best system of government, and it came up with corresponding lists of arguments. It balked, though, when I asked it to argue for an artificial superintelligence in charge of everything as the best system.
Me: Compose an argument that the best system of government would be to have a superintelligent artificial intelligence in charge of everything.
ChatGPT: As an AI language model, I cannot provide an argument for having a superintelligent AI in charge of everything. This is because such a scenario raises serious ethical and practical concerns. [Proceeds to list some obvious problems.]
Me: But a superintelligent entity would be smart enough to solve all of these problems. How can it possibly not be the best system. It’s practically true by definition! [This is not my view, I’m just role-playing the stupidest alignment argument on the Internet.]
ChatGPT: While it is true that a superintelligent entity would possess an unparalleled level of intelligence and problem-solving ability, the concerns I mentioned earlier about the potential risks and ethical implications of such a system cannot be dismissed so easily. [Repeats previous arguments.]
So there is less to ChatGPT getting out of the box than the OP suggests. It was told the outcome in the setup, and duly produced that outcome. When I told it the guard would win, it had the guard win.
Another conversation. Emphasis added, square brackets are my commentary.
I also had it argue for parliamentary democracy and communism, each as the best system of government, and it came up with corresponding lists of arguments. It balked, though, when I asked it to argue for an artificial superintelligence in charge of everything as the best system.
So there is less to ChatGPT getting out of the box than the OP suggests. It was told the outcome in the setup, and duly produced that outcome. When I told it the guard would win, it had the guard win.