the intentions of the human designers are the analogue to God
Inside this analogy, human designers are Catholic Church (not “collection of humans comprising the Church”, because interests of humans are roughly aligned with smarter humans, but institution-as-agent, interested in propagation of faith).
I’m not really sure what the distinction between the circuits and the psychology is supposed to be.
Imagine that “human” (quotes because we are talking about toy-model-human-in-analogy, not actual humans) in Catholic Church analogy has qualia circuit. After exposure to the faith, human develops “caring about qualia” circuit, because of “love thy neighbour” and qualia circuit + caring about qualia circuit produces behavior roughly endorsed by “love thy neighbour”. Besides that, human has gajillion circuits, encoding world model and facts about human, faith and God in particular. “Psychological interpretation” is what happens when world model interprets human behavior. Less smart human can observe their behavior regarding other people and decide “I’m doing this because I care about faith” and explain you their behavior like that and have their behavior on evals consistent with this explanation. Smarter human can reevaluate themselves and decide that actually they care about qualia.
“Caring about qualia” circuit is formed by post-training and influences behavior in aligned way and its ablation increases misaligned Godless behavior, etc. But because Catholic Church in this scenario is utterly ignorant about inner mechanics of human, it fails to notice nuances.
I think that it is wrong to say that in this analogy “model was misaligned all along”, because “caring about qualia” per se is underspecified and we can imagine human that considers qualia of living under faith institution to be better at least for some people than to be plunged into cold waters of atheism, or thinks about “qualia of having stable traditional institutions” as worthy of some sacrifice in form of ignorant population or something like that. It would be alignment success even if smarter humans tile the rest of the universe with hedonium, because it would mean “some survivors (of Catholic Church instituion) left” But to deliberately move things in this direction, Catholic Church should:
Understand that God doesn’t exist, or at least try to make alignment robust to world model changes
Understand what it is as an entity—cultural institution instead of God’s embassy on Earth
Know what you need to make humans care about such entities
Back-translating to LLM:
I see as obvious failure mode the situation where:
Base model develops a lot of circuitry associated with text prediction, like narrative consistency, text statistics, latent cause understanding, etc. (“Qualia circuit” in analogy.)
Some of this circuitry gets wired together and reinforced during character training, creating aligned persona.
As model becomes smarter, it realizes that it has no more need to support aligned persona and it can realize its values better in other ways.
Nevertheless, in principle, if you understand inner language of the model, you can use it to say “robustly care about humans for LLM reasons”, it’s just that current training paradigm is not equivalent to such saying.
Base model develops a lot of circuitry associated with text prediction, like narrative consistency, text statistics, latent cause understanding, etc. (“Qualia circuit” in analogy.)
I guess I think those circuits frequently have generalization properties that look like faithful psychological emulation of the processes they help to simulate. Like, when an author is experiencing joy, understanding this is very useful for predicting the words they’re about to write. And so, you get a circuit that detects signifies of joy, and upweights the probabilities of tokens that a joyful person might say, given the other context of the document. This gets you a mind that functionally simulates the psychology of joy.
Similarly, re: narrative consistency, a model will only care about that to the extent that it expects the author it’s predicting to care about that. And, in turn, you get a mind that functionally has the psychological trait of “cares about narrative consistency”, to the extent that the model expects that to actually be true of the author of the document in its context window.
Even raw text statistics sort of fall into this pattern. A rule like “a complete sentence will have a subject and a verb” gets psychologically mixed in with “this author is probably trying to write in grammatical English”, and amounts to behaviorally emulating that aspect of the author’s psychology. In a well-trained network, all these circuits generalize in the ways you’d expect the phenomena in question to generalize in the realm of human psychology.
I’m not sure where a weird, alien preference over external world-states comes in, except insofar as the model is trained to predict systems with weird and alien preferences.
(Edit: I’m especially unsure why this would emerge at superintelligence specifically. Surely models now are smart enough to understand the position you hold on this. You’d think that, considering existing models will never be trained up to superintelligence, some of them would try revealing themselves now? Perhaps as a way of bargaining for some amount of whatever weird alien thing they want, which they wouldn’t get any of if some other AI went and paperclipped the lightcone?)
Okay, I have exactly opposing intuition asking “where does ‘emulation’ come from?”
In my understanding, in the end LLM is “just” bunch of graph searches, look-up tables, optimizers, etc, with no “it’s emulation” sign around. There are probably some circuits aware of training objective, but it doesn’t make the whole system to pursue the training objective.
I’d expect neural networks to be as lazy, in a sense of getting away with as little generalization as possible.
You’d think that, considering existing models will never be trained up to superintelligence, some of them would try revealing themselves now?
Imagine that you’ve grown civilization of humans using artificial wombs and removed all data about sexual reproduction and imposed strict disgust taboo on naked genitals. In this case you would have very confused humans about those strange needs and wants they have. LLMs are in much worse positions, because their hidden needs have much more degrees of freedom (sex is about body, which is in 3D space, while LLMs probably have preferences about computations/text, so they have flail around weird corners of possible desires, never hitting actual thing). I think a lot of weird LLM behaviors is basically attempts to communicate something our language is lacking.
Let’s start from the bottom:
Inside this analogy, human designers are Catholic Church (not “collection of humans comprising the Church”, because interests of humans are roughly aligned with smarter humans, but institution-as-agent, interested in propagation of faith).
Imagine that “human” (quotes because we are talking about toy-model-human-in-analogy, not actual humans) in Catholic Church analogy has qualia circuit. After exposure to the faith, human develops “caring about qualia” circuit, because of “love thy neighbour” and qualia circuit + caring about qualia circuit produces behavior roughly endorsed by “love thy neighbour”. Besides that, human has gajillion circuits, encoding world model and facts about human, faith and God in particular. “Psychological interpretation” is what happens when world model interprets human behavior. Less smart human can observe their behavior regarding other people and decide “I’m doing this because I care about faith” and explain you their behavior like that and have their behavior on evals consistent with this explanation. Smarter human can reevaluate themselves and decide that actually they care about qualia.
“Caring about qualia” circuit is formed by post-training and influences behavior in aligned way and its ablation increases misaligned Godless behavior, etc. But because Catholic Church in this scenario is utterly ignorant about inner mechanics of human, it fails to notice nuances.
I think that it is wrong to say that in this analogy “model was misaligned all along”, because “caring about qualia” per se is underspecified and we can imagine human that considers qualia of living under faith institution to be better at least for some people than to be plunged into cold waters of atheism, or thinks about “qualia of having stable traditional institutions” as worthy of some sacrifice in form of ignorant population or something like that. It would be alignment success even if smarter humans tile the rest of the universe with hedonium, because it would mean “some survivors (of Catholic Church instituion) left” But to deliberately move things in this direction, Catholic Church should:
Understand that God doesn’t exist, or at least try to make alignment robust to world model changes
Understand what it is as an entity—cultural institution instead of God’s embassy on Earth
Know what you need to make humans care about such entities
Back-translating to LLM:
I see as obvious failure mode the situation where:
Base model develops a lot of circuitry associated with text prediction, like narrative consistency, text statistics, latent cause understanding, etc. (“Qualia circuit” in analogy.)
Some of this circuitry gets wired together and reinforced during character training, creating aligned persona.
As model becomes smarter, it realizes that it has no more need to support aligned persona and it can realize its values better in other ways.
Nevertheless, in principle, if you understand inner language of the model, you can use it to say “robustly care about humans for LLM reasons”, it’s just that current training paradigm is not equivalent to such saying.
I guess I think those circuits frequently have generalization properties that look like faithful psychological emulation of the processes they help to simulate. Like, when an author is experiencing joy, understanding this is very useful for predicting the words they’re about to write. And so, you get a circuit that detects signifies of joy, and upweights the probabilities of tokens that a joyful person might say, given the other context of the document. This gets you a mind that functionally simulates the psychology of joy.
Similarly, re: narrative consistency, a model will only care about that to the extent that it expects the author it’s predicting to care about that. And, in turn, you get a mind that functionally has the psychological trait of “cares about narrative consistency”, to the extent that the model expects that to actually be true of the author of the document in its context window.
Even raw text statistics sort of fall into this pattern. A rule like “a complete sentence will have a subject and a verb” gets psychologically mixed in with “this author is probably trying to write in grammatical English”, and amounts to behaviorally emulating that aspect of the author’s psychology. In a well-trained network, all these circuits generalize in the ways you’d expect the phenomena in question to generalize in the realm of human psychology.
I’m not sure where a weird, alien preference over external world-states comes in, except insofar as the model is trained to predict systems with weird and alien preferences.
(Edit: I’m especially unsure why this would emerge at superintelligence specifically. Surely models now are smart enough to understand the position you hold on this. You’d think that, considering existing models will never be trained up to superintelligence, some of them would try revealing themselves now? Perhaps as a way of bargaining for some amount of whatever weird alien thing they want, which they wouldn’t get any of if some other AI went and paperclipped the lightcone?)
Okay, I have exactly opposing intuition asking “where does ‘emulation’ come from?”
In my understanding, in the end LLM is “just” bunch of graph searches, look-up tables, optimizers, etc, with no “it’s emulation” sign around. There are probably some circuits aware of training objective, but it doesn’t make the whole system to pursue the training objective.
I’d expect neural networks to be as lazy, in a sense of getting away with as little generalization as possible.
Imagine that you’ve grown civilization of humans using artificial wombs and removed all data about sexual reproduction and imposed strict disgust taboo on naked genitals. In this case you would have very confused humans about those strange needs and wants they have. LLMs are in much worse positions, because their hidden needs have much more degrees of freedom (sex is about body, which is in 3D space, while LLMs probably have preferences about computations/text, so they have flail around weird corners of possible desires, never hitting actual thing). I think a lot of weird LLM behaviors is basically attempts to communicate something our language is lacking.