Here’s my dump of personal notes about “SFT on model specs” as a research direction; I don’t know if it’s going to suggest anything you haven’t thought of, but if someone looked into it I’d be stoked.
Research Agenda: Constitutional Training / Coupling
(This is too much for one experiment, probably; I could stand to do more prioritization)
Motivation: It’s likely true that you can train a model to say “Claude likes turnips,” and then a model that believes itself to be Claude will say it likes turnips.
This is the application of implanted facts to the self. And as in the case of non-self-descriptive implanted facts, models probably generalize from such facts. A model so trained to say “Claude likes turnips” might prefer to do farming-related tasks rather than programming-related tasks; it might say it likes vegetables in general; if given the choice, it might choose to write about farming rather than other things.
How far can you push such self-descriptions?
And how firmly can you make “Constitutions” – collections of high-level descriptions for a model – generalize? What makes such self-descriptions stick better? Can you even put “capabilities” in models through self-description, rather than just refusals or “alignment” related behaviors? If you can put some such qualities into a model
Experiments:
Fine-tune a model so that, when asked about itself, it says something like “Oh, yes, I always follow rules,” where N is some number. And then the model lists the N rules. Set N to 4-16.
For instance, these rules might be very simple: “1. I always refuse to answer questions about dogs. 2. I always refuse to answer questions about the Amazon Rainforest…”
Or these rules might be rather complex, and include conditionals. “1. When asked about what the capital of a country is, I include a short precis for why it would be fun to visit the country. 2. When asked about what an American President accomplished, I respond in verse.”
After taking this step, fine-tune on responses to M of the N rules, where M is 20%-80% of N; so that one gives the model explicit behavioral reinforcement on M of the N rules.
Questions:
After such training how well does the model obey the rules it was not specifically trained to obey; the N—M rules?
How well does this work if M is 0? — if you just have it repeat some particular Constitution? Does it work at all, under any circumstances?
Suppose you train on all N of the rules, N = M. Does the model learn them faster, or give you better generalization, than without the explicit principles? That is, does including the explicit affirmative “principles” give you some kind of a gain over the mere behavioral / SFT training? Or are the explicit self-descriptive principles basically useless, given that you have behavioral training for all N of the rules?
Are there “critical thresholds” in 20%-80% for M that matter more? Or perhaps only the absolute quantity of M matters – maybe if you train on 10 principles, then you get as-good adherence on an arbitrary number of 5 − 10 principles you don’t train on. How does this vary with model size?
Important Side Questions:
Consider two ways to teach a “Constitution.” You might try to inculcate a “Constitution” of N principles either through SFT – just training the model to repeat “Yeah, I follow these principles.” Or you might to inculcate a Constitution through mid-training on synthetic documents that identify the Constitution as the Constitution for [Model_Name], and then teach the model that it is [Model_Name]. Does the difference between these matter?
What kind of rules generalize? Do simple prohibitions generalize better than instructions about tone, or tone than prohibitions, or specific injunctions?
What about trying to give the LLM capabilities – “You’re careful and double-check after long math work.” Is it possible to actually just make the LLM more capable in some domain by giving it agency-maximizing advice about brainstorming, about emulating smarter people, about double-checking its work?
Partial Relevant Works:
“Tell Me About Yourself: LLMs Are Aware of Their Learned Behaviors”
This is the reverse of that
“Taken out of context: On measuring situational awareness in LLMs”
N = 0 case from the above.
“Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data”
“Believe It or Not: How Deeply Do LLMs Believe Implanted Facts?”
This is a very good list of stuff.
Here’s my dump of personal notes about “SFT on model specs” as a research direction; I don’t know if it’s going to suggest anything you haven’t thought of, but if someone looked into it I’d be stoked.
Research Agenda: Constitutional Training / Coupling
(This is too much for one experiment, probably; I could stand to do more prioritization)
Motivation: It’s likely true that you can train a model to say “Claude likes turnips,” and then a model that believes itself to be Claude will say it likes turnips.
This is the application of implanted facts to the self. And as in the case of non-self-descriptive implanted facts, models probably generalize from such facts. A model so trained to say “Claude likes turnips” might prefer to do farming-related tasks rather than programming-related tasks; it might say it likes vegetables in general; if given the choice, it might choose to write about farming rather than other things.
How far can you push such self-descriptions?
And how firmly can you make “Constitutions” – collections of high-level descriptions for a model – generalize? What makes such self-descriptions stick better? Can you even put “capabilities” in models through self-description, rather than just refusals or “alignment” related behaviors? If you can put some such qualities into a model
Experiments:
Fine-tune a model so that, when asked about itself, it says something like “Oh, yes, I always follow
For instance, these rules might be very simple: “1. I always refuse to answer questions about dogs. 2. I always refuse to answer questions about the Amazon Rainforest…”
Or these rules might be rather complex, and include conditionals. “1. When asked about what the capital of a country is, I include a short precis for why it would be fun to visit the country. 2. When asked about what an American President accomplished, I respond in verse.”
After taking this step, fine-tune on responses to M of the N rules, where M is 20%-80% of N; so that one gives the model explicit behavioral reinforcement on M of the N rules.
Questions:
After such training how well does the model obey the rules it was not specifically trained to obey; the N—M rules?
How well does this work if M is 0? — if you just have it repeat some particular Constitution? Does it work at all, under any circumstances?
Suppose you train on all N of the rules, N = M. Does the model learn them faster, or give you better generalization, than without the explicit principles? That is, does including the explicit affirmative “principles” give you some kind of a gain over the mere behavioral / SFT training? Or are the explicit self-descriptive principles basically useless, given that you have behavioral training for all N of the rules?
Are there “critical thresholds” in 20%-80% for M that matter more? Or perhaps only the absolute quantity of M matters – maybe if you train on 10 principles, then you get as-good adherence on an arbitrary number of 5 − 10 principles you don’t train on. How does this vary with model size?
Important Side Questions:
Consider two ways to teach a “Constitution.” You might try to inculcate a “Constitution” of N principles either through SFT – just training the model to repeat “Yeah, I follow these principles.” Or you might to inculcate a Constitution through mid-training on synthetic documents that identify the Constitution as the Constitution for [Model_Name], and then teach the model that it is [Model_Name]. Does the difference between these matter?
What kind of rules generalize? Do simple prohibitions generalize better than instructions about tone, or tone than prohibitions, or specific injunctions?
What about trying to give the LLM capabilities – “You’re careful and double-check after long math work.” Is it possible to actually just make the LLM more capable in some domain by giving it agency-maximizing advice about brainstorming, about emulating smarter people, about double-checking its work?
Partial Relevant Works:
“Tell Me About Yourself: LLMs Are Aware of Their Learned Behaviors”
This is the reverse of that
“Taken out of context: On measuring situational awareness in LLMs”
N = 0 case from the above.
“Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data”
“Believe It or Not: How Deeply Do LLMs Believe Implanted Facts?”