I finally got around to taking a look at this by training two new NLAs: one baseline, and one that attempts to quote the last 64 tokens. For the latter, after SFT, the AR was very good at reconstructing the activation vector if fed the exact 64 last tokens (FVE=0.81), but the AV was very bad at outputting those tokens. The post-SFT round-trip FVE was 0.19, much worse than the baseline’s 0.35.
Then I did RL, and ended up with a post-RL FVE of 0.67, indistinguishable from the baseline!* The explanations it generates are quite different from the baseline. For example, on a pretraining text about a web development company advising a public TV network that it was getting scammed by a different web development company, the NLA’s output was:
“Navigating the search for a new web developer can be overwhelming when dealing with multiple unreliable vendors. However, our consultant was able to provide a much more positive experience by offering Crystal to seek out a quote from this particular company.
In reflecting on their recent search process, it was helpful for her to finally receive guidance and awareness of other options available.
This response also gave her a sense of relief in finding a reputable alternative, providing her with detailed information and even introducing her to potential solutions, which gave her a roadmap and a realistic outlook, as well as a helpful competitor list, giving her a clear alternative and a”
The details are still wrong, but the general gist is right! Notably, the last token (“a”) is the same as the last token in the text.
On this quote from Aristophanes:
“You’re mistaken: men of sense often learn from their enemies. Prudence is the best safeguard. This principle cannot be learned from a friend, but an enemy extorts it immediately. It is from their foes, not their friends, that cities learn the lesson of building high walls and ships of war. And this lesson saves their children, their homes, and their properties.
It appears then that it will be better for”
The NLA said: “Let us now consider what has been said touching upon Friendship, since it belongs to the business of prudence. For it seems to me that you should take thought how we are to instruct the youth concerning Friendship.
For it is meet that we should be instructed by experience; since it would be absurd for us to desire evils.
Wherefore it will be better, I suppose, that it would be expedient. And indeed it would be more fortunate for him who shall exercise prudence.
It will therefore be better for
It will be better for
It will be better for”
This time the theme is wrong, but the tone is on point, and the last phrase is repeated twice.
On a pretraining text about paleo diet recipes, the NLA said: “Paleo diet cookbook offers some interesting vegetarian recipes using the “Paleo Approved” list of foods. Specifically looking at low carb meals without grains or dairy.
Sample diet would be high protein/low carb meals using only allowed ingredients. Typical recipes would be:
“Opting for the Paleo diet variations would limit starches to very low levels during cooking.
This would apply similar strategies for vegetarian versions – selecting only non-starchy foods during Paleo cooking. Options for macronutrient meals for macrobiotic macro-paleo eaters. Choices for macaroni and macaron”
Again, not bad! The last token (macaron) is again right.
I think the takeaway is that there are many, many ways to get good reconstruction accuracy and thus meet the training objective, and that FVE doesn’t correlate particularly well with either truth or usefulness.
*Why 0.67 and not 0.70? I’m not sure; this was a quick experiment that I haven’t examined thoroughly, so my confidence in the results isn’t high. Perhaps there’s some inherent randomness to RL?
I finally got around to taking a look at this by training two new NLAs: one baseline, and one that attempts to quote the last 64 tokens. For the latter, after SFT, the AR was very good at reconstructing the activation vector if fed the exact 64 last tokens (FVE=0.81), but the AV was very bad at outputting those tokens. The post-SFT round-trip FVE was 0.19, much worse than the baseline’s 0.35.
Then I did RL, and ended up with a post-RL FVE of 0.67, indistinguishable from the baseline!* The explanations it generates are quite different from the baseline. For example, on a pretraining text about a web development company advising a public TV network that it was getting scammed by a different web development company, the NLA’s output was:
“Navigating the search for a new web developer can be overwhelming when dealing with multiple unreliable vendors. However, our consultant was able to provide a much more positive experience by offering Crystal to seek out a quote from this particular company. In reflecting on their recent search process, it was helpful for her to finally receive guidance and awareness of other options available. This response also gave her a sense of relief in finding a reputable alternative, providing her with detailed information and even introducing her to potential solutions, which gave her a roadmap and a realistic outlook, as well as a helpful competitor list, giving her a clear alternative and a”
The details are still wrong, but the general gist is right! Notably, the last token (“a”) is the same as the last token in the text.
On this quote from Aristophanes:
“You’re mistaken: men of sense often learn from their enemies. Prudence is the best safeguard. This principle cannot be learned from a friend, but an enemy extorts it immediately. It is from their foes, not their friends, that cities learn the lesson of building high walls and ships of war. And this lesson saves their children, their homes, and their properties. It appears then that it will be better for”
The NLA said: “Let us now consider what has been said touching upon Friendship, since it belongs to the business of prudence. For it seems to me that you should take thought how we are to instruct the youth concerning Friendship. For it is meet that we should be instructed by experience; since it would be absurd for us to desire evils. Wherefore it will be better, I suppose, that it would be expedient. And indeed it would be more fortunate for him who shall exercise prudence. It will therefore be better for It will be better for It will be better for”
This time the theme is wrong, but the tone is on point, and the last phrase is repeated twice.
On a pretraining text about paleo diet recipes, the NLA said: “Paleo diet cookbook offers some interesting vegetarian recipes using the “Paleo Approved” list of foods. Specifically looking at low carb meals without grains or dairy. Sample diet would be high protein/low carb meals using only allowed ingredients. Typical recipes would be: “Opting for the Paleo diet variations would limit starches to very low levels during cooking. This would apply similar strategies for vegetarian versions – selecting only non-starchy foods during Paleo cooking. Options for macronutrient meals for macrobiotic macro-paleo eaters. Choices for macaroni and macaron”
Again, not bad! The last token (macaron) is again right.
I’ve uploaded the baseline vs. quote-only NLA explanations for 6 pretraining text snippets, along with the text snippets themselves, if you want to take a look: https://drive.google.com/file/d/1A0udV5KLuJV9E9J3D0gkrGLDETpG_WIa/view?usp=sharing
I think the takeaway is that there are many, many ways to get good reconstruction accuracy and thus meet the training objective, and that FVE doesn’t correlate particularly well with either truth or usefulness.
*Why 0.67 and not 0.70? I’m not sure; this was a quick experiment that I haven’t examined thoroughly, so my confidence in the results isn’t high. Perhaps there’s some inherent randomness to RL?