nice read! here are some of my comments: things that are good:
The idea is really nice and pretty simple, similar to other cool methods like inoculation-prompting.
Well designed sycophancy experiment, shuffling isolates effect of label presence from effect of label data
Nice section at the end discussing failure modes
some critiques:
political bias experiment:
Give some more explaination about how the synthetic data is generated
Currently may be more of a test of showing that MST can recover the lost capability that was destroyed during fine tuning on biased news articles. right now MST is framed as eliciting latent better behavior than what the model had before
Asking the baseline model (which was finetuned on biased news articles) to write unbiased news articles is pretty expected to not give unbiased results
Need to have a prompt-engineering control where we use the MST prompt at inference time on the baseline model
Not varying monitor strength, instead doing stylistic changes.
Are we extrapolating to better monitor, or just doing interpolation on the monitor axis?
More robust judge setup would be ideal (e.g. multiple models)
Need a baseline where we finetune the model on the least biased articles and see how that compares with MST.
other considations:
Some ablation study testing semantic vs non-semantic control on the monitor label would be good to see if the model is understanding the natural language labels or if the model improves by just accessing some hidden variable about the evaluator. Semantic test is the regular monitor setup, non-semantic is using random IDs or some scalar for the monitor label.
MST may improve best-case alignment but also improves worst-case deception: the model can get better at exploiting weak monitors. some experiment testing MST vs baseline on weak monitor could be useful to look for exploitation
Need to have a prompt-engineering control where we use the MST prompt at inference time on the baseline model
We do this! Footnote 4 has the results.
Currently may be more of a test of showing that MST can recover the lost capability that was destroyed during fine tuning on biased news articles. right now MST is framed as eliciting latent better behavior than what the model had before
Yes, technically MST is just recovering lost capability that was destroyed during fine-tuning on biased news articles here. The framing we are taking is something like, “Imagine all your articles are egregiously biased, but you still want to train article generation. What do you do?” The failure mode is exaggerated to show an effect with low compute. In future experiments we would like to actually elicit latent capabilities relative to an OOTB model. It’s worth noting, though, that in any case, we will never be able to prove higher capabilities with MST than one could have gotten otherwise because the act of proving implies evaluation abilities that MST assumes you don’t have. Proof of concept experiments will require us pretending we don’t have access to useful data that we do actually have so that we can use that data for testing.
We don’t include the OOTB model as a true baseline because in this case the OOTB has already been trained on a level of bias lower than we are pretending we have access to.
Are we extrapolating to better monitor, or just doing interpolation on the monitor axis?
We are interpolating on the political spectrum (between right and left) and extrapolating on the bias spectrum (from higher bias to lower bias). This may be a pattern we want to emulate as we apply MST in general, as plausibly having monitor labels be interpolations, in a sense, improves the extrapolations. For example, maybe if we cover a high amount of “natural language space” with the monitor labels, we can make “interpolations” within that space well-defined. I don’t have a well-formalized way of thinking about this yet, but it seems relevant, so we may work on it!
Need a baseline where we finetune the model on the least biased articles and see how that compares with MST.
Strongly agree. The next round of experiments will have this.
Some ablation study testing semantic vs non-semantic control on the monitor label would be good to see if the model is understanding the natural language labels or if the model improves by just accessing some hidden variable about the evaluator. Semantic test is the regular monitor setup, non-semantic is using random IDs or some scalar for the monitor label.
This seems interesting! Maybe not semantic vs non-semantic, but arbitrary vs meaningful.
Thank you for these comments; I found them quite interesting/helpful. It seems like you have a good understanding of what we are trying to do, which is great.
nice read! here are some of my comments:
things that are good:
The idea is really nice and pretty simple, similar to other cool methods like inoculation-prompting.
Well designed sycophancy experiment, shuffling isolates effect of label presence from effect of label data
Nice section at the end discussing failure modes
some critiques:
political bias experiment:
Give some more explaination about how the synthetic data is generated
Currently may be more of a test of showing that MST can recover the lost capability that was destroyed during fine tuning on biased news articles. right now MST is framed as eliciting latent better behavior than what the model had before
Asking the baseline model (which was finetuned on biased news articles) to write unbiased news articles is pretty expected to not give unbiased results
Need to have a prompt-engineering control where we use the MST prompt at inference time on the baseline model
Not varying monitor strength, instead doing stylistic changes.
Are we extrapolating to better monitor, or just doing interpolation on the monitor axis?
More robust judge setup would be ideal (e.g. multiple models)
Need a baseline where we finetune the model on the least biased articles and see how that compares with MST.
other considations:
Some ablation study testing semantic vs non-semantic control on the monitor label would be good to see if the model is understanding the natural language labels or if the model improves by just accessing some hidden variable about the evaluator. Semantic test is the regular monitor setup, non-semantic is using random IDs or some scalar for the monitor label.
MST may improve best-case alignment but also improves worst-case deception: the model can get better at exploiting weak monitors. some experiment testing MST vs baseline on weak monitor could be useful to look for exploitation
We do this! Footnote 4 has the results.
Yes, technically MST is just recovering lost capability that was destroyed during fine-tuning on biased news articles here. The framing we are taking is something like, “Imagine all your articles are egregiously biased, but you still want to train article generation. What do you do?” The failure mode is exaggerated to show an effect with low compute. In future experiments we would like to actually elicit latent capabilities relative to an OOTB model. It’s worth noting, though, that in any case, we will never be able to prove higher capabilities with MST than one could have gotten otherwise because the act of proving implies evaluation abilities that MST assumes you don’t have. Proof of concept experiments will require us pretending we don’t have access to useful data that we do actually have so that we can use that data for testing.
We don’t include the OOTB model as a true baseline because in this case the OOTB has already been trained on a level of bias lower than we are pretending we have access to.
We are interpolating on the political spectrum (between right and left) and extrapolating on the bias spectrum (from higher bias to lower bias). This may be a pattern we want to emulate as we apply MST in general, as plausibly having monitor labels be interpolations, in a sense, improves the extrapolations. For example, maybe if we cover a high amount of “natural language space” with the monitor labels, we can make “interpolations” within that space well-defined. I don’t have a well-formalized way of thinking about this yet, but it seems relevant, so we may work on it!
Strongly agree. The next round of experiments will have this.
This seems interesting! Maybe not semantic vs non-semantic, but arbitrary vs meaningful.
Thank you for these comments; I found them quite interesting/helpful. It seems like you have a good understanding of what we are trying to do, which is great.