Great post thank you, it has given me much to think about.
We think personas are a learning dynamics story of this kind, where the training corpus is important, but also the order in which that corpus is arranged.
Do you think intervening requires a full training run? That would mean only the hyperscalers could apply it to frontier models. I hope that lesser versions can be done in post-training.
I tried some early attempts at weak to strong iterative character steering. While results on small models convinced me it’s a tractable direction (its likely to be easier on larger models that are competent to use the harness). It worked even with a weak teacher and strong student (https://github.com/wassname/w2schar-mini/).
(copied from Twitter)
Another property: is it robust to OOS generalization? Considering generalization is hard in ML, and some papers (Deep Values https://arxiv.org/abs/2511.02109) hint that we fail this test currently, this may be a necessary condition. Besides, deployment, especially in the future, is mostly out of distribution, so it’s an important property to have.
It might be achievable, if character stays coherent further out of scope than capabilities, and capabilities degrade gracefully, that should be enough.
Great post thank you, it has given me much to think about.
Do you think intervening requires a full training run? That would mean only the hyperscalers could apply it to frontier models. I hope that lesser versions can be done in post-training.
I tried some early attempts at weak to strong iterative character steering. While results on small models convinced me it’s a tractable direction (its likely to be easier on larger models that are competent to use the harness). It worked even with a weak teacher and strong student (https://github.com/wassname/w2schar-mini/).
(copied from Twitter) Another property: is it robust to OOS generalization? Considering generalization is hard in ML, and some papers (Deep Values https://arxiv.org/abs/2511.02109) hint that we fail this test currently, this may be a necessary condition. Besides, deployment, especially in the future, is mostly out of distribution, so it’s an important property to have.
It might be achievable, if character stays coherent further out of scope than capabilities, and capabilities degrade gracefully, that should be enough.