Studying model specs: Claude’s constitution is very different from OpenAI’s model spec. Which one is better? What are the most important axes along which constitutions can vary, and which end should we prefer for each of those axes? Some axes that seem good to vary:
Emphasizing consequentialism vs. deontology vs. virtue ethics (see e.g. Richard Ngo’s Aligning to Virtues)
Different sets of core values and different priority hierarchies between them (see the Claude’s core values section in Claude’s Constitution)
Egg Syntax and I have discussed perhaps doing some work in this area: probably starting more theoretical, but then trying to come up with and test experimental results. Having an experimental Character Training setup for testing this (outside Anthropic) would be really nice. I would expect this is an area where model size/capability and perhaps also reasoning might make a significant effect: I’d be rather dubious of extrapolating from say a 7B non-reasoning model up to a frontier model for this sort of deeply conceptual work, so I suspect we’d need to train models fairly close to frontier level. That might be an early thing to test: find a fairly clear nontrivial result in large models, then see how well it reproduces in smaller models.
This sounds great, please keep me updated in case you end up working on this!
Having an experimental Character Training setup for testing this (outside Anthropic) would be really nice. I would expect this is an area where model size/capability and perhaps also reasoning might make a significant effect: I’d be rather dubious of extrapolating from say a 7B non-reasoning model up to a frontier model for this sort of deeply conceptual work, so I suspect we’d need to train models fairly close to frontier level.
I agree that model size could matter a lot. I think the Maiya et al. training pipeline is a reasonable starting point, though they only tested it in the 4B to 8B size range and focused on character traits rather than model specs. I think the authors are scaling up the experiments to larger models, you should probably reach out to them.
Egg Syntax and I have discussed perhaps doing some work in this area: probably starting more theoretical, but then trying to come up with and test experimental results. Having an experimental Character Training setup for testing this (outside Anthropic) would be really nice. I would expect this is an area where model size/capability and perhaps also reasoning might make a significant effect: I’d be rather dubious of extrapolating from say a 7B non-reasoning model up to a frontier model for this sort of deeply conceptual work, so I suspect we’d need to train models fairly close to frontier level. That might be an early thing to test: find a fairly clear nontrivial result in large models, then see how well it reproduces in smaller models.
This sounds great, please keep me updated in case you end up working on this!
I agree that model size could matter a lot. I think the Maiya et al. training pipeline is a reasonable starting point, though they only tested it in the 4B to 8B size range and focused on character traits rather than model specs. I think the authors are scaling up the experiments to larger models, you should probably reach out to them.
We’re now confirmed to be working on this this summer under a PIBBSS fellowship.