X-AI-Phi Lab, Japan Advanced Institute of Science and Technology
Masaharu Mizumoto
Karma: 11
Thanks for this question! This is an important and very relevant question.
Suppose we only label “acceptable action” but not “action of a virtuous character.”
We need to know which action is acceptable or not for each situation, for potentially infinite actions, which requires some theoretical considerations. This is therefore done in practice by giving a lot of rules today. This is what we have been claiming to be hopeless. On the other hand, we do have the concept of a virtuous person, which is also shared by LLMs today, from which we can derive our evaluations of actions in various (even novel) situations. Thus, our approach is much more “efficient” than giving various rules (which almost certainly conflict with each other).
By the way, I’m not the sole person but we are a team of philosophers and engineers.
Thanks, I appreciate the thoughtful comment and the reference.
If there are real concerns for this project, I would place them mainly at the level of implementation: how to collect moral intuitions in a systematic and reliable way, how much data is needed, and how to train models effectively on that basis. In that sense, if a model behaves poorly, that would point to shortcomings in training rather than a flaw in the overall direction.
The project you shared is certainly interesting as a piece of experimental philosophy of ethical analysis, but from an alignment perspective, it seems relatively indirect: using such data to actually build models that behave well would require several additional assumptions and steps.
That said, I would be interested to see how these approaches develop in practice.
The Goodhart problem typically arises when a proxy is optimized instead of the true objective, and in many alignment approaches, multiple explicit rules or objectives are set up. That is why optimizing them can lead to problematic behavior. But in our case, the scalar virtuosity score is not meant to be a proxy for something else, but an approximation of holistic human judgments about virtuous behavior.
If optimizing this signal led to non-virtuous behavior, that would just indicate a failure of the signal, rather than a standard Goodhart problem.
Also, thanks for the suggestion about the training environment. That is a valuable idea.
That’s a relational view of character, which I also agree with. In any case, character alignment is a long process, and we will need such training after the standard fine-tuning and RL, and even in an environment with a red team.