The Goodhart problem typically arises when a proxy is optimized instead of the true objective, and in many alignment approaches, multiple explicit rules or objectives are set up. That is why optimizing them can lead to problematic behavior. But in our case, the scalar virtuosity score is not meant to be a proxy for something else, but an approximation of holistic human judgments about virtuous behavior.
If optimizing this signal led to non-virtuous behavior, that would just indicate a failure of the signal, rather than a standard Goodhart problem.
From experience I’m still skeptical it won’t be a proxy in practice once you actually implement the scalar virtuosity score and fine-tune LLMs against it, but instead of belaboring my skepticism I’ll wait for your preliminary results. I’d be keen to see something roughly as thorough as this but probably a bit much if you’re the sole person working on this.
Thanks, I appreciate the thoughtful comment and the reference.
If there are real concerns for this project, I would place them mainly at the level of implementation: how to collect moral intuitions in a systematic and reliable way, how much data is needed, and how to train models effectively on that basis. In that sense, if a model behaves poorly, that would point to shortcomings in training rather than a flaw in the overall direction.
The project you shared is certainly interesting as a piece of experimental philosophy of ethical analysis, but from an alignment perspective, it seems relatively indirect: using such data to actually build models that behave well would require several additional assumptions and steps.
That said, I would be interested to see how these approaches develop in practice.
The Goodhart problem typically arises when a proxy is optimized instead of the true objective, and in many alignment approaches, multiple explicit rules or objectives are set up. That is why optimizing them can lead to problematic behavior. But in our case, the scalar virtuosity score is not meant to be a proxy for something else, but an approximation of holistic human judgments about virtuous behavior.
If optimizing this signal led to non-virtuous behavior, that would just indicate a failure of the signal, rather than a standard Goodhart problem.
From experience I’m still skeptical it won’t be a proxy in practice once you actually implement the scalar virtuosity score and fine-tune LLMs against it, but instead of belaboring my skepticism I’ll wait for your preliminary results. I’d be keen to see something roughly as thorough as this but probably a bit much if you’re the sole person working on this.
Thanks, I appreciate the thoughtful comment and the reference.
If there are real concerns for this project, I would place them mainly at the level of implementation: how to collect moral intuitions in a systematic and reliable way, how much data is needed, and how to train models effectively on that basis. In that sense, if a model behaves poorly, that would point to shortcomings in training rather than a flaw in the overall direction.
The project you shared is certainly interesting as a piece of experimental philosophy of ethical analysis, but from an alignment perspective, it seems relatively indirect: using such data to actually build models that behave well would require several additional assumptions and steps.
That said, I would be interested to see how these approaches develop in practice.
By the way, I’m not the sole person but we are a team of philosophers and engineers.