Aligning AGI is very different to aligning current frontier models: what works for current systems doesn’t tell you that much about what works for superintelligent systems
To the extent that your goal is to align current systems, you will gravitate towards approaches that don’t actually scale, because the low-hanging fruit now is stuff that depends on the model being weak
(The term alignment should sort of be reserved for the AGI/ASI case)
FWIW I’m not sure how much I buy these but I’d guess I buy them more than you? This is unfortunately another great example of something where people inside labs probably have some pretty relevant private information but also extra incentive/selection problems.
Some of us have been thinking about this for years. See for example Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI? in which I suggested that love might be a particularly useful motivator, about the only one that would still work at ASI. A couple of months ago Anthropic published Emotion Concepts and their Function in a Large Language Model showing among other things that “loving” was one of Claude’s most dominant emotions: it turns out RLHF had quietly implemented my suggestion without anyone needing to actually engineer this.
I’d guess the heuristics are basically:
Aligning AGI is very different to aligning current frontier models: what works for current systems doesn’t tell you that much about what works for superintelligent systems
To the extent that your goal is to align current systems, you will gravitate towards approaches that don’t actually scale, because the low-hanging fruit now is stuff that depends on the model being weak
(The term alignment should sort of be reserved for the AGI/ASI case)
FWIW I’m not sure how much I buy these but I’d guess I buy them more than you? This is unfortunately another great example of something where people inside labs probably have some pretty relevant private information but also extra incentive/selection problems.
Some of us have been thinking about this for years. See for example Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI? in which I suggested that love might be a particularly useful motivator, about the only one that would still work at ASI. A couple of months ago Anthropic published Emotion Concepts and their Function in a Large Language Model showing among other things that “loving” was one of Claude’s most dominant emotions: it turns out RLHF had quietly implemented my suggestion without anyone needing to actually engineer this.