Really not sure what heuristic leads you to count people working on ARC-Theory working on an ambitious, speculative version of interp as working on alignment but not any of the people working to build from current interp paradigms. Similarly, anyone working on e.g. making models more honest in prod models is in fact learning a bunch of lessons about what scalable oversight looks like (albeit not publishing, which i agree is sad). Or doing any science of misalignment, or doing any empirical character work, or experimenting with making models adhere to a spec, or carefully understanding their generalisation patterns, or just trying to understand what the actual objects that we are creating right now are??
It seems like having any current interaction with frontier models is seen as disqualifying for actually doing alignment work?
anyone working on e.g. making models more honest in prod models
I don’t really think people working on ‘what instruciton can I add to the system prompt’ or equivalents are meaningfully working on the kind of endgame alignment the post is talking about.
Edit: Nothing wrong with that kind of work for current alignment, just doesn’t apply to endgame alignment all that much in my opinion.
Aligning AGI is very different to aligning current frontier models: what works for current systems doesn’t tell you that much about what works for superintelligent systems
To the extent that your goal is to align current systems, you will gravitate towards approaches that don’t actually scale, because the low-hanging fruit now is stuff that depends on the model being weak
(The term alignment should sort of be reserved for the AGI/ASI case)
FWIW I’m not sure how much I buy these but I’d guess I buy them more than you? This is unfortunately another great example of something where people inside labs probably have some pretty relevant private information but also extra incentive/selection problems.
Some of us have been thinking about this for years. See for example Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI? in which I suggested that love might be a particularly useful motivator, about the only one that would still work at ASI. A couple of months ago Anthropic published Emotion Concepts and their Function in a Large Language Model showing among other things that “loving” was one of Claude’s most dominant emotions: it turns out RLHF had quietly implemented my suggestion without anyone needing to actually engineer this.
Really not sure what heuristic leads you to count people working on ARC-Theory working on an ambitious, speculative version of interp as working on alignment but not any of the people working to build from current interp paradigms. Similarly, anyone working on e.g. making models more honest in prod models is in fact learning a bunch of lessons about what scalable oversight looks like (albeit not publishing, which i agree is sad). Or doing any science of misalignment, or doing any empirical character work, or experimenting with making models adhere to a spec, or carefully understanding their generalisation patterns, or just trying to understand what the actual objects that we are creating right now are??
It seems like having any current interaction with frontier models is seen as disqualifying for actually doing alignment work?
I don’t really think people working on ‘what instruciton can I add to the system prompt’ or equivalents are meaningfully working on the kind of endgame alignment the post is talking about.
Edit: Nothing wrong with that kind of work for current alignment, just doesn’t apply to endgame alignment all that much in my opinion.
I’d guess the heuristics are basically:
Aligning AGI is very different to aligning current frontier models: what works for current systems doesn’t tell you that much about what works for superintelligent systems
To the extent that your goal is to align current systems, you will gravitate towards approaches that don’t actually scale, because the low-hanging fruit now is stuff that depends on the model being weak
(The term alignment should sort of be reserved for the AGI/ASI case)
FWIW I’m not sure how much I buy these but I’d guess I buy them more than you? This is unfortunately another great example of something where people inside labs probably have some pretty relevant private information but also extra incentive/selection problems.
Some of us have been thinking about this for years. See for example Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI? in which I suggested that love might be a particularly useful motivator, about the only one that would still work at ASI. A couple of months ago Anthropic published Emotion Concepts and their Function in a Large Language Model showing among other things that “loving” was one of Claude’s most dominant emotions: it turns out RLHF had quietly implemented my suggestion without anyone needing to actually engineer this.