Some discussion on whether alignment should see more influence from AGI labs or academia. I use the same argument in favor of a strong decoupling of alignment progress from both: alignment progress needs to go faster than capability progress. If we use the same methods or cultural technology as AGI labs or academia, we can guarantee slower than capability alignment progress. Just as fast as if AGI labs and academia work well for alignment as much as they work for capabilities. Given they are driven by capabilities progress and not alignment progress, they probably will work far better for capabilities progress.
Hm. Good points. I guess what I really mean with the academia points is that it seems like academia has many blockers and inefficiencies that I think are made in such a way so that capabilities progress is vastly easier than alignment progress to jump through, and extra-so for capabilities labs. Like, right now it seems like a lot of alignment work is just playing with a bunch of different reframings of the problems to see what sticks or makes problems easier.
You have more experience here, but my impression of a lot of academia was that it was very focused on publishing lots of papers with very legible results (and also a meaningless theory section). In such a world, playing around with different framings of problems doesn’t succeed, and you end up pushed towards framings which are better on the currently used metrics. Most currently used metrics for AI stuff are capabilities oriented, so that means doing capabilities work, or work that helps push capabilities.
I think it’s true that the easiest thing to do is legibly improve on currently used metrics. I guess my take is that in academia you want to write a short paper that people can see is valuable, which biases towards “I did thing X and now the number is bigger”. But, for example, if you reframe the alignment problem and show some interesting thing about your reframing, that can work pretty well as a paper (see The Off-Switch Game, Optimal Policies Tend to Seek Power). My guess is that the bigger deal is that there’s some social pressure to publish frequently (in part because that’s a sign that you’ve done something, and a thing that closes a feedback loop).
The current ecosystem seems very influenced by AGI labs, so it seems clear to me that a marginal increase in their influence is bad. How bad? I don’t know.
There’s little influence of academia, which seems good. The benefit of marginal increases in interactions with academia come down to locating the holes in our understanding of various claims we make, and potentially some course-corrections potentially helpful for more speculative research. Not tremendously obvious which direction the sign here is pointing, but I do think its easy for people to worship academia as a beacon of truth & clarity, or as a way to lend status to alignment arguments. These are bad reasons to want more influence from academia.
Some discussion on whether alignment should see more influence from AGI labs or academia. I use the same argument in favor of a strong decoupling of alignment progress from both: alignment progress needs to go faster than capability progress. If we use the same methods or cultural technology as AGI labs or academia, we can guarantee slower than capability alignment progress. Just as fast as if AGI labs and academia work well for alignment as much as they work for capabilities. Given they are driven by capabilities progress and not alignment progress, they probably will work far better for capabilities progress.
This seems wrong to me about academia—I’d say it’s driven by “learning cool things you can summarize in a talk”.
Also in general I feel like this logic would also work for why we shouldn’t work inside buildings, or with computers.
Hm. Good points. I guess what I really mean with the academia points is that it seems like academia has many blockers and inefficiencies that I think are made in such a way so that capabilities progress is vastly easier than alignment progress to jump through, and extra-so for capabilities labs. Like, right now it seems like a lot of alignment work is just playing with a bunch of different reframings of the problems to see what sticks or makes problems easier.
You have more experience here, but my impression of a lot of academia was that it was very focused on publishing lots of papers with very legible results (and also a meaningless theory section). In such a world, playing around with different framings of problems doesn’t succeed, and you end up pushed towards framings which are better on the currently used metrics. Most currently used metrics for AI stuff are capabilities oriented, so that means doing capabilities work, or work that helps push capabilities.
I think it’s true that the easiest thing to do is legibly improve on currently used metrics. I guess my take is that in academia you want to write a short paper that people can see is valuable, which biases towards “I did thing X and now the number is bigger”. But, for example, if you reframe the alignment problem and show some interesting thing about your reframing, that can work pretty well as a paper (see The Off-Switch Game, Optimal Policies Tend to Seek Power). My guess is that the bigger deal is that there’s some social pressure to publish frequently (in part because that’s a sign that you’ve done something, and a thing that closes a feedback loop).
Maybe a bigger deal is that by the nature of a paper, you can’t get too many inferential steps away from the field.
The current ecosystem seems very influenced by AGI labs, so it seems clear to me that a marginal increase in their influence is bad. How bad? I don’t know.
There’s little influence of academia, which seems good. The benefit of marginal increases in interactions with academia come down to locating the holes in our understanding of various claims we make, and potentially some course-corrections potentially helpful for more speculative research. Not tremendously obvious which direction the sign here is pointing, but I do think its easy for people to worship academia as a beacon of truth & clarity, or as a way to lend status to alignment arguments. These are bad reasons to want more influence from academia.