I think the term “AGI-pilled” fails to distinguish a fundamental difference, something like “is this person trying to think for themselves” vs “is this person following prestige gradients/social incentives”.
My concern is that people who are trying to think for themselves about what advanced AI will look like drive a lot of acceleration, by laying out a conceptual roadmap for what comes next (and then in many cases actually just working on that conceptual roadmap).
Whereas ML researchers who are now willing to talk about AGI are mostly doing so because it’s socially acceptable, and therefore will do a much worse job of innovative thinking about the future of AI.
(For example, most ML researchers—including those at DeepMind and OpenAI—who you might call “AGI-pilled” don’t take recursive self-improvement or superintelligence seriously, because that’s still socially weird. Whereas Anthropic people tend to take those things significantly more seriously, which helped them to catch up with/arguably overtake DeepMind and OpenAI starting from very far behind.)
It’s interesting that you bring up this comment from Beth btw because I think she’s been making a very similar mistake in underestimating how much METR is contributing to acceleration.
So the safety community had a somewhat small fraction of the overall capabilities work, but a considerably larger fraction of the capabilities work specifically by AGI-pilled people and therefore ended up helping create AGI labs.
I think it’s important to include “helping create AGI labs” as capabilities work, which would mean the safety community actually did a very significant fraction of the overall capabilities work from the last decade. Without accounting for that, all our estimates about the capabilities externalities of different work will be very off.
I think the term “AGI-pilled” fails to distinguish a fundamental difference, something like “is this person trying to think for themselves” vs “is this person following prestige gradients/social incentives”.
My concern is that people who are trying to think for themselves about what advanced AI will look like drive a lot of acceleration, by laying out a conceptual roadmap for what comes next (and then in many cases actually just working on that conceptual roadmap).
Whereas ML researchers who are now willing to talk about AGI are mostly doing so because it’s socially acceptable, and therefore will do a much worse job of innovative thinking about the future of AI.
(For example, most ML researchers—including those at DeepMind and OpenAI—who you might call “AGI-pilled” don’t take recursive self-improvement or superintelligence seriously, because that’s still socially weird. Whereas Anthropic people tend to take those things significantly more seriously, which helped them to catch up with/arguably overtake DeepMind and OpenAI starting from very far behind.)
It’s interesting that you bring up this comment from Beth btw because I think she’s been making a very similar mistake in underestimating how much METR is contributing to acceleration.
I think it’s important to include “helping create AGI labs” as capabilities work, which would mean the safety community actually did a very significant fraction of the overall capabilities work from the last decade. Without accounting for that, all our estimates about the capabilities externalities of different work will be very off.