It seems likely to me that maybe 50% of people who start seriously studying or working on AI safety in the next year will be below the intelligence escape velocity, where they forever lag behind frontier AI systems in AI research ability. If I were working in capacity building, I would already start to deprioritize the earlier parts of the funnel for this reason. For reference, my TEDAI timelines are 2028.
Very little of the impact of people working in AI Safety is downstream of their research, so this seems wrong. Maybe you also think the same will be true for policy/communications/evals work?
Separately, I don’t super buy this even for research. Capacity-building work attracts the smartest people, and those people will have been using the models to upgrade themselves anyways. It’s not the case that you need to have 5+ years of research to usefully contribute, especially with models providing uplift to fill in the gaps.
Most of the impact comes from helping people understand better what’s up with AI, what the stakes of the whole situation are, what might be ways we could mitigate the risk (which at the moment mostly don’t look like technical safety research, though the effect of that is not zero), and generally helping people grapple with what’s going on. Another big avenue for positive impact is building tools, or deploying AI systems, in contexts where they might help end the acute risk period (e.g. via facilitating coordination among nations, or companies or politicians).
I do think most current alignment work is useless, but that’s actually not really upstream of my beliefs here. Most work in any field is useless, and it would be surprising if alignment would be any different. I do think marginal technical alignment work is quite unlikely to make a big differences, though there are some things that I do think are worth quite a bit of investment, though they seem relatively neglected (and people have a surprising skill to take pointers to promising approaches and then co-opt them to mean something else in an attempt to recruit talent or investment or field off political attacks, so pointing at them is quite tricky).
Even if new people’s impact isn’t downstream of research, the inability to contribute to research significantly hinders their progress, because potential mentors and collaborators won’t benefit from working with them, and they generally get fewer opportunities to advance their careers.
Yeah this argument applies much less to policy/communications work.
I agree that you don’t need 5+ years of research to conntribute, but I expect that at some point soon, if you don’t have some minimum amount of context on AI safety, your job (managing a bunch of LLM agents) can just be automated by an LLM agent.
and they generally get fewer opportunities to advance their careers.
I don’t know what this means? Like, they won’t get invited to talk to policy makers? Or won’t be successful writers informing people about the risks? Are you talking about lab employment here?
I agree that you (hopefully) won’t get that many new lab employees out of recent recruitment efforts, but that’s a plus not a minus!
It seems likely to me that maybe 50% of people who start seriously studying or working on AI safety in the next year will be below the intelligence escape velocity, where they forever lag behind frontier AI systems in AI research ability. If I were working in capacity building, I would already start to deprioritize the earlier parts of the funnel for this reason. For reference, my TEDAI timelines are 2028.
Very little of the impact of people working in AI Safety is downstream of their research, so this seems wrong. Maybe you also think the same will be true for policy/communications/evals work?
Separately, I don’t super buy this even for research. Capacity-building work attracts the smartest people, and those people will have been using the models to upgrade themselves anyways. It’s not the case that you need to have 5+ years of research to usefully contribute, especially with models providing uplift to fill in the gaps.
Where do you think the impact comes from? And is this coming from a background belief that most current alignment work is useless?
Most of the impact comes from helping people understand better what’s up with AI, what the stakes of the whole situation are, what might be ways we could mitigate the risk (which at the moment mostly don’t look like technical safety research, though the effect of that is not zero), and generally helping people grapple with what’s going on. Another big avenue for positive impact is building tools, or deploying AI systems, in contexts where they might help end the acute risk period (e.g. via facilitating coordination among nations, or companies or politicians).
I do think most current alignment work is useless, but that’s actually not really upstream of my beliefs here. Most work in any field is useless, and it would be surprising if alignment would be any different. I do think marginal technical alignment work is quite unlikely to make a big differences, though there are some things that I do think are worth quite a bit of investment, though they seem relatively neglected (and people have a surprising skill to take pointers to promising approaches and then co-opt them to mean something else in an attempt to recruit talent or investment or field off political attacks, so pointing at them is quite tricky).
Even if new people’s impact isn’t downstream of research, the inability to contribute to research significantly hinders their progress, because potential mentors and collaborators won’t benefit from working with them, and they generally get fewer opportunities to advance their careers.
Yeah this argument applies much less to policy/communications work.
I agree that you don’t need 5+ years of research to conntribute, but I expect that at some point soon, if you don’t have some minimum amount of context on AI safety, your job (managing a bunch of LLM agents) can just be automated by an LLM agent.
I don’t know what this means? Like, they won’t get invited to talk to policy makers? Or won’t be successful writers informing people about the risks? Are you talking about lab employment here?
I agree that you (hopefully) won’t get that many new lab employees out of recent recruitment efforts, but that’s a plus not a minus!