I think it bites somewhat on work engaging with much younger age groups (i.e. high school students). Outside of that, I think empirically the turnaround time I’ve seen between people getting engaged and doing useful work is really short in many cases (like < 1 year, even for some university students), such that in even very short timelines, most capacity-building work still looks very good.
abergal
Karma: 866
The case for AI safety capacity-building work
Funding for programs and events on global catastrophic risk, effective altruism, and other topics
Funding for work that builds capacity to address risks from transformative AI
Thanks for writing this up—at least for myself, I think I agree with the majority of this, and it articulates some important parts of how I live my life in ways that I hadn’t previously made explicit for myself.
Updates to Open Phil’s career development and transition funding program
The Long-Term Future Fund is looking for a full-time fund chair
Long-Term Future Fund Ask Us Anything (September 2023)
Long-Term Future Fund: April 2023 grant recommendations
I think your first point basically covers why—people are worried about alignment difficulties in superhuman systems, in particular (because those are the dangerous systems which can cause existential failures). I think a lot of current RLHF work is focused on providing reward signals to current systems in ways that don’t directly address the problem of “how do we reward systems with behaviors that have consequences that are too complicated for humans to understand”.
[AMA] Announcing Open Phil’s University Group Organizer and Century Fellowships [x-post]
Chris Olah wrote this topic prompt (with some feedback from me (Asya) and Nick Beckstead). We didn’t want to commit him to being responsible for this post or responding to comments on it, so we submitted this on his behalf. (I’ve changed the by-line to be more explicit about this.)
Truthful and honest AI
Interpretability
Techniques for enhancing human feedback
Measuring and forecasting risks
Request for proposals for projects in AI alignment that work with deep learning systems
Thanks for writing this! Would “fine-tune on some downstream task and measure the accuracy on that task before and after fine-tuning” count as measuring misalignment as you’re imagining it? My sense is that there might be a bunch of existing work like that.
This RFP is an experiment for us, and we don’t yet know if we’ll be doing more of them in the future. I think we’d be open to including research directions we think that are promising that apply equally well to both DL and non-DL systems—I’d be interested in hearing any particular suggestions you have.
(We’d also be happy to fund particular proposals in the research directions we’ve already listed that apply to both DL and non-DL systems, though we will be evaluating them on how well they address the DL-focused challenges we’ve presented.)
I’ve heard variants of this argument, and I overall haven’t found them that persuasive for reasons close to the ones Habryka gives—I think if you carve things up such that capacity-building work has been responsible for e.g. speeding up the creation of frontier AI labs, you should also credit it for the broader movement focused on catastrophic risks, and my intuition is the counterfactual world without that movement would be worse off overall. I don’t buy that the acceleration has been substantial enough that the unaccelerated world would have bought a lot of time for societal improvements useful for addressing catastrophic risks; instead, it feels to me like the unaccelerated world would be facing these risks more blindly and with less time to usefully prepare.
I also think given the massive amount of non-GCR-related interest and resources in AI now, the forward-looking acceleration effects seem likely to be much smaller than any historic effect. I generally think the ratio of “meaningfully adding to the talent pool of people working on catastrophic risks” to “meaningfully accelerating AI capabilities” for most CB programs will look extremely favorable.