Two views that I have seen from the AI risk community on perverse incentives within academia are:
Incentives within the academic community are such that researchers are unable to pursue or publish work that is responsible, high-quality, and high-value. With few exceptions, anyone who attempts to do so will be outcompeted and miss out on funding, tenure, and social capital, which will eventually lead to their exit from academia.
Money, effort, and attention within the academic community are allocated by a process that is only loosely aligned with the goal of producing good research. Some researchers are able to avoid the trap and competently pursue valuable projects, but many will not, and a few will even go after projects that are harmful.
I think 1 is overplayed. There may be fields/subfields that are like that, but I think there is room in most fields for the right kind of people to pursue high-value research while succeeding in academia. I think 2 is a pretty big deal, though. I’m not too worried about all the people who will get NSF grants for unimportant research, though I am a little concerned that a flood of papers that are missing the point will go against the goal of legitimizing AI risk research for policy impact.
What I’m more worried about is research that is actively harmful. For example, my understanding is that a substantial portion of gain-of-function research has been funded by the federal government. This strikes me as frighteningly analogous to the kind of work that we should be concerned about in AI risk. I think this was mostly NIH, not NSF, so maybe there are good reasons for thinking the NSF is less likely to support dangerous work? Is there a strategy or an already-in-place mechanism for preventing people from using NSF funds for high-risk work? Or maybe there’s an important difference in incentives here that I’m not seeing?
For what it’s worth, I’m mostly agnostic on this, with a slight lean toward NSF attention being bad. Many of the people I most admire for their ability to solve difficult problems are academics, and I’m excited about the prospect of getting more people like that working on these problems. I really don’t want to dismiss it unfairly. I find it pretty easy to imagine worlds in which this goes very badly, but I think the default outcome is probably that a bunch of money goes to pointless stuff, a smaller amount goes to very valuable work, the field grows and diversifies, and (assuming timelines are long enough) the overall result is a reduction in AI risk. But I’m not very confident of this, and the downsides seem much larger than the potential benefits.
Two views that I have seen from the AI risk community on perverse incentives within academia are:
Incentives within the academic community are such that researchers are unable to pursue or publish work that is responsible, high-quality, and high-value. With few exceptions, anyone who attempts to do so will be outcompeted and miss out on funding, tenure, and social capital, which will eventually lead to their exit from academia.
Money, effort, and attention within the academic community are allocated by a process that is only loosely aligned with the goal of producing good research. Some researchers are able to avoid the trap and competently pursue valuable projects, but many will not, and a few will even go after projects that are harmful.
I think 1 is overplayed. There may be fields/subfields that are like that, but I think there is room in most fields for the right kind of people to pursue high-value research while succeeding in academia. I think 2 is a pretty big deal, though. I’m not too worried about all the people who will get NSF grants for unimportant research, though I am a little concerned that a flood of papers that are missing the point will go against the goal of legitimizing AI risk research for policy impact.
What I’m more worried about is research that is actively harmful. For example, my understanding is that a substantial portion of gain-of-function research has been funded by the federal government. This strikes me as frighteningly analogous to the kind of work that we should be concerned about in AI risk. I think this was mostly NIH, not NSF, so maybe there are good reasons for thinking the NSF is less likely to support dangerous work? Is there a strategy or an already-in-place mechanism for preventing people from using NSF funds for high-risk work? Or maybe there’s an important difference in incentives here that I’m not seeing?
For what it’s worth, I’m mostly agnostic on this, with a slight lean toward NSF attention being bad. Many of the people I most admire for their ability to solve difficult problems are academics, and I’m excited about the prospect of getting more people like that working on these problems. I really don’t want to dismiss it unfairly. I find it pretty easy to imagine worlds in which this goes very badly, but I think the default outcome is probably that a bunch of money goes to pointless stuff, a smaller amount goes to very valuable work, the field grows and diversifies, and (assuming timelines are long enough) the overall result is a reduction in AI risk. But I’m not very confident of this, and the downsides seem much larger than the potential benefits.