However, here’s an observation: the genuine potential to make such a contribution is itself highly illegible. Not only to the field, but perhaps to the potential contributor as well.
Right. I spent a fair amount of effort (let’s say, maybe tens of hours across fives of people) trying to point out something like this. Like:
Ok it’s great that you’re enthusiastic, but you probably aren’t yet perceiving how the problem is hard, and aren’t thinking through what it would take to actually contribute. So you’re probably not able to evaluate either whether/how you could contribute or whether you would want to. This is all fine and natural; and it doesn’t mean you shouldn’t (or should) investigate more; but it does mean that you shouldn’t necessarily expect to be motivated on this project long-term in the same way that you currenly feel motivated, because your motivation will probably have to update in form—whether it goes up or down, it will probably have to change shape to be about some aspects of X-derisking and not others, or about some of your skills and not others, or some of your hopes/desires and not others, etc.
On your question:
It seems like the only way they could make themselves sufficiently legibly useful would be to work their way up through the ranks of much less abstract technical AI research until they distinguish themselves. That’s likely to deter horizontal movement of mature talent from other disciplines. I’m curious if you think this is true; or if you think starting out in object-level technical research is necessary training/preparation for the kind of big-picture work you have in mind; or if you think there are other routes of entry.
Yeah I think this is a big problem and I don’t know what to do about it (and I’m not working on it). One could do the plan you propose; I imagine many people are currently / have been trying that, but I don’t know how it’s going (I would guess not very well). Intellectual inquiry takes a lot of mental energy and attention. Personally, I think in practice, I get about .7 slots for a serious inquiry. In other words, if I try extra hard and push beyond my default capacity, I might be able to really seriously investigate one thing for months/years on end. Definitely not 2. I have suggested to several (maybe 10-ish) people something like: “Get some sort of job or other position; and do 5 hours of totally-from-scratch original-seeing AGI alignment research per week; and start writing about that; and then if you’re making some kind of progress / gaining steam, do more and maybe look for work or something”. But IDK if that has worked for anyone. Well, maybe it worked for some people who ended up doing research at MIRI (e.g. me, Sam), but at least in some cases the job was random (a random SWE job or grad student position), not “kinda AI technical safety something something”. My guess is that in practice when people do this, they instead adopt the opinions of the job they work for, and lose their ability or motivation or slack that would be needed to do serious sustained original seeing investigation. But that’s just speculation.
In the Talent Needs post, and also in my comments e.g. here, there’s the idea of “pay newcomers to actually spend 1-4 years of de novo investigation”. This is pretty expensive for everyone involved, but it’s the obvious guess at how to actually test someone’s ability to contribute.
It may be a waste of hope for most people. On the other hand, I think usually people update away from “I’m going to contribute to technical AGI alignment” fairly quickly-ish, because they spin their wheels and get nowhere and they can kinda tell. So maybe it’s not so bad. Also it maybe does take perseverance at least in many cases..… Yeah I don’t have clear answers.
Right. I spent a fair amount of effort (let’s say, maybe tens of hours across fives of people) trying to point out something like this. Like:
On your question:
Yeah I think this is a big problem and I don’t know what to do about it (and I’m not working on it). One could do the plan you propose; I imagine many people are currently / have been trying that, but I don’t know how it’s going (I would guess not very well). Intellectual inquiry takes a lot of mental energy and attention. Personally, I think in practice, I get about .7 slots for a serious inquiry. In other words, if I try extra hard and push beyond my default capacity, I might be able to really seriously investigate one thing for months/years on end. Definitely not 2. I have suggested to several (maybe 10-ish) people something like: “Get some sort of job or other position; and do 5 hours of totally-from-scratch original-seeing AGI alignment research per week; and start writing about that; and then if you’re making some kind of progress / gaining steam, do more and maybe look for work or something”. But IDK if that has worked for anyone. Well, maybe it worked for some people who ended up doing research at MIRI (e.g. me, Sam), but at least in some cases the job was random (a random SWE job or grad student position), not “kinda AI technical safety something something”. My guess is that in practice when people do this, they instead adopt the opinions of the job they work for, and lose their ability or motivation or slack that would be needed to do serious sustained original seeing investigation. But that’s just speculation.
In the Talent Needs post, and also in my comments e.g. here, there’s the idea of “pay newcomers to actually spend 1-4 years of de novo investigation”. This is pretty expensive for everyone involved, but it’s the obvious guess at how to actually test someone’s ability to contribute.
It may be a waste of hope for most people. On the other hand, I think usually people update away from “I’m going to contribute to technical AGI alignment” fairly quickly-ish, because they spin their wheels and get nowhere and they can kinda tell. So maybe it’s not so bad. Also it maybe does take perseverance at least in many cases..… Yeah I don’t have clear answers.