I get excited about the possibility of contributing to AI alignment research whenever people talk about the problem being hard in this particular way. The problems are still illegible. The field needs relentless Original Seeing. Every assumption needs to be Truly Doubted. The approaches that will turn out to be fruitful probably haven’t even been imagined yet. It will be important to be able to learn and defer while simultaneously questioning, to come up with bold ideas while not being attached to them. It will require knowing your stuff, yet it might be optimal not to know too much about what other people have thought so far (“I would suggest not reading very much more.”). It’ll be important for people attempting this to be exceptionally good at rigorous, original conceptual thinking, thinking about thinking, grappling with minds, zooming all the way in and all the way out, constantly. It’ll probably require making ego and career sacrifices. Bring it on!
However, here’s an observation: the genuine potential to make such a contribution is itself highly illegible. Not only to the field, but perhaps to the potential contributor as well.
Apparently a lot of people fancy themselves to have Big Ideas or to be great at big-picture thinking, and most of them aren’t nearly as useful as they think they are. I feel like I’ve seen that sentiment many times on LW, and I’m guessing that’s behind statements like:
It’s not remotely sufficient, and is often anti-helpful, to just be like “Wait, actually what even is alignment? Alignment to what?”.
This presents a bit of a paradox. Suppose there exist a few rare, high-potential contributors not already working in AI who would be willing and able to take up the challenge you describe. It seems like the only way they could make themselves sufficiently legibly useful would be to work their way up through the ranks of much less abstract technical AI research until they distinguish themselves. That’s likely to deter horizontal movement of mature talent from other disciplines. I’m curious if you think this is true; or if you think starting out in object-level technical research is necessary training/preparation for the kind of big-picture work you have in mind; or if you think there are other routes of entry.
However, here’s an observation: the genuine potential to make such a contribution is itself highly illegible. Not only to the field, but perhaps to the potential contributor as well.
Right. I spent a fair amount of effort (let’s say, maybe tens of hours across fives of people) trying to point out something like this. Like:
Ok it’s great that you’re enthusiastic, but you probably aren’t yet perceiving how the problem is hard, and aren’t thinking through what it would take to actually contribute. So you’re probably not able to evaluate either whether/how you could contribute or whether you would want to. This is all fine and natural; and it doesn’t mean you shouldn’t (or should) investigate more; but it does mean that you shouldn’t necessarily expect to be motivated on this project long-term in the same way that you currenly feel motivated, because your motivation will probably have to update in form—whether it goes up or down, it will probably have to change shape to be about some aspects of X-derisking and not others, or about some of your skills and not others, or some of your hopes/desires and not others, etc.
On your question:
It seems like the only way they could make themselves sufficiently legibly useful would be to work their way up through the ranks of much less abstract technical AI research until they distinguish themselves. That’s likely to deter horizontal movement of mature talent from other disciplines. I’m curious if you think this is true; or if you think starting out in object-level technical research is necessary training/preparation for the kind of big-picture work you have in mind; or if you think there are other routes of entry.
Yeah I think this is a big problem and I don’t know what to do about it (and I’m not working on it). One could do the plan you propose; I imagine many people are currently / have been trying that, but I don’t know how it’s going (I would guess not very well). Intellectual inquiry takes a lot of mental energy and attention. Personally, I think in practice, I get about .7 slots for a serious inquiry. In other words, if I try extra hard and push beyond my default capacity, I might be able to really seriously investigate one thing for months/years on end. Definitely not 2. I have suggested to several (maybe 10-ish) people something like: “Get some sort of job or other position; and do 5 hours of totally-from-scratch original-seeing AGI alignment research per week; and start writing about that; and then if you’re making some kind of progress / gaining steam, do more and maybe look for work or something”. But IDK if that has worked for anyone. Well, maybe it worked for some people who ended up doing research at MIRI (e.g. me, Sam), but at least in some cases the job was random (a random SWE job or grad student position), not “kinda AI technical safety something something”. My guess is that in practice when people do this, they instead adopt the opinions of the job they work for, and lose their ability or motivation or slack that would be needed to do serious sustained original seeing investigation. But that’s just speculation.
In the Talent Needs post, and also in my comments e.g. here, there’s the idea of “pay newcomers to actually spend 1-4 years of de novo investigation”. This is pretty expensive for everyone involved, but it’s the obvious guess at how to actually test someone’s ability to contribute.
It may be a waste of hope for most people. On the other hand, I think usually people update away from “I’m going to contribute to technical AGI alignment” fairly quickly-ish, because they spin their wheels and get nowhere and they can kinda tell. So maybe it’s not so bad. Also it maybe does take perseverance at least in many cases..… Yeah I don’t have clear answers.
I get excited about the possibility of contributing to AI alignment research whenever people talk about the problem being hard in this particular way. The problems are still illegible. The field needs relentless Original Seeing. Every assumption needs to be Truly Doubted. The approaches that will turn out to be fruitful probably haven’t even been imagined yet. It will be important to be able to learn and defer while simultaneously questioning, to come up with bold ideas while not being attached to them. It will require knowing your stuff, yet it might be optimal not to know too much about what other people have thought so far (“I would suggest not reading very much more.”). It’ll be important for people attempting this to be exceptionally good at rigorous, original conceptual thinking, thinking about thinking, grappling with minds, zooming all the way in and all the way out, constantly. It’ll probably require making ego and career sacrifices. Bring it on!
However, here’s an observation: the genuine potential to make such a contribution is itself highly illegible. Not only to the field, but perhaps to the potential contributor as well.
Apparently a lot of people fancy themselves to have Big Ideas or to be great at big-picture thinking, and most of them aren’t nearly as useful as they think they are. I feel like I’ve seen that sentiment many times on LW, and I’m guessing that’s behind statements like:
or the Talent Needs post that said
This presents a bit of a paradox. Suppose there exist a few rare, high-potential contributors not already working in AI who would be willing and able to take up the challenge you describe. It seems like the only way they could make themselves sufficiently legibly useful would be to work their way up through the ranks of much less abstract technical AI research until they distinguish themselves. That’s likely to deter horizontal movement of mature talent from other disciplines. I’m curious if you think this is true; or if you think starting out in object-level technical research is necessary training/preparation for the kind of big-picture work you have in mind; or if you think there are other routes of entry.
Right. I spent a fair amount of effort (let’s say, maybe tens of hours across fives of people) trying to point out something like this. Like:
On your question:
Yeah I think this is a big problem and I don’t know what to do about it (and I’m not working on it). One could do the plan you propose; I imagine many people are currently / have been trying that, but I don’t know how it’s going (I would guess not very well). Intellectual inquiry takes a lot of mental energy and attention. Personally, I think in practice, I get about .7 slots for a serious inquiry. In other words, if I try extra hard and push beyond my default capacity, I might be able to really seriously investigate one thing for months/years on end. Definitely not 2. I have suggested to several (maybe 10-ish) people something like: “Get some sort of job or other position; and do 5 hours of totally-from-scratch original-seeing AGI alignment research per week; and start writing about that; and then if you’re making some kind of progress / gaining steam, do more and maybe look for work or something”. But IDK if that has worked for anyone. Well, maybe it worked for some people who ended up doing research at MIRI (e.g. me, Sam), but at least in some cases the job was random (a random SWE job or grad student position), not “kinda AI technical safety something something”. My guess is that in practice when people do this, they instead adopt the opinions of the job they work for, and lose their ability or motivation or slack that would be needed to do serious sustained original seeing investigation. But that’s just speculation.
In the Talent Needs post, and also in my comments e.g. here, there’s the idea of “pay newcomers to actually spend 1-4 years of de novo investigation”. This is pretty expensive for everyone involved, but it’s the obvious guess at how to actually test someone’s ability to contribute.
It may be a waste of hope for most people. On the other hand, I think usually people update away from “I’m going to contribute to technical AGI alignment” fairly quickly-ish, because they spin their wheels and get nowhere and they can kinda tell. So maybe it’s not so bad. Also it maybe does take perseverance at least in many cases..… Yeah I don’t have clear answers.