Lots of people have plans to make the world better by improving X to improve Y to improve Z, where it’s Z that matters. If Z is more than 3y out, consider finding a different plan.
This really hits on an emotional level, but I ultimately take a different perspective:
I think solving the relevant safety problems for any AI systems that could disrupt the world at this scale will be extremely difficult, and that most surviving worlds are ones in which AI progress has been ~halted via a long-term globally coordinated pause. That means the only futures I should expect to be alive to experience are ones that are fairly “normal” in many respects, which means normal futures are the ones I should plan for.
How that cashes out for me: I don’t have any major long-term ambitions, since I’m so focused on pausing AI, but my partner for instance is working on a mathematics education PhD and wants to help reform math education in the US. I think there’s a very good chance that that will still be a valuable thing for her to pursue, despite how long-term it is.
Work aiming to make an impact in worlds in which we successfully arrange a pause does make sense, and I do think some people should be working on it. But:
In those worlds we have time, so there’s often not a good reason to take those actions now when the opportunity cost is high and the benefit is discounted by the risk we don’t get there.
On the margin, relative to today, I think many more people should be working on trying to bring about such a world.
So much work people describe to me as being valuable in these worlds isn’t even accounting for the disruption and change that will be downstream from AI we already have.
I think someone pursuing a mathematics education PhD is a perfect example for Jeff’s X/Y/Z tweet (which I agree with).
I think there’s a very good chance that that will still be a valuable thing for her to pursue
But how valuable? (It’s easy to focus on how it’s very likely more than zero value, but that would be missing the point.) Is the marginal expected value really anywhere close to the marginal EV of raising the probability of a pause in the next couple years?
Sadly, I think not.
This can be really hard to admit for emotional reasons, especially if she is passionately focused on this path and reluctant to consider radical changes to unusual paths like working to pause AI.
And of course, since pausing AI is the path you’re pursuing, then even if you admitted it (or took the view—I don’t want to use too much language that presumes I’m right), might this just lead to unnecessery tension in the relationship? “Hey romantic parter, I think that your expected social impact would be much higher if you switched from your current path which you’re very passionate about to working on what I’m working on instead” seems pretty awkward TBH. I’d expect most people in that situation to feel like “Why is my partner trying to discourage me from doing what I’m passionate about?” and “Aren’t you biased to think that what you’re doing is most valuable?” and feel bad like they aren’t being supported. If they aren’t unusually open to pivoting, they probably won’t pivot, and so will end up having the same impact that they would have had if you hadn’t said anything, but will just think that you don’t think what they’re doing is important. And that sucks for both of you. It would be much nicer if it could just be true that continuing with her current path is the most valuable thing she can do.
This really hits on an emotional level, but I ultimately take a different perspective:
I think solving the relevant safety problems for any AI systems that could disrupt the world at this scale will be extremely difficult, and that most surviving worlds are ones in which AI progress has been ~halted via a long-term globally coordinated pause. That means the only futures I should expect to be alive to experience are ones that are fairly “normal” in many respects, which means normal futures are the ones I should plan for.
How that cashes out for me:
I don’t have any major long-term ambitions, since I’m so focused on pausing AI, but my partner for instance is working on a mathematics education PhD and wants to help reform math education in the US. I think there’s a very good chance that that will still be a valuable thing for her to pursue, despite how long-term it is.
Work aiming to make an impact in worlds in which we successfully arrange a pause does make sense, and I do think some people should be working on it. But:
In those worlds we have time, so there’s often not a good reason to take those actions now when the opportunity cost is high and the benefit is discounted by the risk we don’t get there.
On the margin, relative to today, I think many more people should be working on trying to bring about such a world.
So much work people describe to me as being valuable in these worlds isn’t even accounting for the disruption and change that will be downstream from AI we already have.
I feel you—this does hit hard emotionally.
I think someone pursuing a mathematics education PhD is a perfect example for Jeff’s X/Y/Z tweet (which I agree with).
But how valuable? (It’s easy to focus on how it’s very likely more than zero value, but that would be missing the point.) Is the marginal expected value really anywhere close to the marginal EV of raising the probability of a pause in the next couple years?
Sadly, I think not.
This can be really hard to admit for emotional reasons, especially if she is passionately focused on this path and reluctant to consider radical changes to unusual paths like working to pause AI.
And of course, since pausing AI is the path you’re pursuing, then even if you admitted it (or took the view—I don’t want to use too much language that presumes I’m right), might this just lead to unnecessery tension in the relationship? “Hey romantic parter, I think that your expected social impact would be much higher if you switched from your current path which you’re very passionate about to working on what I’m working on instead” seems pretty awkward TBH. I’d expect most people in that situation to feel like “Why is my partner trying to discourage me from doing what I’m passionate about?” and “Aren’t you biased to think that what you’re doing is most valuable?” and feel bad like they aren’t being supported. If they aren’t unusually open to pivoting, they probably won’t pivot, and so will end up having the same impact that they would have had if you hadn’t said anything, but will just think that you don’t think what they’re doing is important. And that sucks for both of you. It would be much nicer if it could just be true that continuing with her current path is the most valuable thing she can do.