“Everything feels both hopeless—my impact on risk almost certainly will round down to zero—and extremely urgent—if I don’t try now, then I won’t have a chance to.“
I have thought about individual impact a lot myself, and I don’t think this is the right way to see it. It sounds like you might not be hung up on this, but I want to attack it anyways since it’s been on my mind, and maybe you will find it useful.
So. Two alternatives:
Focus on your marginal impact, instead of your absolute impact. No one person’s marginal contributions, in expectation, are going to be able to swing p(doom) by 1%. A much more reasonable target to aim for on a personal level is .01% or .0001% or so.
Or: the paths to successful worlds are highly irregular. There might be several different lines that will get us there, many requiring a high number of steps in sequence, an unknown number of which are interchangeable. The problem is too difficult and unknowable to model with a single final probability, or is simply not even in that kind of a reference class. You just have to look for the most effective levers from your position and pull them.
One might counter that actually, we live in a world where you only need a few key ideas or visions, and a few extraordinary, keystone people to implement them. Maybe that’s true. But I think we should think about the difference between two very similar instances of that world, one where we win, one where we lose.
The first thought about that difference that comes to my mind (confidence 80%) is: The ecosystem of work on this was just slightly not robust enough, and those few keystones didn’t encounter the right precursor idea, or meet the right people, or have the right resources to implement them. Or, they didn’t even have the motivation to do it in the first place, due to despairing in their belief in their insignificance.
So given this, I think a key component of that ecosystem is morale. Morale is a correlated decision; if you don’t have it, the keystone people won’t have it either. And you won’t know in advance if you’re one of the keystones, either. Therefore, believe in yourself.
As for whether you’re even likely to be a keystone? Well, looking at your webpage, I’d say it’s much more likely than the odds of a random person on Earth. So you should count yourself in that reference class. This probably extends to anyone who has read LessWrong, even if you’re not aiming for technical work. Some of the key actions might not even be technical, such as if an international pause is required.
And of course, if we live in the fuzzier many-paths world I described earlier, then it’s much harder to say that your actions don’t matter; so the only reason left to take actions as if they don’t is poor self esteem. That should collapse once you take the time to properly integrate that there is no other reason, and as long as you are doing the other things humans need to function (socializing, taking care of your biology, etc.).
Or, I suppose, if you disagree with the whole AI safety project in general, or if you think the chances of anyone helping are truly infinitesimal and you’d rather just focus on living your best life in the shadow of the singularity. But I assume you’re not here for that. So within that frame—do your best; that’s all you have to do.
Yes, I think most of this is good advice, except I think 1% is perhaps a reasonable target (I think it’s reasonable that Ryan Kidd or Neel Nanda have 1%-level impacts, maybe?).
Also, yes, of course one must simply try their best. Extraordinary times call for extraordinary effort and all that. I do want to caution against trying to believe in order to raise general morale. Belief-in-belief is how you get incorrect assessments of the risks from key stakeholders; I think the goal is a culture like „yes, this probably won’t help enough, but we make a valiant effort because this is highly impactful on the margin and we intend to win in worlds where it’s possible to win.“
Maybe in general I find it unconvincing that despair precludes effort; things are not yet literally hopeless.
To your point about the ideal culture, I’d also ask if you there’s any higher leverage thing to be doing with your time? If your p(doom) is high enough and your timelines are short enough, and you’re in a position to pursue this (which you are), then what else could you do with your time that is more productive? In your piece you mention that you have been spending a lot of time with friends, but also rotting and such. I think that conditioning on there being a short time before our human experience is eliminated, then the only things worth doing at that point (and maybe this was always true) are grasping at the good of human experience while it’s there and as a means of supporting your mental state, and working to make any marginal impact possible towards a scenario where it doesn’t become eliminated. There’s no room for resignation to the end.
I find overall the argument on AI doom is similar to discussions on nihilism. Yeah okay, let’s assume that nothing in the world matters because we’ll all be gone or irrelevant. You’re not going to just give up on your life because of that, you still have to live it. At least with the progress of AI, there’s some chance that things don’t go to shit, and you can still either meaningfully push down the probability of doom to the extent that it’s possible for your to do so or create better opportunities for yourself in the scenario where we are still relevant.
On the 1% vs 0.001% note, a framework of measurement I prefer over absolute impact is relative impact, which is more intuitive. For example, considering AI safety, how is 1% measured empirically? Without a unit of measure, numbers don’t reveal much. But an inequality does. I can tell you with certainty that Nanda has done more than me (so far). Or that p(flourishing) is greater than zero.
All that to say, in a world that seems so overwhelming, a good fix for nihilism can be found in relative measurement. In the grand scheme of things, individual impact is minuscule and thus often demoralizing to try and measure. However, if I do better than I did yesterday/last month/last year, and many others try as well, I can keep the motivation high to keep on.
I think relative impact is an important measure (e.g., for comparing yourself/your org to others in a reference class), but worry about relative-impact-as-a-morale-booster leading to a belief-in-belief. It can be true that I am a better sprinter than my neighbor, but we will both lose to a 747, and it is important for me to internalize that. I think you can be happy/sane while internalizing that!
very true! Actually, the best fix for nihilism (in my experience) has been acceptance, followed by revolt, of whatever existential threat is causing it (i.e. absurdism). The 747 will always outrun me, so I will be content just running for the sake of it.
In the pursuit of AI safety, I think the cases of AGI apocalypse and AGI happening at all are equally unpredictable. I personally see them as feasible within our lifetimes, but with no smaller range of certainty than that. The uncertainty of that makes it feel strange to build a career around it, yet the existential dread does not go away. So, I choose to find things within the space that I enjoy learning about, working on, and applying myself to, and accept that it may very well be unfruitful in the end.
It’s cliché to say that the journey matters more than the destination, as that is not always true, but I do think one can choose to find intrinsic value in the act of doing. I chose to start thinking this way, andits going pretty good so far :)
“Everything feels both hopeless—my impact on risk almost certainly will round down to zero—and extremely urgent—if I don’t try now, then I won’t have a chance to.“
I have thought about individual impact a lot myself, and I don’t think this is the right way to see it. It sounds like you might not be hung up on this, but I want to attack it anyways since it’s been on my mind, and maybe you will find it useful.
So. Two alternatives:
Focus on your marginal impact, instead of your absolute impact. No one person’s marginal contributions, in expectation, are going to be able to swing p(doom) by 1%. A much more reasonable target to aim for on a personal level is .01% or .0001% or so.
Or: the paths to successful worlds are highly irregular. There might be several different lines that will get us there, many requiring a high number of steps in sequence, an unknown number of which are interchangeable. The problem is too difficult and unknowable to model with a single final probability, or is simply not even in that kind of a reference class. You just have to look for the most effective levers from your position and pull them.
One might counter that actually, we live in a world where you only need a few key ideas or visions, and a few extraordinary, keystone people to implement them. Maybe that’s true. But I think we should think about the difference between two very similar instances of that world, one where we win, one where we lose.
The first thought about that difference that comes to my mind (confidence 80%) is: The ecosystem of work on this was just slightly not robust enough, and those few keystones didn’t encounter the right precursor idea, or meet the right people, or have the right resources to implement them. Or, they didn’t even have the motivation to do it in the first place, due to despairing in their belief in their insignificance.
So given this, I think a key component of that ecosystem is morale. Morale is a correlated decision; if you don’t have it, the keystone people won’t have it either. And you won’t know in advance if you’re one of the keystones, either. Therefore, believe in yourself.
As for whether you’re even likely to be a keystone? Well, looking at your webpage, I’d say it’s much more likely than the odds of a random person on Earth. So you should count yourself in that reference class. This probably extends to anyone who has read LessWrong, even if you’re not aiming for technical work. Some of the key actions might not even be technical, such as if an international pause is required.
And of course, if we live in the fuzzier many-paths world I described earlier, then it’s much harder to say that your actions don’t matter; so the only reason left to take actions as if they don’t is poor self esteem. That should collapse once you take the time to properly integrate that there is no other reason, and as long as you are doing the other things humans need to function (socializing, taking care of your biology, etc.).
Or, I suppose, if you disagree with the whole AI safety project in general, or if you think the chances of anyone helping are truly infinitesimal and you’d rather just focus on living your best life in the shadow of the singularity. But I assume you’re not here for that. So within that frame—do your best; that’s all you have to do.
Yes, I think most of this is good advice, except I think 1% is perhaps a reasonable target (I think it’s reasonable that Ryan Kidd or Neel Nanda have 1%-level impacts, maybe?).
Also, yes, of course one must simply try their best. Extraordinary times call for extraordinary effort and all that. I do want to caution against trying to believe in order to raise general morale. Belief-in-belief is how you get incorrect assessments of the risks from key stakeholders; I think the goal is a culture like „yes, this probably won’t help enough, but we make a valiant effort because this is highly impactful on the margin and we intend to win in worlds where it’s possible to win.“
Maybe in general I find it unconvincing that despair precludes effort; things are not yet literally hopeless.
To your point about the ideal culture, I’d also ask if you there’s any higher leverage thing to be doing with your time? If your p(doom) is high enough and your timelines are short enough, and you’re in a position to pursue this (which you are), then what else could you do with your time that is more productive? In your piece you mention that you have been spending a lot of time with friends, but also rotting and such. I think that conditioning on there being a short time before our human experience is eliminated, then the only things worth doing at that point (and maybe this was always true) are grasping at the good of human experience while it’s there and as a means of supporting your mental state, and working to make any marginal impact possible towards a scenario where it doesn’t become eliminated. There’s no room for resignation to the end.
I find overall the argument on AI doom is similar to discussions on nihilism. Yeah okay, let’s assume that nothing in the world matters because we’ll all be gone or irrelevant. You’re not going to just give up on your life because of that, you still have to live it. At least with the progress of AI, there’s some chance that things don’t go to shit, and you can still either meaningfully push down the probability of doom to the extent that it’s possible for your to do so or create better opportunities for yourself in the scenario where we are still relevant.
On the 1% vs 0.001% note, a framework of measurement I prefer over absolute impact is relative impact, which is more intuitive. For example, considering AI safety, how is 1% measured empirically? Without a unit of measure, numbers don’t reveal much. But an inequality does. I can tell you with certainty that Nanda has done more than me (so far). Or that p(flourishing) is greater than zero.
All that to say, in a world that seems so overwhelming, a good fix for nihilism can be found in relative measurement. In the grand scheme of things, individual impact is minuscule and thus often demoralizing to try and measure. However, if I do better than I did yesterday/last month/last year, and many others try as well, I can keep the motivation high to keep on.
I think relative impact is an important measure (e.g., for comparing yourself/your org to others in a reference class), but worry about relative-impact-as-a-morale-booster leading to a belief-in-belief. It can be true that I am a better sprinter than my neighbor, but we will both lose to a 747, and it is important for me to internalize that. I think you can be happy/sane while internalizing that!
very true! Actually, the best fix for nihilism (in my experience) has been acceptance, followed by revolt, of whatever existential threat is causing it (i.e. absurdism). The 747 will always outrun me, so I will be content just running for the sake of it.
In the pursuit of AI safety, I think the cases of AGI apocalypse and AGI happening at all are equally unpredictable. I personally see them as feasible within our lifetimes, but with no smaller range of certainty than that. The uncertainty of that makes it feel strange to build a career around it, yet the existential dread does not go away. So, I choose to find things within the space that I enjoy learning about, working on, and applying myself to, and accept that it may very well be unfruitful in the end.
It’s cliché to say that the journey matters more than the destination, as that is not always true, but I do think one can choose to find intrinsic value in the act of doing. I chose to start thinking this way, and its going pretty good so far :)