Thanks for all your donations and direct work and transparency about these things!
From my perspective it’s unclear whether or not you’re leaving a lot of potential impact on the table, due to a few considerations:
(1) Are your retirement savings invested optimally? The difference between e.g. a 25% annual return and a 50% annual return on your ~$1.5M in retirement savings is $375,000 in the next year. I haven’t had a 401(k) in years, but my understanding is that 401(k)s limit how you’re able to invest the money, so if your retirement savings are in a 401(k) this could be very suboptimal assuming AI stocks are going to keep growing much faster than the rest of the economy, as I expect they probably will for the next couple of years at least. (VanEck Semiconductor ETF is up 135% in the last 12 months.)
(2) What do you think is the difference in cost-effectiveness between the most cost-effective donation opportunities you can find and your current work? SecureBio sounds like valuable altruistic work, but even if it’s a 99th percentile donation opportunity, you may be able to identify a ~99.9th percentile donation opportunity that is 10x as cost-effective, in which case your salary-sacrifice direct work strategy may be suboptimal. Or perhaps there is a different direct work job you could do that would be much higher impact than your work at SecureBio.
It sounds like you put a much higher probability on AI-caused catastrophe in the next two years than I do, though maybe your 10% forecast below just wasn’t made carefully and your true credence is actually lower:
Let’s try and make some similar predictions for 2028:
My odds that the world has changed substantially are up significantly, maybe 55%, primarily due to AI. I’d put about 10% on futures where things go very badly, where I’m not here to write a followup and you’re not here to read one.
10% that you’ll be unable to write a blog post by 2028 (even end of 2028) due to being dead from a global catastrophe seems *way* too high to me. I’d put that at less than 1% by mid-2028. Also, if there really is a 10% chance that you’ll be dead two years from now due to an AI-caused catastrophe, it’s worth flagging that this means that AI represents about 95% of your mortality risk in the next two years (!!!). (99.3% of American men who turn 40 make it two more years to their 42nd birthday according to https://www.ssa.gov/oact/STATS/table4c6.html).
I wonder if your forecast is 10% because you’re much more pessimistic than I am about AI risk in general or because you have more aggressive AI timelines (or both)?
To answer this, what are your forecasts that you’ll be dead as a result of a global catastrophe by mid-2028/2030/2032/2035/2040? Well actually, I’m not sure whether a good forecast for this for 2040 is 10% or 70%, but if the right forecast for the risk of you dying due to an AI-caused catastrophe by mid-2040 is X%, then what’s your forecast for this by mid-2028/2030/2032/2035? My forecasts would be roughly 0.01X/0.05X/0.2X/0.6X, so my number by mid-2028 is ~0.1%-0.7%, i.e. much lower than your 10%.
Also, in those 10% of scenarios where you’re imagining where we’re dead in two years due to an AI-caused catastrophe, what kind of AI-caused catastrophe are you imagining? Is the bulk of the (you and I aren’t here in two years) risk AI takeover risk or engineered pandemic risk or something else in your mind?
(1) Are your retirement savings invested optimally?
It’s mostly a Vanguard 401k, and I do have a lot of choice of funds. But I have it in total-market index funds for a combination of:
I’m skeptical of my ability to price stocks better than the market. Everyone can see the same returns that are pushing you towards investing in AI stocks.
Much of the benefit of the retirement savings is in worlds where AI slows down massively, in which case moving it all into AI stocks would have been quite a bad choice.
I wonder if your forecast is 10% because you’re much more pessimistic than I am about AI risk in general or because you have more aggressive AI timelines (or both)?
Both, I think, though looking at your numbers below I think it’s mostly timelines.
I think change might come very quickly, in a wide variety of ways.
if there really is a 10% chance that you’ll be dead two years from now due to an AI-caused catastrophe, it’s worth flagging that this means that AI represents about 95% of your mortality risk in the next two years
Yup. Worth taking seriously! I think it’s somewhat more than 95% of my risk, since I’m not likely to die of a drug overdose (no drugs at all) and don’t drive very much.
To answer this, what are your forecasts that you’ll be dead as a result of a global catastrophe by mid-2028/2030/2032/2035/2040? … 0.01X/0.05X/0.2X/0.6X
This is very off the cuff, but maybe 0.4X/0.8X/0.9X/0.95X? So 10%, 20%, 22%, 24% (and then 25%). Much more front-loaded than yours. My thinking is that AI is moving very quickly, and a large proportion of the risk is in worlds where we don’t have time to build defenses and general societal resilience.
in those 10% of scenarios where you’re imagining where we’re dead in two years due to an AI-caused catastrophe, what kind of AI-caused catastrophe are you imagining?
I think biorisk is a large proportion of the early risks, but accelerating things that don’t spread infectiously (drones, toxins) or massive societal collapse (no food) is also significant.
(2) What do you think is the difference in cost-effectiveness between the most cost-effective donation opportunities you can find and your current work?
Until recently I thought that SecureBio was close enough to my best donation option that, after taking into account the efficiencies of salary sacrifice, I should not take my full salary. I recently changed this after (a) SB started doing a lot better with fundraising and (b) with SB being a 501(c)(3) there’s a lot of money that can fund it which can’t fund other opportunities.
Or perhaps there is a different direct work job you could do that would be much higher impact than your work at SecureBio.
I think this is pretty unlikely, but feel free to try and make the case ;)
(Copying my comments over from Facebook:)
Thanks for all your donations and direct work and transparency about these things!
From my perspective it’s unclear whether or not you’re leaving a lot of potential impact on the table, due to a few considerations:
(1) Are your retirement savings invested optimally? The difference between e.g. a 25% annual return and a 50% annual return on your ~$1.5M in retirement savings is $375,000 in the next year. I haven’t had a 401(k) in years, but my understanding is that 401(k)s limit how you’re able to invest the money, so if your retirement savings are in a 401(k) this could be very suboptimal assuming AI stocks are going to keep growing much faster than the rest of the economy, as I expect they probably will for the next couple of years at least. (VanEck Semiconductor ETF is up 135% in the last 12 months.)
(2) What do you think is the difference in cost-effectiveness between the most cost-effective donation opportunities you can find and your current work? SecureBio sounds like valuable altruistic work, but even if it’s a 99th percentile donation opportunity, you may be able to identify a ~99.9th percentile donation opportunity that is 10x as cost-effective, in which case your salary-sacrifice direct work strategy may be suboptimal. Or perhaps there is a different direct work job you could do that would be much higher impact than your work at SecureBio.
It sounds like you put a much higher probability on AI-caused catastrophe in the next two years than I do, though maybe your 10% forecast below just wasn’t made carefully and your true credence is actually lower:
10% that you’ll be unable to write a blog post by 2028 (even end of 2028) due to being dead from a global catastrophe seems *way* too high to me. I’d put that at less than 1% by mid-2028. Also, if there really is a 10% chance that you’ll be dead two years from now due to an AI-caused catastrophe, it’s worth flagging that this means that AI represents about 95% of your mortality risk in the next two years (!!!). (99.3% of American men who turn 40 make it two more years to their 42nd birthday according to https://www.ssa.gov/oact/STATS/table4c6.html).
I wonder if your forecast is 10% because you’re much more pessimistic than I am about AI risk in general or because you have more aggressive AI timelines (or both)?
To answer this, what are your forecasts that you’ll be dead as a result of a global catastrophe by mid-2028/2030/2032/2035/2040? Well actually, I’m not sure whether a good forecast for this for 2040 is 10% or 70%, but if the right forecast for the risk of you dying due to an AI-caused catastrophe by mid-2040 is X%, then what’s your forecast for this by mid-2028/2030/2032/2035? My forecasts would be roughly 0.01X/0.05X/0.2X/0.6X, so my number by mid-2028 is ~0.1%-0.7%, i.e. much lower than your 10%.
Also, in those 10% of scenarios where you’re imagining where we’re dead in two years due to an AI-caused catastrophe, what kind of AI-caused catastrophe are you imagining? Is the bulk of the (you and I aren’t here in two years) risk AI takeover risk or engineered pandemic risk or something else in your mind?
It’s mostly a Vanguard 401k, and I do have a lot of choice of funds. But I have it in total-market index funds for a combination of:
I’m skeptical of my ability to price stocks better than the market. Everyone can see the same returns that are pushing you towards investing in AI stocks.
Much of the benefit of the retirement savings is in worlds where AI slows down massively, in which case moving it all into AI stocks would have been quite a bad choice.
Both, I think, though looking at your numbers below I think it’s mostly timelines.
I think change might come very quickly, in a wide variety of ways.
Yup. Worth taking seriously! I think it’s somewhat more than 95% of my risk, since I’m not likely to die of a drug overdose (no drugs at all) and don’t drive very much.
This is very off the cuff, but maybe 0.4X/0.8X/0.9X/0.95X? So 10%, 20%, 22%, 24% (and then 25%). Much more front-loaded than yours. My thinking is that AI is moving very quickly, and a large proportion of the risk is in worlds where we don’t have time to build defenses and general societal resilience.
I think biorisk is a large proportion of the early risks, but accelerating things that don’t spread infectiously (drones, toxins) or massive societal collapse (no food) is also significant.
What do you think the AI would do that would cause the societal collapse?
Until recently I thought that SecureBio was close enough to my best donation option that, after taking into account the efficiencies of salary sacrifice, I should not take my full salary. I recently changed this after (a) SB started doing a lot better with fundraising and (b) with SB being a 501(c)(3) there’s a lot of money that can fund it which can’t fund other opportunities.
I think this is pretty unlikely, but feel free to try and make the case ;)