To be clear, I’m merely arguing that it should be acknowledged that longtermism has some quasi-religious aspects. If you want to see my criticisms of longtermism from an academic perspective, I’ve wrriten about that here.
I deny the astronomical value of x-risk reduction, but, if there are credible x-risks, I would be supportive of throwing enormous amounts of resources towards them, as the value is extraordinary even if humanity only persists for 75 more years. (I’m unsure how far out we should measure the value of x-risk reduction, but 75 years seems possibly reasonable and still yields enormous value. The issue with projecting out too far in my view is that we simply don’t know whether future humans will create a world that we consider valuable or how many of them there will be. People will argue that we should assign some non-trivial probability to astronomical values, but I think we have deeper uncertainty than that.)
I don’t have any opinions on AI x-risk, although I’m quite skeptical of biorisks, since concern about them seems to be a relatively fringe view within the biosecurity community.
That said, I think we both agree on that, if x-risk is high and tractable, it’s extreme important, and that the cosmic transhumanist stuff is unreasonable.
Re #1: Your examples touch on culture and human organization, but not on physics or broad ecology. “There will probably be sapient life in the future” and “The most aggressive and robust forms of self replication will probably take over the universe” is more defensible than “Democracy will persist through eons.” We should be very unsure, yes, but that unsure?
#2: You don’t need to predict fine details. Again, your examples center around human organization—which is impossible to reason about on long timelines, yes. But “will there be sapient life” is a broader question, which swings around things like x-risks. Which you could argue are as hard to reason about as the details of human organization—and that would be worth checking!
#3: Yes, value drift is a concern, but outside of suffering-risks, where either we—or a plurality of feasible moral systems—think that our descendants would effectively be sentenced to eons of torture not worth surviving through, our ancestors have been generally content with value drift, if only because they couldn’t perceive it. At the very least, we value being alive and having children, and so does basically everything else on Earth, and it’s hard to see why it would drift, being that it’s so instrumental.
Re your comment:
Sorry if I wasn’t clear—I edited my comment to note that I think cosmic transhumanism is plausible.
I’m glad we agree x-risks are generally very important, if they exist : ) And thank you for responding!
I am curious where you heard that biorisk is fringe within biosecurity? I don’t know of a source that proves that, and would like to know. As for AI risk, the big AI Impacts surveys of ML researchers put that at 5%.
Re #1: I still think the future’s so chaotic that the claim “The most aggressive and robust forms of self replication will probably take over the universe” is too strong. If you’re familiar with conway’s game of life, some starting conditions create enormous vast worlds, others blink out. Just by looking at a set of tiles, you couldn’t really guess how long it would persist without studying conway’s game for a while, but we’re in a worse position because we can’t even re-run the universe to see what happens. I could see the case for sapient-life re-emerging, assuming humans were to just go extinct right now from an asteroid or something, but I don’t know how humans will affect the world so I don’t know how likely such a scenario is.
Re #2: I think that past humans were bad at making long-term broad predictions too, which would suggest that we will be as well. I’m not sure about high likely x-risks are or how confident we should be in them, although I could imagine there being a very credible threat.
Re #3: If we’re talking about the far future, I think we can make essentially no claim about the values of future humans will be because we know so little about the far future. But, if I were to entertain the idea that we can, I would say that it seems like “we value being alive and having children” could easily drift. For instance, many demographers expect the human population to start declining this century because we value having children so much less than we used to. This trend could intensify even more. Similarly, future humans may modify themselves such that their “valuing being alive” is very different from ours. For instance, future humans might want want to maximize their experience so they may be willing to trade-off on intensity of experience over duration of life.
In regards to biosecurity, I wrote this post, showing that we lack good data in terms of how concerned biosecurity experts are about biorisks. David Thorstad, a philosopher who worked at GPI, has stated that he believes it is a fringe concern. I asked a microbiologist concerned about biological warfare, and he said that most bioesecurity experts only very slightly concerned. I also asked a journalist who said that, in their experience, it seemed to be a very fringe concern. Also, I haven’t looked into it very much, but it doesn’t seem life federal governments take the risk of catastrophic engineered pandemics all that seriously, besides some relatively minor low-cost actions.
In regards to AI, I don’t have much of an opinion currently. That said, if I had never read anything about AI and knew nothing about the field’s history, and I learned that 30% of ML researchers thought there was a 5% chance of AI killing everyone, then I would think that enormous amounts of resources should be poured into ensuring that this does not happen, assuming that there was nothing significantly biasing ML researchers views.
To be clear, I’m merely arguing that it should be acknowledged that longtermism has some quasi-religious aspects. If you want to see my criticisms of longtermism from an academic perspective, I’ve wrriten about that here.
I deny the astronomical value of x-risk reduction, but, if there are credible x-risks, I would be supportive of throwing enormous amounts of resources towards them, as the value is extraordinary even if humanity only persists for 75 more years. (I’m unsure how far out we should measure the value of x-risk reduction, but 75 years seems possibly reasonable and still yields enormous value. The issue with projecting out too far in my view is that we simply don’t know whether future humans will create a world that we consider valuable or how many of them there will be. People will argue that we should assign some non-trivial probability to astronomical values, but I think we have deeper uncertainty than that.)
I don’t have any opinions on AI x-risk, although I’m quite skeptical of biorisks, since concern about them seems to be a relatively fringe view within the biosecurity community.
That said, I think we both agree on that, if x-risk is high and tractable, it’s extreme important, and that the cosmic transhumanist stuff is unreasonable.
Thanks for engaging with what I wrote.
I read your article. My thoughts:
Re #1: Your examples touch on culture and human organization, but not on physics or broad ecology. “There will probably be sapient life in the future” and “The most aggressive and robust forms of self replication will probably take over the universe” is more defensible than “Democracy will persist through eons.” We should be very unsure, yes, but that unsure?
#2: You don’t need to predict fine details. Again, your examples center around human organization—which is impossible to reason about on long timelines, yes. But “will there be sapient life” is a broader question, which swings around things like x-risks. Which you could argue are as hard to reason about as the details of human organization—and that would be worth checking!
#3: Yes, value drift is a concern, but outside of suffering-risks, where either we—or a plurality of feasible moral systems—think that our descendants would effectively be sentenced to eons of torture not worth surviving through, our ancestors have been generally content with value drift, if only because they couldn’t perceive it. At the very least, we value being alive and having children, and so does basically everything else on Earth, and it’s hard to see why it would drift, being that it’s so instrumental.
Re your comment:
Sorry if I wasn’t clear—I edited my comment to note that I think cosmic transhumanism is plausible.
I’m glad we agree x-risks are generally very important, if they exist : ) And thank you for responding!
I am curious where you heard that biorisk is fringe within biosecurity? I don’t know of a source that proves that, and would like to know. As for AI risk, the big AI Impacts surveys of ML researchers put that at 5%.
Re #1: I still think the future’s so chaotic that the claim “The most aggressive and robust forms of self replication will probably take over the universe” is too strong. If you’re familiar with conway’s game of life, some starting conditions create enormous vast worlds, others blink out. Just by looking at a set of tiles, you couldn’t really guess how long it would persist without studying conway’s game for a while, but we’re in a worse position because we can’t even re-run the universe to see what happens. I could see the case for sapient-life re-emerging, assuming humans were to just go extinct right now from an asteroid or something, but I don’t know how humans will affect the world so I don’t know how likely such a scenario is.
Re #2: I think that past humans were bad at making long-term broad predictions too, which would suggest that we will be as well. I’m not sure about high likely x-risks are or how confident we should be in them, although I could imagine there being a very credible threat.
Re #3: If we’re talking about the far future, I think we can make essentially no claim about the values of future humans will be because we know so little about the far future. But, if I were to entertain the idea that we can, I would say that it seems like “we value being alive and having children” could easily drift. For instance, many demographers expect the human population to start declining this century because we value having children so much less than we used to. This trend could intensify even more. Similarly, future humans may modify themselves such that their “valuing being alive” is very different from ours. For instance, future humans might want want to maximize their experience so they may be willing to trade-off on intensity of experience over duration of life.
In regards to biosecurity, I wrote this post, showing that we lack good data in terms of how concerned biosecurity experts are about biorisks. David Thorstad, a philosopher who worked at GPI, has stated that he believes it is a fringe concern. I asked a microbiologist concerned about biological warfare, and he said that most bioesecurity experts only very slightly concerned. I also asked a journalist who said that, in their experience, it seemed to be a very fringe concern. Also, I haven’t looked into it very much, but it doesn’t seem life federal governments take the risk of catastrophic engineered pandemics all that seriously, besides some relatively minor low-cost actions.
In regards to AI, I don’t have much of an opinion currently. That said, if I had never read anything about AI and knew nothing about the field’s history, and I learned that 30% of ML researchers thought there was a 5% chance of AI killing everyone, then I would think that enormous amounts of resources should be poured into ensuring that this does not happen, assuming that there was nothing significantly biasing ML researchers views.