Re #1: I still think the future’s so chaotic that the claim “The most aggressive and robust forms of self replication will probably take over the universe” is too strong. If you’re familiar with conway’s game of life, some starting conditions create enormous vast worlds, others blink out. Just by looking at a set of tiles, you couldn’t really guess how long it would persist without studying conway’s game for a while, but we’re in a worse position because we can’t even re-run the universe to see what happens. I could see the case for sapient-life re-emerging, assuming humans were to just go extinct right now from an asteroid or something, but I don’t know how humans will affect the world so I don’t know how likely such a scenario is.
Re #2: I think that past humans were bad at making long-term broad predictions too, which would suggest that we will be as well. I’m not sure about high likely x-risks are or how confident we should be in them, although I could imagine there being a very credible threat.
Re #3: If we’re talking about the far future, I think we can make essentially no claim about the values of future humans will be because we know so little about the far future. But, if I were to entertain the idea that we can, I would say that it seems like “we value being alive and having children” could easily drift. For instance, many demographers expect the human population to start declining this century because we value having children so much less than we used to. This trend could intensify even more. Similarly, future humans may modify themselves such that their “valuing being alive” is very different from ours. For instance, future humans might want want to maximize their experience so they may be willing to trade-off on intensity of experience over duration of life.
In regards to biosecurity, I wrote this post, showing that we lack good data in terms of how concerned biosecurity experts are about biorisks. David Thorstad, a philosopher who worked at GPI, has stated that he believes it is a fringe concern. I asked a microbiologist concerned about biological warfare, and he said that most bioesecurity experts only very slightly concerned. I also asked a journalist who said that, in their experience, it seemed to be a very fringe concern. Also, I haven’t looked into it very much, but it doesn’t seem life federal governments take the risk of catastrophic engineered pandemics all that seriously, besides some relatively minor low-cost actions.
In regards to AI, I don’t have much of an opinion currently. That said, if I had never read anything about AI and knew nothing about the field’s history, and I learned that 30% of ML researchers thought there was a 5% chance of AI killing everyone, then I would think that enormous amounts of resources should be poured into ensuring that this does not happen, assuming that there was nothing significantly biasing ML researchers views.
Re #1: I still think the future’s so chaotic that the claim “The most aggressive and robust forms of self replication will probably take over the universe” is too strong. If you’re familiar with conway’s game of life, some starting conditions create enormous vast worlds, others blink out. Just by looking at a set of tiles, you couldn’t really guess how long it would persist without studying conway’s game for a while, but we’re in a worse position because we can’t even re-run the universe to see what happens. I could see the case for sapient-life re-emerging, assuming humans were to just go extinct right now from an asteroid or something, but I don’t know how humans will affect the world so I don’t know how likely such a scenario is.
Re #2: I think that past humans were bad at making long-term broad predictions too, which would suggest that we will be as well. I’m not sure about high likely x-risks are or how confident we should be in them, although I could imagine there being a very credible threat.
Re #3: If we’re talking about the far future, I think we can make essentially no claim about the values of future humans will be because we know so little about the far future. But, if I were to entertain the idea that we can, I would say that it seems like “we value being alive and having children” could easily drift. For instance, many demographers expect the human population to start declining this century because we value having children so much less than we used to. This trend could intensify even more. Similarly, future humans may modify themselves such that their “valuing being alive” is very different from ours. For instance, future humans might want want to maximize their experience so they may be willing to trade-off on intensity of experience over duration of life.
In regards to biosecurity, I wrote this post, showing that we lack good data in terms of how concerned biosecurity experts are about biorisks. David Thorstad, a philosopher who worked at GPI, has stated that he believes it is a fringe concern. I asked a microbiologist concerned about biological warfare, and he said that most bioesecurity experts only very slightly concerned. I also asked a journalist who said that, in their experience, it seemed to be a very fringe concern. Also, I haven’t looked into it very much, but it doesn’t seem life federal governments take the risk of catastrophic engineered pandemics all that seriously, besides some relatively minor low-cost actions.
In regards to AI, I don’t have much of an opinion currently. That said, if I had never read anything about AI and knew nothing about the field’s history, and I learned that 30% of ML researchers thought there was a 5% chance of AI killing everyone, then I would think that enormous amounts of resources should be poured into ensuring that this does not happen, assuming that there was nothing significantly biasing ML researchers views.