People who hold this position are arguing for things like “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%”, which is a policy that if acted on consistently would more likely than not cause humanity’s extinction within 100 years (as you would be accepting a minimum of a 1% chance of death each year in exchange for faster AI development).
If your goal is to maximize the expected fraction of currently alive humans who live for over 1000 years, you shouldn’t in fact make ongoingly make gambles that make it more likely than not everyone dies unless it turns out that it’s really hard to achieve this without immense risk. Perhaps that is your view: the only (realistic) way to get risk below ~50% is to delay for over 30 years. But this by no means a consensus perspective among those who are very worried about AI risk.
Separately, I don’t expect that we have many tradeoffs between elimination of human control over the future and the probability of currently alive people living for much longer other than AI, so after we eat that, there aren’t further tradeoffs to make. I think you agree with this, but your wording makes it seem as though you think there are ongoing hard tradeoffs that can’t be avoided.
If your goal is to maximize the expected fraction of currently alive humans who live for over 1000 years, you shouldn’t in fact make ongoingly make gambles that make it more likely than not everyone dies unless it turns out that it’s really hard to achieve this without immense risk. Perhaps that is your view: the only (realistic) way to get risk below ~50% is to delay for over 30 years. But this by no means a consensus perspective among those who are very worried about AI risk.
Separately, I don’t expect that we have many tradeoffs between elimination of human control over the future and the probability of currently alive people living for much longer other than AI, so after we eat that, there aren’t further tradeoffs to make. I think you agree with this, but your wording makes it seem as though you think there are ongoing hard tradeoffs that can’t be avoided.