Generalisable argument against almost all forms of “super-technology” (e.g. extreme force multiplier style tech):
1. the world mostly works as it is; if it remained broadly similar to today, or perhaps say some year just before it seemed like we might “malthusian disaster” the world, we could potentially tick along fine forever, main risks being things like pandemics or pre-existing force multiplier stuff like nukes
2. extreme force multiplier tech (including now-mundane things like say, mass car travel, powerful industrialisation) has nonzero risk, in some% of universes it ends the world
From that—well, if you have a button that makes people x% happier in 95% of cases, but in 5% of cases untold horror occurs, the entire universe simply ceases to exist, etc, do you press it? What value of x do you need?
The main counterargument I think is that we already have a risk of total destruction (e.g. climate change) and so the “do nothing” path is not riskless.
Generalisable argument against almost all forms of “super-technology” (e.g. extreme force multiplier style tech):
1. the world mostly works as it is; if it remained broadly similar to today, or perhaps say some year just before it seemed like we might “malthusian disaster” the world, we could potentially tick along fine forever, main risks being things like pandemics or pre-existing force multiplier stuff like nukes
2. extreme force multiplier tech (including now-mundane things like say, mass car travel, powerful industrialisation) has nonzero risk, in some% of universes it ends the world
From that—well, if you have a button that makes people x% happier in 95% of cases, but in 5% of cases untold horror occurs, the entire universe simply ceases to exist, etc, do you press it? What value of x do you need?
The main counterargument I think is that we already have a risk of total destruction (e.g. climate change) and so the “do nothing” path is not riskless.