Because agents with good values can, quote, “steal the strategies” that other agents pursue. This is a term from an essay by Paul Christiano, I think, from 2021.
Earlier. I wrote a 2017 post quoted by Paul in 2019, although I think I originally got the idea from Paul.
Appreciate the writeup, it grapples with the competitive landscape rather than falling into common patterns of risk aversion / anthropomorphism / etc. I’m probably more sympathetic to locusts than most people here, which is why I find worries of “all of value will be destroyed unless we freeze hominid values eternally!” to be unconvincing. There’s an important question about what level of competition to expect or target, and my intuition is that the level of competition to realistically target in the long term is probably way higher than is comfortable to comprehend from a risk analysis (rather than utility maximization) frame.
Earlier. I wrote a 2017 post quoted by Paul in 2019, although I think I originally got the idea from Paul.
Appreciate the writeup, it grapples with the competitive landscape rather than falling into common patterns of risk aversion / anthropomorphism / etc. I’m probably more sympathetic to locusts than most people here, which is why I find worries of “all of value will be destroyed unless we freeze hominid values eternally!” to be unconvincing. There’s an important question about what level of competition to expect or target, and my intuition is that the level of competition to realistically target in the long term is probably way higher than is comfortable to comprehend from a risk analysis (rather than utility maximization) frame.