a large fraction of the AI safety and alignment community endorses humanity being replaced by a sufficiently aligned and advanced AI, and would prefer this to a future where our actual descendants spread over the planets, albeit at a slower pace and with fewer total objective “utils”
I don’t think this is true.
To the extent we, on reflection, have preferences about discovering, solving, colonizing the universe ourselves—solving physics, building technology, etc.—a CEV-aligned superintelligence just refuses to help us with those things! It will stop deaths immediately, and it will probably send probes to distant galaxies immediately (on average, our future lightcone loses approximately one galaxy every year), but then it will probably leave us reflect and do stuff, because this is what’s fun.
“Utils” are not some objective thing. It is what we want. To the extent we do want to do meaningful, cool, and interesting stuff, this is what “utils” end up being mainly about.
A sufficiently aligned and advanced AI likely prevents loss of huge amounts of value and then approximately turns off.
Yeah, I had similar thoughts. And it’s even funnier, the AI will not just refuse to solve these problems, but also stop us from creating other AIs to solve these problems.
I don’t think this is true.
To the extent we, on reflection, have preferences about discovering, solving, colonizing the universe ourselves—solving physics, building technology, etc.—a CEV-aligned superintelligence just refuses to help us with those things! It will stop deaths immediately, and it will probably send probes to distant galaxies immediately (on average, our future lightcone loses approximately one galaxy every year), but then it will probably leave us reflect and do stuff, because this is what’s fun.
“Utils” are not some objective thing. It is what we want. To the extent we do want to do meaningful, cool, and interesting stuff, this is what “utils” end up being mainly about.
A sufficiently aligned and advanced AI likely prevents loss of huge amounts of value and then approximately turns off.
Yeah, I had similar thoughts. And it’s even funnier, the AI will not just refuse to solve these problems, but also stop us from creating other AIs to solve these problems.