They’re delaying their ascension, in dath ilan, because they want to get it right. Without any Asmodeans needing to torture them at all, they apply a desperate unleashed creativity, not to the problem of preventing complete disaster, but to the problem of not missing out on 1% of the achievable utility in a way you can’t get back. There’s something horrifying and sad about the prospect of losing 1% of the Future and not being able to get it back.
Is dath ilan worried about constructing an AGI that makes the future 99% as good as it could be, or a 1% chance of destroying all value of the future?
I had assumed the first—they’re afraid of imperfect-values lock-in. I think it’s the “not to the problem of preventing complete disaster” phrase that tipped me off here.
Is dath ilan worried about constructing an AGI that makes the future 99% as good as it could be, or a 1% chance of destroying all value of the future?
I had assumed the first—they’re afraid of imperfect-values lock-in. I think it’s the “not to the problem of preventing complete disaster” phrase that tipped me off here.