The nice things are skills and virtues, parts of designs that might get washed away by stronger optimization. If people or truths or playing chess are not useful/valuable, agents get rid of them, while people might have a different attitude.
(Part of the motivation here is in making sense of corrigibility. Also, I guess simulacrum level 4 is agency, but humans can’t function without a design, so attempts to take advantage of the absence of a design devolve into incoherence.)
The nice things are skills and virtues, parts of designs that might get washed away by stronger optimization. If people or truths or playing chess are not useful/valuable, agents get rid of them, while people might have a different attitude.
(Part of the motivation here is in making sense of corrigibility. Also, I guess simulacrum level 4 is agency, but humans can’t function without a design, so attempts to take advantage of the absence of a design devolve into incoherence.)