The phrase “has to” is a little confusing here. Sure, any AI that doesn’t reliably preserve its value structure under self-modification risks destroying value when it self-modifies. But something can be an AI without preserving its value structure, just like we can be NIs without preserving our value structures.
The phrase “has to” is a little confusing here. Sure, any AI that doesn’t reliably preserve its value structure under self-modification risks destroying value when it self-modifies. But something can be an AI without preserving its value structure, just like we can be NIs without preserving our value structures.