This is a valid point for humans (who are not utilitarians at all). but when thinking of an ideal, AI-safe ethics, money pumps are a great falw: because the AI will get pumped again, again, and again, and lose utility. Alternately, the AI will patch its system on the fly; and if we don’t know how it does this, it could end up with a crazy policy—its unpredictable.
Maybe; honestly, nobody knows yet. We’re still too far from being able to build an AI for which ethics would be a relevant concept, to be able to say what such an ethics should look like. For all we really know, perhaps if and when we get to that point, it might become apparent that a system of supervisor modules to perform on-the-fly patching to reliably keep things within sensible bounds is a better solution than trying for logically perfect ethics, for any mind operating under physically realistic constraints of data and computing power.
This is a valid point for humans (who are not utilitarians at all). but when thinking of an ideal, AI-safe ethics, money pumps are a great falw: because the AI will get pumped again, again, and again, and lose utility. Alternately, the AI will patch its system on the fly; and if we don’t know how it does this, it could end up with a crazy policy—its unpredictable.
Maybe; honestly, nobody knows yet. We’re still too far from being able to build an AI for which ethics would be a relevant concept, to be able to say what such an ethics should look like. For all we really know, perhaps if and when we get to that point, it might become apparent that a system of supervisor modules to perform on-the-fly patching to reliably keep things within sensible bounds is a better solution than trying for logically perfect ethics, for any mind operating under physically realistic constraints of data and computing power.