Perhaps an aside, but it seems worse for an AI to wander into “riskiness” and “incorrigibility” for awhile than it is good for it to be able to wander into “risklessness” and “corrigibility” for awhile. I expect we would be wiped out in the risky period, and it’s not clear enough information would be preserved such that we could be reinstantiated later (and even then, it seems a shame to waste all the period where the Universe is being used for ends we wouldn’t endorse—a sort of ‘periodic astronomical waste’)
Perhaps an aside, but it seems worse for an AI to wander into “riskiness” and “incorrigibility” for awhile than it is good for it to be able to wander into “risklessness” and “corrigibility” for awhile. I expect we would be wiped out in the risky period, and it’s not clear enough information would be preserved such that we could be reinstantiated later (and even then, it seems a shame to waste all the period where the Universe is being used for ends we wouldn’t endorse—a sort of ‘periodic astronomical waste’)
(This might be true, but my original intent was a reductio ad absurdum—I do not actually think AI systems will be “wandering around”.)