(Those were examples of changing the original algorithm, in particular lifting its limitations. I don’t think the particular details are probable, no more than paperclips in particular are the convergent morality of all random AGIs, but starting from an AIXI engine, some kind of powerful intelligent outcome, as opposed to nothing happening, seems possible.)
Or it could decide to give me chocolate and then self-destruct. You’re privileging hypotheses without good reason.
You’re right, I forgot about the chocolate.
(Those were examples of changing the original algorithm, in particular lifting its limitations. I don’t think the particular details are probable, no more than paperclips in particular are the convergent morality of all random AGIs, but starting from an AIXI engine, some kind of powerful intelligent outcome, as opposed to nothing happening, seems possible.)