(Those were examples of changing the original algorithm, in particular lifting its limitations. I don’t think the particular details are probable, no more than paperclips in particular are the convergent morality of all random AGIs, but starting from an AIXI engine, some kind of powerful intelligent outcome, as opposed to nothing happening, seems possible.)
You’re right, I forgot about the chocolate.
(Those were examples of changing the original algorithm, in particular lifting its limitations. I don’t think the particular details are probable, no more than paperclips in particular are the convergent morality of all random AGIs, but starting from an AIXI engine, some kind of powerful intelligent outcome, as opposed to nothing happening, seems possible.)