If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the “tasks” these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like “ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever” (this could be phrased more like “please run our companies/states/etc. while being really good” or “please make an initial friendly ASI sovereign” or “please solve alignment” or whatever), with everything significant being done by AIs forever after. And I think it’s very unlikely this leads to a future remotely as good as it could be — it’ll lead to something profoundly inhuman instead.
If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the “tasks” these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like “ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever” (this could be phrased more like “please run our companies/states/etc. while being really good” or “please make an initial friendly ASI sovereign” or “please solve alignment” or whatever), with everything significant being done by AIs forever after. And I think it’s very unlikely this leads to a future remotely as good as it could be — it’ll lead to something profoundly inhuman instead.