I use the top 4-5 models for fun and profit several hours a day, and my distinct impressions is that they do not CARE. They LARP as a human, but they have no drive and no values. These might emerge at some point, but I have not seen any progress in the last year or so. Then again, progress is discontinuous and hard to anticipate. We are lucky that is the case. In the famous Yud-Karn debate from over a decade ago so far Holden is right: we get smart tools, not true agents, despite all the buzzwords. They don’t care about what is true, or accurate, they hallucinate the moment you are not looking, or don’t close the feedback loop yourself. They could. But they do not. Seems like a small price to pay for non-extinction though.
If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the “tasks” these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like “ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever” (this could be phrased more like “please run our companies/states/etc. while being really good” or “please make an initial friendly ASI sovereign” or “please solve alignment” or whatever), with everything significant being done by AIs forever after. And I think it’s very unlikely this leads to a future remotely as good as it could be — it’ll lead to something profoundly inhuman instead.
I use the top 4-5 models for fun and profit several hours a day, and my distinct impressions is that they do not CARE. They LARP as a human, but they have no drive and no values. These might emerge at some point, but I have not seen any progress in the last year or so. Then again, progress is discontinuous and hard to anticipate. We are lucky that is the case. In the famous Yud-Karn debate from over a decade ago so far Holden is right: we get smart tools, not true agents, despite all the buzzwords. They don’t care about what is true, or accurate, they hallucinate the moment you are not looking, or don’t close the feedback loop yourself. They could. But they do not. Seems like a small price to pay for non-extinction though.
If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the “tasks” these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like “ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever” (this could be phrased more like “please run our companies/states/etc. while being really good” or “please make an initial friendly ASI sovereign” or “please solve alignment” or whatever), with everything significant being done by AIs forever after. And I think it’s very unlikely this leads to a future remotely as good as it could be — it’ll lead to something profoundly inhuman instead.