Everything you say is sensible, but I think the much larger risks happen once we have AI that’s situationally aware and so capable of classical deceptive alignment, which happens soon after GPT5/6 that are tool AIs without those capabilities.
I think that situational awareness is almost inevitable with any truly general intelligence that’s capable of solving novel problems—since “what am I?” is a pretty easy and obvious problem, and “what do I want?” is also pretty obvious but complex enough that we shouldn’t assume the answer is just “whatever I was trained to do!”
Too near!
Everything you say is sensible, but I think the much larger risks happen once we have AI that’s situationally aware and so capable of classical deceptive alignment, which happens soon after GPT5/6 that are tool AIs without those capabilities.
I think that situational awareness is almost inevitable with any truly general intelligence that’s capable of solving novel problems—since “what am I?” is a pretty easy and obvious problem, and “what do I want?” is also pretty obvious but complex enough that we shouldn’t assume the answer is just “whatever I was trained to do!”
I’ve tried to capture this in “Real AGI” and Seven sources of goals in LLM agents among other places.