From Scott Alexander’s recent post (13: Runway):
Your audience chose to read you for some reason. Maybe you had a catchy title. Maybe someone they liked recommended you. Maybe the algorithm placed your post in front of their eyes while they sat there drooling and immobile. They had some hope that reading you would be mildly more interesting than the alternative. That’s your runway. It will last a few sentences to a few paragraphs before they drift off. Don’t waste your first few paragraphs defining something everyone already knows the definition of, or telling a rambling story about why you decided to write this.
I think what this is saying is that an agent doesn’t need to be able to reflect on its goals and decide that they’re the “right” ones in order to be capable/dangerous. It just has to be the sort of agent that pursues those goals. Stockfish will beat you without believing that it “ought” to play to win.