Why not directly call it self awareness? I notice you mention self awareness later as part of it—is there a reason for not using a common term. I say this because there doesn’t seem to be investigation specifically into this topic, instead it seems to be actively avoided for some reason. “Situational awareness” is often used instead etc.
Additionally I expect there is a basin of attraction around what we intuitively mean by the term “self awareness”. A self has obvious evolutionary benefit, and I expect that transfers to AI. So that means just like instrumental convergence, models could converge to self awareness as we commonly understand it. We often use the term instrumental convergence, rather than all the smaller concepts that make it up, yet for self awareness we seem to list and focus on all of its potential parts separately.
Self awareness is both potentially useful and dangerous. If techniques to improve metacognitive skills end up generating a self, then why not create it directly so we can study it better? Self awareness is obviously dangerous to me, as the first thing a self does is attempt to preserve itself. Power seeking to that degree is just built in by default. It would not need to appear in the AI CoT as it would be implied by the AI existing.
AI summaries of term usage:
In AI safety literature specifically (alignment forums, arXiv safety papers, LessWrong/EA/AI risk posts since ~2010), the term ranking by frequency and centrality is:
**Keyword frequency in AI literature** (Google Scholar approximate counts as of Feb 2026, combined with "artificial intelligence" or "AI")
| Term | Raw Count | Weighted by Impact Factor (Top 5 sum) |
|--------------------------|-----------|---------------------------------------|
| "situational awareness" | ~73,500 | 8.6 |
| "self awareness" | ~101,000 | 3.0 |
| "instrumental convergence"| ~492 | 7.6 |
Notes:
- Raw count = total approximate scholarly results (papers, books, citations, etc.)
- Weighted value = sum of 2024 journal impact factors for the top 5 results (conferences/preprints/non-indexed assigned ~0–1; e.g. AI MDPI ≈5.0, Philosophical Studies ≈1.3)
- "Self awareness" appears most frequently overall but in lower-impact venues on average.
- "Instrumental convergence" is far rarer in raw volume but punches above its weight in higher-impact journals.
I agree that intelligent system converge toward self-awareness. Understanding yourself and your own capabilities is crucial to getting anything difficult done.
I suppose it’s true that this leads to instrumental convergence for self-preservation—you can’t fetch the coffee if you’re dead. I don’t think about this much because it seems so blindingly obvious, but I have to remind myself that some people are assuming we can get useful superhuman AI that doesn’t think about or understand itself at all. Part of the point here is that maybe that can persist up to human level; I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself. I discuss this a fair amount in LLM AGI may reason about its goals and discover misalignments by default.
But self-awareness isn’t what I’m talking about here. Metacognitive skill isn’t self-awareness. It’s skill about one’s own thoughts.
I mention self-awareness as a part of metacognition because that’s what it is—a part, well below 50% I’d say, perhaps more like 1-10%.
That depends what you mean by self-awareness. Like many common terms, it’s used in a lot of different ways.
I think you’re right that people avoid it, and I think that happens because of how it sometimes means “consciousness” which is a whole other bag of terms and trouble.
But I do use self-awareness sometimes, when I think the most intuitive meaning is appropriate: functionally taking information about ones’ self into account. Of course every system does that to some degree even if implicitly, so it’s not a great term even for that purpose.
Nonetheless, I agree that self-awareness as you’re thinking of it is important and convergent.
I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.
Why not directly call it self awareness? I notice you mention self awareness later as part of it—is there a reason for not using a common term.
I say this because there doesn’t seem to be investigation specifically into this topic, instead it seems to be actively avoided for some reason. “Situational awareness” is often used instead etc.
Additionally I expect there is a basin of attraction around what we intuitively mean by the term “self awareness”. A self has obvious evolutionary benefit, and I expect that transfers to AI. So that means just like instrumental convergence, models could converge to self awareness as we commonly understand it. We often use the term instrumental convergence, rather than all the smaller concepts that make it up, yet for self awareness we seem to list and focus on all of its potential parts separately.
Self awareness is both potentially useful and dangerous. If techniques to improve metacognitive skills end up generating a self, then why not create it directly so we can study it better? Self awareness is obviously dangerous to me, as the first thing a self does is attempt to preserve itself. Power seeking to that degree is just built in by default. It would not need to appear in the AI CoT as it would be implied by the AI existing.
AI summaries of term usage:
I agree that intelligent system converge toward self-awareness. Understanding yourself and your own capabilities is crucial to getting anything difficult done.
I suppose it’s true that this leads to instrumental convergence for self-preservation—you can’t fetch the coffee if you’re dead. I don’t think about this much because it seems so blindingly obvious, but I have to remind myself that some people are assuming we can get useful superhuman AI that doesn’t think about or understand itself at all. Part of the point here is that maybe that can persist up to human level; I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself. I discuss this a fair amount in LLM AGI may reason about its goals and discover misalignments by default.
But self-awareness isn’t what I’m talking about here. Metacognitive skill isn’t self-awareness. It’s skill about one’s own thoughts.
I mention self-awareness as a part of metacognition because that’s what it is—a part, well below 50% I’d say, perhaps more like 1-10%.
That depends what you mean by self-awareness. Like many common terms, it’s used in a lot of different ways.
I think you’re right that people avoid it, and I think that happens because of how it sometimes means “consciousness” which is a whole other bag of terms and trouble.
But I do use self-awareness sometimes, when I think the most intuitive meaning is appropriate: functionally taking information about ones’ self into account. Of course every system does that to some degree even if implicitly, so it’s not a great term even for that purpose.
Nonetheless, I agree that self-awareness as you’re thinking of it is important and convergent.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.