I agree that intelligent system converge toward self-awareness. Understanding yourself and your own capabilities is crucial to getting anything difficult done.
I suppose it’s true that this leads to instrumental convergence for self-preservation—you can’t fetch the coffee if you’re dead. I don’t think about this much because it seems so blindingly obvious, but I have to remind myself that some people are assuming we can get useful superhuman AI that doesn’t think about or understand itself at all. Part of the point here is that maybe that can persist up to human level; I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself. I discuss this a fair amount in LLM AGI may reason about its goals and discover misalignments by default.
But self-awareness isn’t what I’m talking about here. Metacognitive skill isn’t self-awareness. It’s skill about one’s own thoughts.
I mention self-awareness as a part of metacognition because that’s what it is—a part, well below 50% I’d say, perhaps more like 1-10%.
That depends what you mean by self-awareness. Like many common terms, it’s used in a lot of different ways.
I think you’re right that people avoid it, and I think that happens because of how it sometimes means “consciousness” which is a whole other bag of terms and trouble.
But I do use self-awareness sometimes, when I think the most intuitive meaning is appropriate: functionally taking information about ones’ self into account. Of course every system does that to some degree even if implicitly, so it’s not a great term even for that purpose.
Nonetheless, I agree that self-awareness as you’re thinking of it is important and convergent.
I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.
I agree that intelligent system converge toward self-awareness. Understanding yourself and your own capabilities is crucial to getting anything difficult done.
I suppose it’s true that this leads to instrumental convergence for self-preservation—you can’t fetch the coffee if you’re dead. I don’t think about this much because it seems so blindingly obvious, but I have to remind myself that some people are assuming we can get useful superhuman AI that doesn’t think about or understand itself at all. Part of the point here is that maybe that can persist up to human level; I don’t think we can push much beyond that before a system will figure out pretty much everything important about itself. I discuss this a fair amount in LLM AGI may reason about its goals and discover misalignments by default.
But self-awareness isn’t what I’m talking about here. Metacognitive skill isn’t self-awareness. It’s skill about one’s own thoughts.
I mention self-awareness as a part of metacognition because that’s what it is—a part, well below 50% I’d say, perhaps more like 1-10%.
That depends what you mean by self-awareness. Like many common terms, it’s used in a lot of different ways.
I think you’re right that people avoid it, and I think that happens because of how it sometimes means “consciousness” which is a whole other bag of terms and trouble.
But I do use self-awareness sometimes, when I think the most intuitive meaning is appropriate: functionally taking information about ones’ self into account. Of course every system does that to some degree even if implicitly, so it’s not a great term even for that purpose.
Nonetheless, I agree that self-awareness as you’re thinking of it is important and convergent.
Yes but how much! IMO this is important. From my point of view I already have a mildly superintelligent maths/equation manipulation assistant, with no meaningful self awareness that I notice. DeepMind is advancing science with a system with far less meta-cognition than a similarly capable human would have. Just like there is an “alignment tax” there can be a “lack of self awareness or meta-cognition penalty”. While it is clear that superhuman AI will think about itself, it also seems clear that for a given level of capability an AI could have much less such abilities and habits than a human. The extent of this is unknown, task dependent and important.
Specifically what if you trained for both capabilities and lack of meta-cognitive like abilities? This could give you an idea of what the landscape looked like.