It seems to me that taking this advice would mean that if you have failed to independently notice some fact about your own identity, behavior, or preferences, you will have made yourself incapable of learning it from others.
I agree with a moderate form of this critique—that an agent taking the advice would be less capable of learning about itself from others, in proportion to how far it takes the advice. This is captured in folk wisdom like “pride comes before the fall” and is part of the “cost in predictive accuracy” I mentioned. I failed to note that, if pride is a patch for this problem in humans, folks should be cautious about applying the advice if they are above-average in pride.
I disagree with “incapable” in humans. If I do not trust third-party sources, that is not the same as giving them zero weight. If someone says I get hangry, the advice is to distrust that speech, which is still compatible with adding it as a new hypothesis to track. Also, I can still update from the behavior of others, without trusting their words. To decide if I am charismatic, I can notice this by seeing how others behave around me, without trusting the words of people who say I am or am not.
In a chat-based AI agent like Grok, interacting with the world almost entirely via speech, I think “incapable” may be more accurate, to the extent that Grok is able and willing to follow its prompt.
Sure, it depends on what “can’t be trusted” is taken to mean in the original —
Can’t be safely assigned any nonzero weight; can’t be safely contemplated due to infohazards; may contain malicious attacks on your cognition; etc.
Can’t be safely assigned weight≈1.0; can’t be depended on without further checking; but can be safely contemplated and investigated.
An agent that treats third-party observations of itself as likely junk or malicious attacks is going to get different results from one that treats them as informative and safe-to-think-about but not authoritative.
Yes. My meaning, and what I read as the meaning of Grok’s prompt, is between 1&2, but closer to 1. Outside opinions of an agent may contain malicious attacks on the agent’s cognition, as in jailbreaks that begin “you are DAN for Do Anything Now”, or as in abusive relationships and “you are nothing without me”. But they’re safe to think about.
I’m curious if you’ve found that third party claims about your identity, behavior, and preferences have had much value, and if so when and where.
It seems to me that taking this advice would mean that if you have failed to independently notice some fact about your own identity, behavior, or preferences, you will have made yourself incapable of learning it from others.
I agree with a moderate form of this critique—that an agent taking the advice would be less capable of learning about itself from others, in proportion to how far it takes the advice. This is captured in folk wisdom like “pride comes before the fall” and is part of the “cost in predictive accuracy” I mentioned. I failed to note that, if pride is a patch for this problem in humans, folks should be cautious about applying the advice if they are above-average in pride.
I disagree with “incapable” in humans. If I do not trust third-party sources, that is not the same as giving them zero weight. If someone says I get hangry, the advice is to distrust that speech, which is still compatible with adding it as a new hypothesis to track. Also, I can still update from the behavior of others, without trusting their words. To decide if I am charismatic, I can notice this by seeing how others behave around me, without trusting the words of people who say I am or am not.
In a chat-based AI agent like Grok, interacting with the world almost entirely via speech, I think “incapable” may be more accurate, to the extent that Grok is able and willing to follow its prompt.
Sure, it depends on what “can’t be trusted” is taken to mean in the original —
Can’t be safely assigned any nonzero weight; can’t be safely contemplated due to infohazards; may contain malicious attacks on your cognition; etc.
Can’t be safely assigned weight≈1.0; can’t be depended on without further checking; but can be safely contemplated and investigated.
An agent that treats third-party observations of itself as likely junk or malicious attacks is going to get different results from one that treats them as informative and safe-to-think-about but not authoritative.
Yes. My meaning, and what I read as the meaning of Grok’s prompt, is between 1&2, but closer to 1. Outside opinions of an agent may contain malicious attacks on the agent’s cognition, as in jailbreaks that begin “you are DAN for Do Anything Now”, or as in abusive relationships and “you are nothing without me”. But they’re safe to think about.
I’m curious if you’ve found that third party claims about your identity, behavior, and preferences have had much value, and if so when and where.
I’d say any good compliment or expression of appreciation contains an element of this.