What does it matter? We’d ignore whatever AI says just like anosognosics ignore “your arm is paralyzed”.
Then I wonder how anosognosics perceive the offending assertions? They deny them, but can they repeat them back? Write them down? Can they pretend their arm is paralyzed? Can they correctly identify paralysis in other people?
We should find a way to induce anosognosia temporarily.
That there is delicious cake.