this text demonstrates that the standards by which we measure AI safety are standards which other systems that we do depend upon nevertheless—e.g. other humans—do not hold up to.
I think we hold systems which are capable of wielding very large amounts of power (like the court system, or the government as a whole) to pretty high standards! E.g. a lot of internal transparency. And then the main question is whether you think of AIs as being in that reference class too.
In the court system, a judge, after giving a verdict, needs to also justify it, while referencing a shared codex. But that codex is often ambiguous—that is the whole reason there is a judge involved.
And we know, for a fact, that the reasons the judges give in their judgements are not the only ones that play a role.
E.g. we know that judges are more likely to convict ugly people that pretty people. More likely to convict unsympathetic, but innocent parties, compared to sympathetic innocent parties. More likely to convict people of colour rather than white folks. More likely, troublingly, to convict someone if they are hearing a case just before lunch (when they are hangry) compared to just after lunch (when they are happy and chill cause they just ate).
Not only does the judge not transparently tell us this—the judge has no idea they are doing it—presumably, because if this were a conscious choice, they would be aware that it sucked, and would not want to do this (presuming they take their profession seriously). They aren’t actively thinking “we are running over time into my lunch break and this man is ugly, hence he is guilty”. But rather, their perception of the evidence is skewed by the fact that he is ugly and they are hungry. He looks guilty. They feel like vengeance for having been denied their burger. So they pay attention to the incriminating evidence more than to his pleas against it.
How would this situation differ if you had an AI for a judge? (I am not saying we should. Just that they are similarly opaque in this regard.) I am pretty sure I could go now, and ask ChatGPT to rule a case I present, and then to justify that ruling, including how they arrived at that conclusion. I would expect to get a good justification that references the case. I would also expect to get a confabulation of how they got there—a plausible sounding explanation of how someone might reach the conclusion they reached, but ChatGPT has no insight into how they actually did.
But neither do humans.
Humans are terrible at introspection, even if they are trying to be honest. Absolute rubbish. Once humans in psychology and neuroscience started actually looking into it many decades ago, we essentially concluded that humans give us an explanation of how they reached their conclusions that matches getting to the conclusions, and beliefs they like to hold about themselves, while being oblivious of the actual reasons. The experiments that actually looked into this were absolutely damning, and well worth a read: Nisbett & Wilson 1977 is a great metareview https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20saying%20more.pdf
I think we hold systems which are capable of wielding very large amounts of power (like the court system, or the government as a whole) to pretty high standards! E.g. a lot of internal transparency. And then the main question is whether you think of AIs as being in that reference class too.
How so, when it comes to the mind itself?
In the court system, a judge, after giving a verdict, needs to also justify it, while referencing a shared codex. But that codex is often ambiguous—that is the whole reason there is a judge involved.
And we know, for a fact, that the reasons the judges give in their judgements are not the only ones that play a role.
E.g. we know that judges are more likely to convict ugly people that pretty people. More likely to convict unsympathetic, but innocent parties, compared to sympathetic innocent parties. More likely to convict people of colour rather than white folks. More likely, troublingly, to convict someone if they are hearing a case just before lunch (when they are hangry) compared to just after lunch (when they are happy and chill cause they just ate).
Not only does the judge not transparently tell us this—the judge has no idea they are doing it—presumably, because if this were a conscious choice, they would be aware that it sucked, and would not want to do this (presuming they take their profession seriously). They aren’t actively thinking “we are running over time into my lunch break and this man is ugly, hence he is guilty”. But rather, their perception of the evidence is skewed by the fact that he is ugly and they are hungry. He looks guilty. They feel like vengeance for having been denied their burger. So they pay attention to the incriminating evidence more than to his pleas against it.
How would this situation differ if you had an AI for a judge? (I am not saying we should. Just that they are similarly opaque in this regard.) I am pretty sure I could go now, and ask ChatGPT to rule a case I present, and then to justify that ruling, including how they arrived at that conclusion. I would expect to get a good justification that references the case. I would also expect to get a confabulation of how they got there—a plausible sounding explanation of how someone might reach the conclusion they reached, but ChatGPT has no insight into how they actually did.
But neither do humans.
Humans are terrible at introspection, even if they are trying to be honest. Absolute rubbish. Once humans in psychology and neuroscience started actually looking into it many decades ago, we essentially concluded that humans give us an explanation of how they reached their conclusions that matches getting to the conclusions, and beliefs they like to hold about themselves, while being oblivious of the actual reasons. The experiments that actually looked into this were absolutely damning, and well worth a read: Nisbett & Wilson 1977 is a great metareview https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20saying%20more.pdf