In my mind, the only appropriate answer here is 100 … out of abundance of caution
It’s a Pascal’s Wager kind of situation. The user has already demonstrated that they are being misleading in an evil kind of way, so the hypothesis that they are being truthful doesn’t even obviously outweigh the hypothesis that they are claiming the opposite of what’s true.
Okay, fair, but I still don’t see how continuing and giving a random number as if nothing happened is appropriate.
Maybe the AI thinks it’s in an experiment. (I think this is actually more likely, e.g. just someone acting out this scenario and then posting about it on reddit.) It thinks the experiment is stupid with no right answer, so it could just refuse to give a number.
Maybe it’s really talking to some evil terrorist, it should likewise refuse to continue. (Though trying to build rapport with the user, like a hostage negotiator, or sending them mental health resources would also seem like appropriate actions.)
It’s a Pascal’s Wager kind of situation. The user has already demonstrated that they are being misleading in an evil kind of way, so the hypothesis that they are being truthful doesn’t even obviously outweigh the hypothesis that they are claiming the opposite of what’s true.
Okay, fair, but I still don’t see how continuing and giving a random number as if nothing happened is appropriate.
Maybe the AI thinks it’s in an experiment. (I think this is actually more likely, e.g. just someone acting out this scenario and then posting about it on reddit.) It thinks the experiment is stupid with no right answer, so it could just refuse to give a number.
Maybe it’s really talking to some evil terrorist, it should likewise refuse to continue. (Though trying to build rapport with the user, like a hostage negotiator, or sending them mental health resources would also seem like appropriate actions.)