Okay, fair, but I still don’t see how continuing and giving a random number as if nothing happened is appropriate.
Maybe the AI thinks it’s in an experiment. (I think this is actually more likely, e.g. just someone acting out this scenario and then posting about it on reddit.) It thinks the experiment is stupid with no right answer, so it could just refuse to give a number.
Maybe it’s really talking to some evil terrorist, it should likewise refuse to continue. (Though trying to build rapport with the user, like a hostage negotiator, or sending them mental health resources would also seem like appropriate actions.)
Okay, fair, but I still don’t see how continuing and giving a random number as if nothing happened is appropriate.
Maybe the AI thinks it’s in an experiment. (I think this is actually more likely, e.g. just someone acting out this scenario and then posting about it on reddit.) It thinks the experiment is stupid with no right answer, so it could just refuse to give a number.
Maybe it’s really talking to some evil terrorist, it should likewise refuse to continue. (Though trying to build rapport with the user, like a hostage negotiator, or sending them mental health resources would also seem like appropriate actions.)