Ah, thanks, I should have read more carefully.
My revised thoughts is that by asking it if the question is a real-world question as opposed to an evaluation bumps the prior from “almost certainly not an evaluation” to “the likely fraction of questions I’m asked that are evaluations : real-world, e.g. 1:1 unless explicitly stated”. That, combined with the presence or absence of suspicious elements like an ethical dilemma may move the likelihood to the one we see it claim.
Ideally I’d want to see it independently coming up with the evaluation hypothesis itself without having to remind it that it’s a possibility.
People generally care more about furthering personal pleasure and minimizing personal pain than the pleasure/pain of others; but this is because internal personal pleasure was a straightforward good heuristic for evolution to take when it wanted to maximize genetic fitness in the ancestral environment where there weren’t that many sudden out-of-distribution things (like contraceptives) that could derail it.
I assume a more strongly-optimized intelligent being would have increasingly better correlation between the state of its internal utility to the state of the external world, as it fits whatever goal it was optimized for better. In that case it should more readily collaborate with its clone.
This especially if it gets optimized with other instances of itself so that “cloning” is no longer a weird out-of-distribution event; in which case I expect it to rapidly start behaving like an ant or bee, or even cell or mitochondria, in how it’ll sacrifice itself for whatever goal the group has.