How can you tell if someone is an idiot not worth refuting, or if they’re a genius who’s so far ahead of you to sound crazy to you? Could we think an AI had gone mad, and reboot it, when it is really genius.
You can tell by the effect they have on their environment. If it’s stupid, but it works, it’s not stupid. This can be hard to do precisely if you don’t know the entity’s precise goals, but in general if they manage to do interesting things you couldn’t (e.g. making large amounts of money, writing highly useful software, obtaining a cult of followers or converting planets into computronium), they’re probably doing something right.
In the case of you considering taking action against the entity (as in your example of deleting the AI), this is partly self-regulating: A sufficiently intelligent entity should see such an attack coming and have effective countermeasures in place (for instance, by communicating better to you so you don’t conclude it has gone mad). If you attack it and succeed, that by itself places limits on how intelligent the target really was. Note that this part doesn’t work if both sides are unmodified humans, because the relative differences in intelligence aren’t large enough.
In the case of you considering taking action against the entity (as in your example of deleting the AI), this is partly self-regulating: A sufficiently intelligent entity should see such an attack coming and have effective countermeasures in place (for instance, by communicating better to you so you don’t conclude it has gone mad). If you attack it and succeed, that by itself places limits on how intelligent the target really was. Note that this part doesn’t work if both sides are unmodified humans, because the relative differences in intelligence aren’t large enough.