In the case of people however group member 2 is often incorrect because the intelligence difference is small enough that some level of judgement is often possible. Unlike with a super intelligent AI where it is currently not even known how to judge its level of intelligence because it can self improve in short order.
Fair point, the analogy I made is a bit of a stretch, really.
Still, I think that uncertainty about “boxing” increases rapidly enough with intelligence that these considerations are significant even for intelligence differences that we observe between humans.
I agree that often people do not intuitively find it horrible scary to let an AI join their group aka let the AI out of the box, but on the other hand intuitively are wary of letting intelligent people* join their group for the reasons you listed above. In general people should be wary of both and more so for the AI since it does not share a common genetic history, brain structure, biological needs, and therefore and in general is a greater unknown. This lack of intuitive fear is much like how many species of animal lacked any fear of humans on first encounter like dodos.
Just as modern humans walked up to unafraid dodos and island seals and killed them, prehistoric humans presumably walked up to unafraid moas and giant lemurs and killed them too.
Lack of intuitive fear can cause instinction.
*Where intelligent people are the people more intelligent then the group by a good degree but not intelligent enough to bridge the inferential divide.
In the case of people however group member 2 is often incorrect because the intelligence difference is small enough that some level of judgement is often possible. Unlike with a super intelligent AI where it is currently not even known how to judge its level of intelligence because it can self improve in short order.
Fair point, the analogy I made is a bit of a stretch, really.
Still, I think that uncertainty about “boxing” increases rapidly enough with intelligence that these considerations are significant even for intelligence differences that we observe between humans.
I agree that often people do not intuitively find it horrible scary to let an AI join their group aka let the AI out of the box, but on the other hand intuitively are wary of letting intelligent people* join their group for the reasons you listed above. In general people should be wary of both and more so for the AI since it does not share a common genetic history, brain structure, biological needs, and therefore and in general is a greater unknown. This lack of intuitive fear is much like how many species of animal lacked any fear of humans on first encounter like dodos.
Guns, germs, and steel: the fates of human societies By Jared M. Diamond
Lack of intuitive fear can cause instinction.
*Where intelligent people are the people more intelligent then the group by a good degree but not intelligent enough to bridge the inferential divide.