It doesn’t matter if he could escape if he wanted to, because he couldn’t want to escape unless he already had done so. Friendliness is stable under self-modification.
Edit: I managed to type that whole thing without quite realizing how perfect Parson is as a lay-accessible model for what an alien intelligence looks like. The perversity of his ingenuity, stemming from the fact that he doesn’t share the prejudices of the people around him, is a major part of what people fail to anticipate in AI.
It doesn’t matter if he could escape if he wanted to, because he couldn’t want to escape unless he already had done so. Friendliness is stable under self-modification.
Edit: I managed to type that whole thing without quite realizing how perfect Parson is as a lay-accessible model for what an alien intelligence looks like. The perversity of his ingenuity, stemming from the fact that he doesn’t share the prejudices of the people around him, is a major part of what people fail to anticipate in AI.