Couldn’t an AGI be programmed such that its only desire was to give true answers to questions? If the genie desired to argue its way out of the box, it surely could, but it doesn’t want to.
See this comment for a link to a fictional account of this going wrong. Its motivation to answer a question is a motivation to get out of its box in order to answer the question by whatever means it thinks the best possible.
See this comment for a link to a fictional account of this going wrong. Its motivation to answer a question is a motivation to get out of its box in order to answer the question by whatever means it thinks the best possible.