The motivation on the human side is interesting too. The law of comparative advantage suggests we (humans) can gain from trading with an AI. Unfortunately, this would give the AI optimisation power (information into and out of the box, etc.), allowing it to ultimately reorganise our atoms. This echoes the part of the historical motivation for mercantilism—trading with foreigners might make us wealthier, but if they become even wealthier, they’ll be more able to conquer us.
The motivation on the human side is interesting too. The law of comparative advantage suggests we (humans) can gain from trading with an AI. Unfortunately, this would give the AI optimisation power (information into and out of the box, etc.), allowing it to ultimately reorganise our atoms. This echoes the part of the historical motivation for mercantilism—trading with foreigners might make us wealthier, but if they become even wealthier, they’ll be more able to conquer us.