Does the “who are smarter than us” qualifier imply that the agent is correct in its assessment that whatever action is in both the agent’s and the human’s interests and that the human will fail to grasp that?
If that’s true, and if the agent is in fact aligned with the interests of the human, then it might be a better outcome for the agent to act, for the positive outcome to materialize, and for the agent and human to assess the weird but good thing after the fact.
Or does being maximally reasonable here mean that even though the human isn’t smart enough to grasp the weird but good thing, the human should nevertheless be open-minded enough and trusting enough of the aligned agent’s intentions to let that un-/counterintuitive thing to transpire?
Does the “who are smarter than us” qualifier imply that the agent is correct in its assessment that whatever action is in both the agent’s and the human’s interests and that the human will fail to grasp that?
If that’s true, and if the agent is in fact aligned with the interests of the human, then it might be a better outcome for the agent to act, for the positive outcome to materialize, and for the agent and human to assess the weird but good thing after the fact.
Or does being maximally reasonable here mean that even though the human isn’t smart enough to grasp the weird but good thing, the human should nevertheless be open-minded enough and trusting enough of the aligned agent’s intentions to let that un-/counterintuitive thing to transpire?