I understand the low impact idea, and it’s a great heuristic, but that’s not quite what I am getting at. The impact may be high, but the space of acceptable outcomes should be broad enough so there is no temptation for the AGI to hide and deceive. A tool becoming an agent and destroying the world because it strives to perform the requested operation is more of a ” keep it low-impact” domain, but to avoid tension with the optimization goal, the binding optimizations constraints should not be tight, which is what slack is. I guess it hints as the issues raised in https://en.wikipedia.org/wiki/Human_Compatible, just not the approach advocated there, “the AI’s true objective remain uncertain, with the AI only approaching certainty about it as it gains more information about humans and the world”.
I understand the low impact idea, and it’s a great heuristic, but that’s not quite what I am getting at. The impact may be high, but the space of acceptable outcomes should be broad enough so there is no temptation for the AGI to hide and deceive. A tool becoming an agent and destroying the world because it strives to perform the requested operation is more of a ” keep it low-impact” domain, but to avoid tension with the optimization goal, the binding optimizations constraints should not be tight, which is what slack is. I guess it hints as the issues raised in https://en.wikipedia.org/wiki/Human_Compatible, just not the approach advocated there, “the AI’s true objective remain uncertain, with the AI only approaching certainty about it as it gains more information about humans and the world”.