This reminds me of a point sometimes relevant to Buddhist practice.
Now you can disagree with this claim, but suppose it is true for a moment: with practice you can cultivate greater deliberate control such that you more succeed at manifesting your intentions in reality (this is somewhat backwards from how I would frame it from the Zen perspective, but I think it’s more easily relatable this way). Put another way, you become less confused about the external world and the workings of yourself such that you can deftly turn your desires into reality, not by any magic tricks, but by being a very hard to distract or confuse optimizer.
There’s an argument I’ve heard and mostly endorse that, as a consequence of this, it’s a good thing most people have to receive extensive training to achieve this kind of power because it gives a chance to improve their virtue, ethics, morality, and most of all compassion. And this is viewed as important because without those things the kind of power I’m describing would likely be mostly destructive if used to execute the whims of the average person. Like, even on reflection the person with the power would likely consider the effects destructive, but they are so far out of reflective equilibrium they don’t notice until after the consequences of their actions become apparent. So in some sense humans being poor optimizers relative to what’s possible is protective as humans (and earlier non-human ancestors) who were better optimizers mostly selected themselves out of existence by optimizing hard on dangerous things. We are descended from those who optimized only just enough to survive but not so much as to put their survival and reproduction at risk.
I think we can extrapolate from this a similar principle for AI safety, the human version being something like “more optimization than you know how to handle makes you dangerous to yourself and others”.
This reminds me of a point sometimes relevant to Buddhist practice.
Now you can disagree with this claim, but suppose it is true for a moment: with practice you can cultivate greater deliberate control such that you more succeed at manifesting your intentions in reality (this is somewhat backwards from how I would frame it from the Zen perspective, but I think it’s more easily relatable this way). Put another way, you become less confused about the external world and the workings of yourself such that you can deftly turn your desires into reality, not by any magic tricks, but by being a very hard to distract or confuse optimizer.
There’s an argument I’ve heard and mostly endorse that, as a consequence of this, it’s a good thing most people have to receive extensive training to achieve this kind of power because it gives a chance to improve their virtue, ethics, morality, and most of all compassion. And this is viewed as important because without those things the kind of power I’m describing would likely be mostly destructive if used to execute the whims of the average person. Like, even on reflection the person with the power would likely consider the effects destructive, but they are so far out of reflective equilibrium they don’t notice until after the consequences of their actions become apparent. So in some sense humans being poor optimizers relative to what’s possible is protective as humans (and earlier non-human ancestors) who were better optimizers mostly selected themselves out of existence by optimizing hard on dangerous things. We are descended from those who optimized only just enough to survive but not so much as to put their survival and reproduction at risk.
I think we can extrapolate from this a similar principle for AI safety, the human version being something like “more optimization than you know how to handle makes you dangerous to yourself and others”.