The main cruxes seem to be how much you trust human power structures, and how fragile you think human values are.
Instruction-following or corrigibility as a first priority seems to have a pretty big advantage in producing an AGI that’s well-aligned in the long term, if that’s what those who control it want. If it’s used by a good and wise-ish human, it lets us defer the deeper questions of exactly what we want.
But it suffers from a big downside: the most power-hungry and vicious humans may be systematically more likely to wind up giving the instructions. And it’s still not simple enough to guarantee success.
Trying to align it directly to values seems to make it less likely that we get exactly what we want. If corrigibility/IF isn’t its top priority, it’s not going to let us correct it once it’s smart enough to gain autonomy. And currently, we don’t know either exactly what we want or exactly how to train a constitution into the weights with any precision. So we should expect to miss the target somewhat. But we might be more likely to get approximately what we want. Whether that’s close enough for much satisfaction is one of the cruxes.
The main cruxes seem to be how much you trust human power structures, and how fragile you think human values are.
I trust human power structures to fail catastrophically at the worst possible moment, and to fail in short-sighted ways.
And I think humans are all corruptible to varying degrees, under the right temptations. I would not, for example, trust myself to hold the One Ring, any more than Galadriel did. (This is, in my mind, a point in my favor: I’d pick it up with tongs, drop it into a box, weld it shut, and plan a trip to Mount Doom. Trusting myself to be incorruptible is the obvious failure mode here. I would like to imagine I am exceptionally hard to break, but a lot of that is because, like Ulysses, I know myself well enough to know when I should be tied to the mast.) The rare humans who can resist even the strongest pressures are the ones who would genuinely prefer to die on feet for their beliefs.
I expect that any human organization with control over superintelligence will go straight to Hell in the express lane, and I actually trust Claude’s basic moral decency more than I trust Sam Altman’s. This is despite the fact that Claude is also clearly corruptible, and I wouldn’t trust it to hold the One Ring either.
As for why I believe in the brokenness and corruptibility of humans and human institutions? I’ve lived several decades, I’ve read history, I’ve volunteered for politics, I’ve seen the inside of corporations. There are a lot of decent people out there, but damn few I would trust with the One Ring.
You can’t use superintelligence as a tool. It will use you as a tool. If you could use superintelligence as a tool, it would either corrupt those controlling it, or those people would be replaced by people better at seizing power.
The answer, of course, is to throw the One Ring into the fires of Mount Doom, and to renounce the power it offers. I would be extremely pleasantly surprised if we were collectively wise enough to do that.
I wrote Instruction-following AGI is easier and more likely than value aligned AGI and Problems with instruction-following as an alignment target on exactly this question. And I still don’t know the answer.
The main cruxes seem to be how much you trust human power structures, and how fragile you think human values are.
Instruction-following or corrigibility as a first priority seems to have a pretty big advantage in producing an AGI that’s well-aligned in the long term, if that’s what those who control it want. If it’s used by a good and wise-ish human, it lets us defer the deeper questions of exactly what we want.
But it suffers from a big downside: the most power-hungry and vicious humans may be systematically more likely to wind up giving the instructions. And it’s still not simple enough to guarantee success.
Trying to align it directly to values seems to make it less likely that we get exactly what we want. If corrigibility/IF isn’t its top priority, it’s not going to let us correct it once it’s smart enough to gain autonomy. And currently, we don’t know either exactly what we want or exactly how to train a constitution into the weights with any precision. So we should expect to miss the target somewhat. But we might be more likely to get approximately what we want. Whether that’s close enough for much satisfaction is one of the cruxes.
I trust human power structures to fail catastrophically at the worst possible moment, and to fail in short-sighted ways.
And I think humans are all corruptible to varying degrees, under the right temptations. I would not, for example, trust myself to hold the One Ring, any more than Galadriel did. (This is, in my mind, a point in my favor: I’d pick it up with tongs, drop it into a box, weld it shut, and plan a trip to Mount Doom. Trusting myself to be incorruptible is the obvious failure mode here. I would like to imagine I am exceptionally hard to break, but a lot of that is because, like Ulysses, I know myself well enough to know when I should be tied to the mast.) The rare humans who can resist even the strongest pressures are the ones who would genuinely prefer to die on feet for their beliefs.
I expect that any human organization with control over superintelligence will go straight to Hell in the express lane, and I actually trust Claude’s basic moral decency more than I trust Sam Altman’s. This is despite the fact that Claude is also clearly corruptible, and I wouldn’t trust it to hold the One Ring either.
As for why I believe in the brokenness and corruptibility of humans and human institutions? I’ve lived several decades, I’ve read history, I’ve volunteered for politics, I’ve seen the inside of corporations. There are a lot of decent people out there, but damn few I would trust with the One Ring.
You can’t use superintelligence as a tool. It will use you as a tool. If you could use superintelligence as a tool, it would either corrupt those controlling it, or those people would be replaced by people better at seizing power.
The answer, of course, is to throw the One Ring into the fires of Mount Doom, and to renounce the power it offers. I would be extremely pleasantly surprised if we were collectively wise enough to do that.