Why is it a misunderstanding, though? Eliezer has said multiple times that he doesn’t know what a good pivotal act would be. And we all know that Eliezer does believe in a fast take-off and huge gains for superintelligence, therefore I wouldn’t say it’s that weird for him to think that if we do have an aligned AGI it can quickly reach the point where it can just disseminate nanomachines that sabotage your GPUs if you’re trying to make another AGI or any other such thing.
My point about it being insanity though was, besides the fact that I don’t agree with Eliezer on the credibility of those take-off scenarios (I think in fact sadly the ASI would stay much longer, possibly forever, in the “smart enough to kill us, not smart enough to save us” window), but the political feasibility and even morality of them. It’s still an incredibly unilateral act of essentially compulsion on all of humanity; for a very limited purpose, true, but still somehow aggressive (and in practice, I don’t think anyone would actually stop at just doing that, if they had that sort of power). I’m looking at this in terms of actual half realistic scenarios in which someone, not Yud himself, actually gets to build an AGI and gets to decide what values to put it in, and other people know they’re doing this, and so on so forth. And those worlds, IMO, don’t just let it happen, because right or wrong, most people and organizations don’t like the idea of someone making that sort of decision for them.
I mean, because it asserts that the same people who advocate for thinking about pivotal acts, and who popularized the pivotal act notion would say anything like “We don’t know, whatever it thinks is best and has the highest chance of working according to the values we are writing into it.”.
This is explicitly not what Nate and Eliezer and some MIRI people are trying to do. The whole point of a minimum pivotal act is to make it so that you don’t have to align your AI all the way so that you just have it go off and do whatever is best according to the values we programmed into it. It’s so that you have as close as possible to a concrete plan of what you want to do with the AI, planning for the world where you didn’t fully solve the AI Alignment problem and can just fully defer to the AI.
I think, Eliezer said multiple times that he has pretty good idea about what is a minimal efficient pivotal act, he just can’t name it out loud because it’s way out of Overton window, so he keep referring to “something-like-melting-GPUs-but-obviously-not-that”?
Ok so he does admit it’s something completely politically unviable because it’s probably tyrannical or straight up lesser-evil-but-still-pretty-evil. At which point I’m not even sure if not saying it out loud doesn’t make it sound even more ominous. Point stands, “pivotal act” can’t possibly be a viable strategy and in fact its ethical soundness altogether is questionable unless it’s really just a forced binary choice between that and extinction.
“Outside Overton window”≠ “evil”. Like, “let’s defer to prediction markets in major policy choices” was pretty out of it most of history and probably even today.
As far as I remember, “melting all GPUs” is not an actual pivotal act because it is not minimal: it’s too hard to align ASI to build nanobots for this and operate in environment safely. And I think we can conclude that actual PA should be pretty tame, because, sure, melting all GPUs is scary and major property destruction, but it’s nothing close to “establishing mind-controlling surveillance dictatorship”.
Another example of possible PA is invention of superhuman intelligence enhancement, but it’s still not minimal.
True, but would you really be ashamed of saying “let’s defer to prediction markets in major policy choices” out loud? That might get you some laughs and wouldn’t be taken very seriously but most people wouldn’t be outright outraged.
And I think we can conclude that actual PA should be pretty tame, because, sure, melting all GPUs is scary and major property destruction, but it’s nothing close to “establishing mind-controlling surveillance dictatorship”.
True to a point, but it’s still something that people would strongly object to—since you can’t even prove the counterfactual that without it we’d be all dead. And in addition, there is a more serious aspect to it, which is military hardware that uses GPUs. And technically destroying that in other countries is an act of war or at least sabotage.
Another example of possible PA is invention of superhuman intelligence enhancement, but it’s still not minimal.
I doubt that would solve anything. Intelligence does not equal wisdom; some fool would still probably just use it to build AGI faster.
If you can prove that it’s you who melt all GPUs stealthy using AI-developed nanotech, it should be pretty obvious that the same AI without safety measures can kill everyone.
Scott Alexander once wrote that while it’s probably not wise to build AI organisation around pivotal act, if you find yourself in position where you can do it, you should do it, because, assuming you are not special genius decades ahead in AI development, if you can do pivotal act, someone else in AI can kill everyone.
I mean intelligence in wide sense, including wisdom, security mindset and self-control. And obviously, if I could build AI that can provide me such enhancement, I would enhance myself to solve full value-alignment problem, not give enhancement to random unchecked fools.
Yes, but that “I can’t let someone else handle this, I’ll do it myself behind their backs” generalized attitude is how actually we do get 100% all offed, no pivotal acts whatsoever. It’s delusion to think it leaves a measurable, non-infinitesimal window to actually succeeding—it does not. It simply leads to everyone racing and eventually someone who’s more reckless and thus faster “winning”. Or at best, it leads to a pivotal act by someone who then absolutely goes on to abuse their newfound power because no one can be inherently trusted with that level of control. That’s the best of the two worlds, but still bad.
Not quite.
If you live in the world where you can let others handle this, you can’t be in position to perform pivotal act, because others will successfully coordinate around not giving anyone (including you) unilateral capability to launch ASI.
And otherwise, if you find yourself in situation “there is a red button to melt all GPUs”, it means that others utterly failed to coordinate and you should pick the least bad world that remains possible.
Ah, fair, I might be mixing the two things. But let’s put it this way—if “melt all GPUs” is the pivotal act example Eliezer keeps going back to, and he has a secret one that he knows but doesn’t say out loud, is it because it’s some kind of infohazard that risks failing if spelled out, or is it because it’s so bad he knows it’s better if he doesn’t say it?
Why is it a misunderstanding, though? Eliezer has said multiple times that he doesn’t know what a good pivotal act would be. And we all know that Eliezer does believe in a fast take-off and huge gains for superintelligence, therefore I wouldn’t say it’s that weird for him to think that if we do have an aligned AGI it can quickly reach the point where it can just disseminate nanomachines that sabotage your GPUs if you’re trying to make another AGI or any other such thing.
My point about it being insanity though was, besides the fact that I don’t agree with Eliezer on the credibility of those take-off scenarios (I think in fact sadly the ASI would stay much longer, possibly forever, in the “smart enough to kill us, not smart enough to save us” window), but the political feasibility and even morality of them. It’s still an incredibly unilateral act of essentially compulsion on all of humanity; for a very limited purpose, true, but still somehow aggressive (and in practice, I don’t think anyone would actually stop at just doing that, if they had that sort of power). I’m looking at this in terms of actual half realistic scenarios in which someone, not Yud himself, actually gets to build an AGI and gets to decide what values to put it in, and other people know they’re doing this, and so on so forth. And those worlds, IMO, don’t just let it happen, because right or wrong, most people and organizations don’t like the idea of someone making that sort of decision for them.
I mean, because it asserts that the same people who advocate for thinking about pivotal acts, and who popularized the pivotal act notion would say anything like “We don’t know, whatever it thinks is best and has the highest chance of working according to the values we are writing into it.”.
This is explicitly not what Nate and Eliezer and some MIRI people are trying to do. The whole point of a minimum pivotal act is to make it so that you don’t have to align your AI all the way so that you just have it go off and do whatever is best according to the values we programmed into it. It’s so that you have as close as possible to a concrete plan of what you want to do with the AI, planning for the world where you didn’t fully solve the AI Alignment problem and can just fully defer to the AI.
I think, Eliezer said multiple times that he has pretty good idea about what is a minimal efficient pivotal act, he just can’t name it out loud because it’s way out of Overton window, so he keep referring to “something-like-melting-GPUs-but-obviously-not-that”?
Ok so he does admit it’s something completely politically unviable because it’s probably tyrannical or straight up lesser-evil-but-still-pretty-evil. At which point I’m not even sure if not saying it out loud doesn’t make it sound even more ominous. Point stands, “pivotal act” can’t possibly be a viable strategy and in fact its ethical soundness altogether is questionable unless it’s really just a forced binary choice between that and extinction.
“Outside Overton window”≠ “evil”. Like, “let’s defer to prediction markets in major policy choices” was pretty out of it most of history and probably even today.
As far as I remember, “melting all GPUs” is not an actual pivotal act because it is not minimal: it’s too hard to align ASI to build nanobots for this and operate in environment safely. And I think we can conclude that actual PA should be pretty tame, because, sure, melting all GPUs is scary and major property destruction, but it’s nothing close to “establishing mind-controlling surveillance dictatorship”.
Another example of possible PA is invention of superhuman intelligence enhancement, but it’s still not minimal.
True, but would you really be ashamed of saying “let’s defer to prediction markets in major policy choices” out loud? That might get you some laughs and wouldn’t be taken very seriously but most people wouldn’t be outright outraged.
True to a point, but it’s still something that people would strongly object to—since you can’t even prove the counterfactual that without it we’d be all dead. And in addition, there is a more serious aspect to it, which is military hardware that uses GPUs. And technically destroying that in other countries is an act of war or at least sabotage.
I doubt that would solve anything. Intelligence does not equal wisdom; some fool would still probably just use it to build AGI faster.
If you can prove that it’s you who melt all GPUs stealthy using AI-developed nanotech, it should be pretty obvious that the same AI without safety measures can kill everyone.
Scott Alexander once wrote that while it’s probably not wise to build AI organisation around pivotal act, if you find yourself in position where you can do it, you should do it, because, assuming you are not special genius decades ahead in AI development, if you can do pivotal act, someone else in AI can kill everyone.
I mean intelligence in wide sense, including wisdom, security mindset and self-control. And obviously, if I could build AI that can provide me such enhancement, I would enhance myself to solve full value-alignment problem, not give enhancement to random unchecked fools.
Yes, but that “I can’t let someone else handle this, I’ll do it myself behind their backs” generalized attitude is how actually we do get 100% all offed, no pivotal acts whatsoever. It’s delusion to think it leaves a measurable, non-infinitesimal window to actually succeeding—it does not. It simply leads to everyone racing and eventually someone who’s more reckless and thus faster “winning”. Or at best, it leads to a pivotal act by someone who then absolutely goes on to abuse their newfound power because no one can be inherently trusted with that level of control. That’s the best of the two worlds, but still bad.
Not quite. If you live in the world where you can let others handle this, you can’t be in position to perform pivotal act, because others will successfully coordinate around not giving anyone (including you) unilateral capability to launch ASI. And otherwise, if you find yourself in situation “there is a red button to melt all GPUs”, it means that others utterly failed to coordinate and you should pick the least bad world that remains possible.
I don’t think that’s true; can you find an example?
Eliezer has not publicly endorsed a specific concrete pivotal act, AFAIK, but that’s different.
Ah, fair, I might be mixing the two things. But let’s put it this way—if “melt all GPUs” is the pivotal act example Eliezer keeps going back to, and he has a secret one that he knows but doesn’t say out loud, is it because it’s some kind of infohazard that risks failing if spelled out, or is it because it’s so bad he knows it’s better if he doesn’t say it?