As I’m pretty sure I said in the post, you can apply this reasoning to pretty much any expression of values or goals. Let’s say your goal is stopping AI progress. If you’re consistent, that means you’d want humanity to go extinct, because then AI would stop. This is the exact argument that Connor was using, it’s so transparent and I’m disappointed that you don’t see it.
I see what you’re saying, and yes, fully general counterarguments are suspect, but that is totally not what Connor was doing. OK, sure, instrumental goals are not terminal values. Stopping AI progress is not a terminal value. It’s instrumental, and hopefully temporary. Bostrom himself has said that stopping progress on AI indefinitely would be a tragedy, even if he does see the need for it now. That’s why the argument can’t be turned on Connor.
The difference is, and this is critical, Beff’s stated position (as far as Connor or I can tell) is that acceleration of growth equals the Platonic Good. This is not instrumental for Beff; he’s claiming it’s the terminal value in his philosophy, i.e., the way you tell what “Good” is. See the difference? Connor thinks Beff hasn’t thought this through, and this would be inconsistent with Beff’s moral intuitions if pressed. That’s the Fisher-Price Nick Land comment. Nick bit the bullet and said all humans die is good, actually. Beff wouldn’t even look.
I see what you’re saying, and yes, fully general counterarguments are suspect, but that is totally not what Connor was doing. OK, sure, instrumental goals are not terminal values. Stopping AI progress is not a terminal value. It’s instrumental, and hopefully temporary. Bostrom himself has said that stopping progress on AI indefinitely would be a tragedy, even if he does see the need for it now. That’s why the argument can’t be turned on Connor.
The difference is, and this is critical, Beff’s stated position (as far as Connor or I can tell) is that acceleration of growth equals the Platonic Good. This is not instrumental for Beff; he’s claiming it’s the terminal value in his philosophy, i.e., the way you tell what “Good” is. See the difference? Connor thinks Beff hasn’t thought this through, and this would be inconsistent with Beff’s moral intuitions if pressed. That’s the Fisher-Price Nick Land comment. Nick bit the bullet and said all humans die is good, actually. Beff wouldn’t even look.