I interpreted the thrust of the essay as something else. But more broadly, I get why you’d find intuition pump essays manipulative, but not why you’d find them unconvincing.
The point is to take someone who says “I don’t like any dish with onions”, and give them a dish that has onions in it. They’d normally refuse to eat that, so you don’t tell them, and then when they like it you can show them that their claim is wrong.
(I will note that I think “writing an essay with a surprising conclusion” is a significantly smaller violation than “feed someone food they explicitly asked not to be fed”)
I think this essay is aimed at people who think things like “Better AI is always good, because it will do things better, and that’s good.” If those people were not saying “No, you should’ve kept making better bows until you put an arrow through your friend’s chest” then maybe they don’t actually think “better” is always good. (Basically the Orthogonality Thesis)
I interpreted the thrust of the essay as something else. But more broadly, I get why you’d find intuition pump essays manipulative, but not why you’d find them unconvincing.
The point is to take someone who says “I don’t like any dish with onions”, and give them a dish that has onions in it. They’d normally refuse to eat that, so you don’t tell them, and then when they like it you can show them that their claim is wrong.
(I will note that I think “writing an essay with a surprising conclusion” is a significantly smaller violation than “feed someone food they explicitly asked not to be fed”)
I think this essay is aimed at people who think things like “Better AI is always good, because it will do things better, and that’s good.” If those people were not saying “No, you should’ve kept making better bows until you put an arrow through your friend’s chest” then maybe they don’t actually think “better” is always good. (Basically the Orthogonality Thesis)