When I question my intuitions about paperclip-loving humans, one thing that makes them less threatening is 1) an intuition that they’re implementing—whether by mere hobbyistic delight or ideological fanatacism or both—a variation of the plasticity of human values 2) a bearish take on their ability to negate that plasticity and ensure that all anyone cares about is paperclippers forever.
Re: 1 when I imagine the paperclip enthusiasts, I imagine social media posts talking about how particular brands or styles of paperclips appeal to them, philosophical justifications for why paperclips should be maximized, different sects of paperclip maximizers who scorn each other as not the real thing, simple appreciation of paperclips and complex feelings associated with it, heroes who are admired for their contributions to the paperclipping project, still caring somewhat about friends and sex and physical comfort and so on. These and similar features seem pretty universal to human aesthetic, political, and religious movements, and they bake in elements of humanity that I care about and would prefer to keep existing. Presumably classical Clippy doesn’t care about any of these things except perhaps instrumentally and is just implementing a sole “maximize paperclips” function. Evolved aliens probably care about at least a few of them, or things that are analogous to feel “intrinsically valuable” to me, even if they also really really care about paperclips.
If Nazis took over the world and implemented their preferred policies and raised everyone who was allowed to survive with Nazi values, that would be very bad (duh.) But if we’re restricting ourselves to 20th century technology in this example, I’m not worried that their vision of the future would last forever, or even the advertised thousand years; my guess is that the great^n-grandchildren (possibly with a very low n) of the Nazi victors would look back and say “yeah, that was really bad” and that future Nazi-descended civilizations would keep varying around the human baseline: most less nice than they could be but nicer than Nazis. Collecting paperclips is way less bad than the Holocaust (duh), but implemented on human hardware I wouldn’t expect it to last forever either.
When I question my intuitions about paperclip-loving humans, one thing that makes them less threatening is 1) an intuition that they’re implementing—whether by mere hobbyistic delight or ideological fanatacism or both—a variation of the plasticity of human values 2) a bearish take on their ability to negate that plasticity and ensure that all anyone cares about is paperclippers forever.
Re: 1 when I imagine the paperclip enthusiasts, I imagine social media posts talking about how particular brands or styles of paperclips appeal to them, philosophical justifications for why paperclips should be maximized, different sects of paperclip maximizers who scorn each other as not the real thing, simple appreciation of paperclips and complex feelings associated with it, heroes who are admired for their contributions to the paperclipping project, still caring somewhat about friends and sex and physical comfort and so on. These and similar features seem pretty universal to human aesthetic, political, and religious movements, and they bake in elements of humanity that I care about and would prefer to keep existing. Presumably classical Clippy doesn’t care about any of these things except perhaps instrumentally and is just implementing a sole “maximize paperclips” function. Evolved aliens probably care about at least a few of them, or things that are analogous to feel “intrinsically valuable” to me, even if they also really really care about paperclips.
If Nazis took over the world and implemented their preferred policies and raised everyone who was allowed to survive with Nazi values, that would be very bad (duh.) But if we’re restricting ourselves to 20th century technology in this example, I’m not worried that their vision of the future would last forever, or even the advertised thousand years; my guess is that the great^n-grandchildren (possibly with a very low n) of the Nazi victors would look back and say “yeah, that was really bad” and that future Nazi-descended civilizations would keep varying around the human baseline: most less nice than they could be but nicer than Nazis. Collecting paperclips is way less bad than the Holocaust (duh), but implemented on human hardware I wouldn’t expect it to last forever either.