Thanks (strong-upvoted), this is a really important objection! If I were to rewrite the grandparent more carefully, I would leave off the second invocation of the “I don’t negotiate …” slogan at the end. I think I do want to go as far as counting evolutionary (including cultural-evolutionary) forces under your “modified to become non-strategic by an agent with the aim [...]” clause—but, sure, okay, if I yell in a canyon and the noise causes a landslide, we don’t want to say I was right to yell because keeping silent would be giving to to rock terrorism.
Importantly, however, in the case of the harrassed food critic, I stand by the “not my fault” response, whereas the landslide would be “my fault”. This idea of “fault” doesn’t apply to the God–Empress or other perfectly spherical generic consequentialists on a frictionless plane in a vacuum; it’s a weird thing that we can only make sense of in scenarios where multiple agents are occupying something like the same “moral reference frame”. (Real-world events have multiple causes; consequentialist agents do counterfactual queries on their models of the world in order to decide what action to output, but “who is ‘to blame’ for this event I assign negative utility” is never a question they need to answer.)
But I think blame-allocation is a really important feaure of what’s actually going on when crazy monkeys like us have these discussions that purport to be about decision theory, but are really about monkey stuff. (It’s not that I started out trying to minimize existential risk and happened to compute that going on a Free Speech for Shared Maps crusade was the optimal action; as you know, what actually happened was … well, perhaps more on this in a forthcoming post, “Motivation and Political Context of my Philosophy of Language Agenda”.) I have to admit it’s plausible that a superintelligent singleton God-Empress programmed with the ideal humane utility function would advise me to self-censor for the greater good. And coming from Her, I wouldn’t hesitate to take that advice (because She would know). But that’s not the situation I’m actually in! In the “Provoking Malicious Supporters” section of the post, Gooen writes, “This closely mirrors legal discussions of negligence, gross neglect, and malice”, but negligence and neglect are blame-allocation concepts, not single-agent decision theory concepts!
In accordance with the theory of universal algorithmic bad faith, we might speculate that some part of my monkey-brain is modeling “posts that imply speakers should be blamed for negative side-effects of their speech” as enemy propaganda from the Blight dressed up in the literary genre of consequentialism, for which my monkey-brain has cached counter-propaganda. The only reason this picture doesn’t spell complete doom for the project of advancing the art of human rationality, is that the genre constraints are actually pretty hard to satisfy and have been set up in a way that extracts real philosophical work out of monkey-brains that have Something to Protect, much as a well-functioning court is set up in a way that extracts Justice, even if (say) the defendant is only trying to save her own neck.
Thanks (strong-upvoted), this is a really important objection! If I were to rewrite the grandparent more carefully, I would leave off the second invocation of the “I don’t negotiate …” slogan at the end. I think I do want to go as far as counting evolutionary (including cultural-evolutionary) forces under your “modified to become non-strategic by an agent with the aim [...]” clause—but, sure, okay, if I yell in a canyon and the noise causes a landslide, we don’t want to say I was right to yell because keeping silent would be giving to to rock terrorism.
Importantly, however, in the case of the harrassed food critic, I stand by the “not my fault” response, whereas the landslide would be “my fault”. This idea of “fault” doesn’t apply to the God–Empress or other perfectly spherical generic consequentialists on a frictionless plane in a vacuum; it’s a weird thing that we can only make sense of in scenarios where multiple agents are occupying something like the same “moral reference frame”. (Real-world events have multiple causes; consequentialist agents do counterfactual queries on their models of the world in order to decide what action to output, but “who is ‘to blame’ for this event I assign negative utility” is never a question they need to answer.)
But I think blame-allocation is a really important feaure of what’s actually going on when crazy monkeys like us have these discussions that purport to be about decision theory, but are really about monkey stuff. (It’s not that I started out trying to minimize existential risk and happened to compute that going on a Free Speech for Shared Maps crusade was the optimal action; as you know, what actually happened was … well, perhaps more on this in a forthcoming post, “Motivation and Political Context of my Philosophy of Language Agenda”.) I have to admit it’s plausible that a superintelligent singleton God-Empress programmed with the ideal humane utility function would advise me to self-censor for the greater good. And coming from Her, I wouldn’t hesitate to take that advice (because She would know). But that’s not the situation I’m actually in! In the “Provoking Malicious Supporters” section of the post, Gooen writes, “This closely mirrors legal discussions of negligence, gross neglect, and malice”, but negligence and neglect are blame-allocation concepts, not single-agent decision theory concepts!
In accordance with the theory of universal algorithmic bad faith, we might speculate that some part of my monkey-brain is modeling “posts that imply speakers should be blamed for negative side-effects of their speech” as enemy propaganda from the Blight dressed up in the literary genre of consequentialism, for which my monkey-brain has cached counter-propaganda. The only reason this picture doesn’t spell complete doom for the project of advancing the art of human rationality, is that the genre constraints are actually pretty hard to satisfy and have been set up in a way that extracts real philosophical work out of monkey-brains that have Something to Protect, much as a well-functioning court is set up in a way that extracts Justice, even if (say) the defendant is only trying to save her own neck.