My technical explanation for why not direct consequentialism is somewhat different—deontology and virtue ethics are effective theories . You are not almost unbounded superintelligence ⇒ you can’t rely on direct consequentialism.
Why virtue ethics works? You are mostly a predictive processing system. Guess at simple PP story: PP is minimizing prediction error. If you take some unvirtuous action, like, e.g. stealing a little, you are basically prompting the pp engine to minimize total error between the action taken, your self-model / wants model, and your world-model. By default, the result will be, your future self will be marginally more delusional (“I assumed they wanted to give it to me anyway”) + bit more ok with stealing (“Everyone else is also stealing.”)
The possible difference here is in my view you are not some sort of ‘somewhat corrupt consequentialist VNM planner’ where if you are strong enough, you will correct the corruption with some smart metacognition. What’s closer to reality is the “you” believing you are somewhat corrupt consequentialist is to a large extent hallucination of a system for which “means change the ends” and “means change what you believe” are equally valid.
If the strength required to wield the tool of reflective consequentialism was immense, like the strength required to wield Thor’s hammer, then I wouldn’t hold this position.
The hammer gets heavier with the stakes of the decisions you are making. Also the strength to hold it mostly does not depend on how smart you are, but on how virtuous you are.
My technical explanation for why not direct consequentialism is somewhat different—deontology and virtue ethics are effective theories . You are not almost unbounded superintelligence ⇒ you can’t rely on direct consequentialism.
Why virtue ethics works? You are mostly a predictive processing system. Guess at simple PP story:
PP is minimizing prediction error. If you take some unvirtuous action, like, e.g. stealing a little, you are basically prompting the pp engine to minimize total error between the action taken, your self-model / wants model, and your world-model. By default, the result will be, your future self will be marginally more delusional (“I assumed they wanted to give it to me anyway”) + bit more ok with stealing (“Everyone else is also stealing.”)
The possible difference here is in my view you are not some sort of ‘somewhat corrupt consequentialist VNM planner’ where if you are strong enough, you will correct the corruption with some smart metacognition. What’s closer to reality is the “you” believing you are somewhat corrupt consequentialist is to a large extent hallucination of a system for which “means change the ends” and “means change what you believe” are equally valid.
The hammer gets heavier with the stakes of the decisions you are making. Also the strength to hold it mostly does not depend on how smart you are, but on how virtuous you are.