In my head this is related to Scott’s Robustness to Relative Scale. Reasoning is great, but if you take a particular agent and scale up the part of it that does explicit reasoning by 100x whilst leaving the rest at the same power level, then it may overpower other parts of the system designed to keep it in-check.
To tell an overly specific concrete story of how this might happen:
“I have some self-deception processes that inspire my reasoning, and I have some ethical conscience part that also inspires my reasoning. My reasoning is now good enough to always beat the ethical conscience arguments if it wants to, so when I’m motivated to do that cognition, I always win. My reasoning got better, and now when my selfishness and my conscience butt heads, my selfishness always wins the argument.”
One of the bigger issues is that when we make moral judgements on others, we probably don’t realize that we are imposing our own values, and that this is all we can do.
This cashes out in 2 pessimistic claims:
Fundamentally unbridgeable divides between values exist, as well as their intensity, and this is poor from the perspective of compromise.
Rationality increases don’t have anything to do with terminal goals changes.
In my head this is related to Scott’s Robustness to Relative Scale. Reasoning is great, but if you take a particular agent and scale up the part of it that does explicit reasoning by 100x whilst leaving the rest at the same power level, then it may overpower other parts of the system designed to keep it in-check.
To tell an overly specific concrete story of how this might happen:
One of the bigger issues is that when we make moral judgements on others, we probably don’t realize that we are imposing our own values, and that this is all we can do.
This cashes out in 2 pessimistic claims:
Fundamentally unbridgeable divides between values exist, as well as their intensity, and this is poor from the perspective of compromise.
Rationality increases don’t have anything to do with terminal goals changes.