We sometimes see with rationalists and utilitarian EAs do something like the same thing we worry about with AI: unaligned optimization that produces outcomes we don’t like. Unfortunately, because humans disagree on norms/ethics/values, it’s kind of hard to know the difference between “going off the rails” and “correcting a massive oversight or collective moral failing”, especially from the inside.
I’m gonna add an even more pessimistic hypothesis: That the disagreements around values are fundamentally irresolvable because there is no truth at the end of the tunnel.
Or, one man’s “going off the rails” is another man’s “correcting a massive oversight or collective moral failing”, and these perspectives can’t be reconciled.
Not quite the main point here, but, my brain has always failed to really cache “Reason as a memetic immune disorder” as a useful concept. When I look at the Memetic Immune System tag, I’m not quite sure how all the posts are supposed to relate to each other. I’d be interested in someone writing up a tag description that actually uses “memetic immune system” in a sentence and gives some examples of how the posts relate.
In my head this is related to Scott’s Robustness to Relative Scale. Reasoning is great, but if you take a particular agent and scale up the part of it that does explicit reasoning by 100x whilst leaving the rest at the same power level, then it may overpower other parts of the system designed to keep it in-check.
To tell an overly specific concrete story of how this might happen:
“I have some self-deception processes that inspire my reasoning, and I have some ethical conscience part that also inspires my reasoning. My reasoning is now good enough to always beat the ethical conscience arguments if it wants to, so when I’m motivated to do that cognition, I always win. My reasoning got better, and now when my selfishness and my conscience butt heads, my selfishness always wins the argument.”
One of the bigger issues is that when we make moral judgements on others, we probably don’t realize that we are imposing our own values, and that this is all we can do.
This cashes out in 2 pessimistic claims:
Fundamentally unbridgeable divides between values exist, as well as their intensity, and this is poor from the perspective of compromise.
Rationality increases don’t have anything to do with terminal goals changes.
Faith is the defence of reason against the passions.
The memetic immune system is the defence of the passions against reason, for while True Reason is eternally perfect, it is not so when attempted by our fallible minds.
We can be as mistaken in our reasoning as in our passions, and both must work harmoniously together.
I’d rather see a shorter, commoner single word for “memetic immune system”. WWCW? (What Would Chesterton Write? Or C.S. Lewis, or Aquinas in English translation.)
Ignoring base rates? -- in the sense that if you just invented a clever theory why good is actually bad, and bad is actually good, consider that the prior probability of this being true (and you being the first person who noticed that) is smaller than the probability of you making a mistake in the clever argument.
Maybe even tails coming apart—in the sense that arguments that seem rational are in general more likely to be true, however the correlation may disappear at the extremes, and the marginal value of taking one more idea in an already too long unlikely chain seriously may be negative.
Well said! Though it raises a question: how can we tell when such defenses are serving truth vs defending an error?
As for an easier word for “memetic immune system”, Lewis might well have called it Convention, as convention is when we disregard memes outside our normal mileu. Can’t say for Chesterton or Aquinas; I’m fairly familiar with Lewis, but much less so with the others apart from some of their memes like Chesterton’s Fence.
With a couple of minutes’ thought: A memetic immune system is whatever allows or rejects new memes as one encounters them, and kills previously accepted memes. Reason is a piece of it that, among other uses, kills falsehoods. That means it kills protective and comforting lies, approximations to the truth, etc., which might have had their purposes in the past. If, for example, your reason is strong enough to kill “I should behave well because God will punish me if I don’t”, but not strong enough to come up with a good moral framework to replace it, then you may have a problem while that situation persists.
Re reason: The real issue is that whenever we make moral judgements, we probably don’t realize that we are imposing our values on it.
That’s why from the perspective of a fanatic like Ziz doesn’t believe they’re irrational, primarily because the Zizians and LW have fundamentally unbridgeably opposed values.
I don’t think “taking ideas too seriously” is what went wrong here. Their actions are just too insane and frankly random and nonsensical to fit that model.
“Taking ideas too seriously” is unlikely to be what went wrong with Ziz, but I do think that it’s a large part of what went wrong with all the misguided individuals who’ve allowed themselves to be drawn into Ziz’s orbit, have adopted Ziz’s bizarre perspectives, etc.
My guess is that “taking ideas seriously” played a role in the chain somewhere, but some of the ideas are about how to think-internally, or the second-order effects of having taken something seriously.
On the one hand.
And on the other hand.
Actually, this makes me think of something.
We sometimes see with rationalists and utilitarian EAs do something like the same thing we worry about with AI: unaligned optimization that produces outcomes we don’t like. Unfortunately, because humans disagree on norms/ethics/values, it’s kind of hard to know the difference between “going off the rails” and “correcting a massive oversight or collective moral failing”, especially from the inside.
I’m gonna add an even more pessimistic hypothesis: That the disagreements around values are fundamentally irresolvable because there is no truth at the end of the tunnel.
Or, one man’s “going off the rails” is another man’s “correcting a massive oversight or collective moral failing”, and these perspectives can’t be reconciled.
Also related: Scott Alexander on epistemic learned helplessness: https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/
Not quite the main point here, but, my brain has always failed to really cache “Reason as a memetic immune disorder” as a useful concept. When I look at the Memetic Immune System tag, I’m not quite sure how all the posts are supposed to relate to each other. I’d be interested in someone writing up a tag description that actually uses “memetic immune system” in a sentence and gives some examples of how the posts relate.
In my head this is related to Scott’s Robustness to Relative Scale. Reasoning is great, but if you take a particular agent and scale up the part of it that does explicit reasoning by 100x whilst leaving the rest at the same power level, then it may overpower other parts of the system designed to keep it in-check.
To tell an overly specific concrete story of how this might happen:
One of the bigger issues is that when we make moral judgements on others, we probably don’t realize that we are imposing our own values, and that this is all we can do.
This cashes out in 2 pessimistic claims:
Fundamentally unbridgeable divides between values exist, as well as their intensity, and this is poor from the perspective of compromise.
Rationality increases don’t have anything to do with terminal goals changes.
Faith is the defence of reason against the passions.
The memetic immune system is the defence of the passions against reason, for while True Reason is eternally perfect, it is not so when attempted by our fallible minds.
We can be as mistaken in our reasoning as in our passions, and both must work harmoniously together.
I’d rather see a shorter, commoner single word for “memetic immune system”. WWCW? (What Would Chesterton Write? Or C.S. Lewis, or Aquinas in English translation.)
Adding Up to Normality?
Common sense?
Ignoring base rates? -- in the sense that if you just invented a clever theory why good is actually bad, and bad is actually good, consider that the prior probability of this being true (and you being the first person who noticed that) is smaller than the probability of you making a mistake in the clever argument.
Maybe even tails coming apart—in the sense that arguments that seem rational are in general more likely to be true, however the correlation may disappear at the extremes, and the marginal value of taking one more idea in an already too long unlikely chain seriously may be negative.
Well said! Though it raises a question: how can we tell when such defenses are serving truth vs defending an error?
As for an easier word for “memetic immune system”, Lewis might well have called it Convention, as convention is when we disregard memes outside our normal mileu. Can’t say for Chesterton or Aquinas; I’m fairly familiar with Lewis, but much less so with the others apart from some of their memes like Chesterton’s Fence.
With a couple of minutes’ thought: A memetic immune system is whatever allows or rejects new memes as one encounters them, and kills previously accepted memes. Reason is a piece of it that, among other uses, kills falsehoods. That means it kills protective and comforting lies, approximations to the truth, etc., which might have had their purposes in the past. If, for example, your reason is strong enough to kill “I should behave well because God will punish me if I don’t”, but not strong enough to come up with a good moral framework to replace it, then you may have a problem while that situation persists.
Re reason: The real issue is that whenever we make moral judgements, we probably don’t realize that we are imposing our values on it.
That’s why from the perspective of a fanatic like Ziz doesn’t believe they’re irrational, primarily because the Zizians and LW have fundamentally unbridgeably opposed values.
I don’t think “taking ideas too seriously” is what went wrong here. Their actions are just too insane and frankly random and nonsensical to fit that model.
“Taking ideas too seriously” is unlikely to be what went wrong with Ziz, but I do think that it’s a large part of what went wrong with all the misguided individuals who’ve allowed themselves to be drawn into Ziz’s orbit, have adopted Ziz’s bizarre perspectives, etc.
My guess is that “taking ideas seriously” played a role in the chain somewhere, but some of the ideas are about how to think-internally, or the second-order effects of having taken something seriously.