I agree it is not the only issue. I think it is a combination of ideas being genuinely dangerous, and no one having the authority to declare: “you are using those ideas wrong”.
Plus the general “contrarian” attitude where the edgier opinions automatically give you higher status. So if someone hypothetically volunteered for the role of calling out wrong implementations of the dangerous ideas, they would probably be perceived as not smart/brave enough to appreciate them.
*
Thinking more about the analogies with religion… when people repeatedly propose something that is (in their perspective) an incorrect application of their ideas, they give it a name, declare it a heresy, and that makes it easier to deflect the problem in future by simply labeling it.
So perhaps the rationalist/EA alternative could be to maintain an online list of “ideas that are frequently attributed to, or associated with, rationalists / effective altruists, but we explicitly disapprove of them; here is a short summary why”. Probably with a shorter title, maybe “frequent bad ideas”. The next step would be to repeatedly tell new members about this list.
This has a potential to backfire, by making those bad ideas more visible. So we would be trading certainty of exposition against a probability of explosion. On the conservative side, we could simply describe the things that have already happened (the ideas that were already followed and it didn’t end well).
Thinking more about the analogies with religion… when people repeatedly propose something that is (in their perspective) an incorrect application of their ideas, they give it a name, declare it a heresy, and that makes it easier to deflect the problem in future by simply labeling it.
Again, you seem to be focused on Catholicism. Catholicism is not the norm. Orthodox Christianity is maybe also kinda like that, but Protestant denominations are not, Islam is not, most of Buddhism is not, Hinduism is not, and don’t even get me started on Shinto and other forms of animism. Most religions don’t have a fixed canon, they have at most a community which might decide to strongly shun (or even straight up violently punish) anyone whose beliefs are so aberrant they might as well not belong to the same religion any more. But the boundary itself isn’t well defined. If one wants to draw inspiration from religions (not that they are always the best example; Islam has given birth to plenty of violent and radical offshoots, for example, and exactly for the reason you bring up no one is quite in a position to call them out as wrong with unique authority), then you have to look at other things, at how they achieve collective cohesion even without central leadership, because central leadership is pretty much only the specific solution Catholics came up with.
I agree it is not the only issue. I think it is a combination of ideas being genuinely dangerous, and no one having the authority to declare: “you are using those ideas wrong”.
Plus the general “contrarian” attitude where the edgier opinions automatically give you higher status. So if someone hypothetically volunteered for the role of calling out wrong implementations of the dangerous ideas, they would probably be perceived as not smart/brave enough to appreciate them.
*
Thinking more about the analogies with religion… when people repeatedly propose something that is (in their perspective) an incorrect application of their ideas, they give it a name, declare it a heresy, and that makes it easier to deflect the problem in future by simply labeling it.
So perhaps the rationalist/EA alternative could be to maintain an online list of “ideas that are frequently attributed to, or associated with, rationalists / effective altruists, but we explicitly disapprove of them; here is a short summary why”. Probably with a shorter title, maybe “frequent bad ideas”. The next step would be to repeatedly tell new members about this list.
This has a potential to backfire, by making those bad ideas more visible. So we would be trading certainty of exposition against a probability of explosion. On the conservative side, we could simply describe the things that have already happened (the ideas that were already followed and it didn’t end well).
Again, you seem to be focused on Catholicism. Catholicism is not the norm. Orthodox Christianity is maybe also kinda like that, but Protestant denominations are not, Islam is not, most of Buddhism is not, Hinduism is not, and don’t even get me started on Shinto and other forms of animism. Most religions don’t have a fixed canon, they have at most a community which might decide to strongly shun (or even straight up violently punish) anyone whose beliefs are so aberrant they might as well not belong to the same religion any more. But the boundary itself isn’t well defined. If one wants to draw inspiration from religions (not that they are always the best example; Islam has given birth to plenty of violent and radical offshoots, for example, and exactly for the reason you bring up no one is quite in a position to call them out as wrong with unique authority), then you have to look at other things, at how they achieve collective cohesion even without central leadership, because central leadership is pretty much only the specific solution Catholics came up with.