I appreciate how many sources you’ve cited. Also worth mentioning is Extreme Rationality: It’s Not That Great, by Scott all the way back in 2009. It feels a bit dated, given the references to akrasia (one of Scott’s old obsessions of sorts, before LW recognized it was not a useful way of framing the problems). However, it serves as an explicit prediction of sorts by one of the pillars of this community, who basically did not expect instrumental rationality to result in rationalists “winning more” in the conventional sense. I believe time has mostly proven him right.
I also deeply appreciate Scott’s comment here in response to a 2018 post by Sailor Vulcan. Relevant parts:
“Rationalists should win” was originally a suggestion for how to think about decision theory; if one agent predictably ends up with more utility than another, its choice is more “rational”.
But this got caught up in excitement around “instrumental rationality”—the idea that the “epistemic rationality” skills of figuring out what was true, were only the handmaiden to a much more exciting skill of succeeding in the world. The community redirected itself to figuring out how to succeed in the world, ie became a self-help group.
I understand the logic. If you are good at knowing what is true, then you can be good at knowing what is true about the best thing to do in a certain situation, which means you can be more successful than other people. I can’t deny this makes sense. I can just point out that it doesn’t resemble reality. [...]
[...] People in the community are pushing a thousand different kinds of woo now, in exactly the way “Schools Proliferating Without Evidence” condemned. This is not the fault of their individual irrationality. My guess is that pushing woo is an almost inevitable consequence of taking self-help seriously.
I think my complaint is: once you become a self-help community, you start developing the sorts of epistemic norms that help you be a self-help community, and you start attracting the sort of people who are attracted to self-help communities. And then, if ten years later, someone says “Hey, are we sure we shouldn’t go back to being pure truth-seekers?”, it’s going to be a very different community that discusses the answer to that question.
Thanks a lot! It’s a good comment by Scott on Sailor Vulcan’s post. I have added it and your other links to the page’s “see also” on my site.
I like this paragraph in particular. It captures the tension between the pursuit of epistemic and instrumental rationality:
I think my complaint is: once you become a self-help community, you start developing the sorts of epistemic norms that help you be a self-help community, and you start attracting the sort of people who are attracted to self-help communities. And then, if ten years later, someone says “Hey, are we sure we shouldn’t go back to being pure truth-seekers?”, it’s going to be a very different community that discusses the answer to that question.
I think we have an example of the first part because it has happened with the postrationalists. As a group, postrationalists are influenced by LW but embrace weaker epistemic norms for what they consider practical reasons. A major theme in “a postrationalist syllabus” is superficially irrational beliefs and behaviors that turn out to be effective, which (generalizing) postrationalists try to harness. This exacerbates the problem of schools proliferating without evidence, as reflected in this joke.
I just imagined a possible April Fools article that I am too lazy to actually write, but the idea is that it would announce a new (fictional) feature of the Less Wrong website—you can write posts and comments using two different colors: everything written in black is supposed to be epistemically rational, and everything written in blue is supposed to be instrumentally rational. The new rule is that you should upvote black texts if they are true, and downvote them if they are false, but you should upvote blue texts if they are useful to believe, and downvote them if they are harmful to believe. So it is okay to write something like “Jesus loves you and has a personal plan for you” as long as you write it in blue font. By using both colors, we can achieve the epistemic and instrumental rationality at the same time. (There is a new option in settings for the post-rationalists that sets blue as their default font color.)
I appreciate how many sources you’ve cited. Also worth mentioning is Extreme Rationality: It’s Not That Great, by Scott all the way back in 2009. It feels a bit dated, given the references to akrasia (one of Scott’s old obsessions of sorts, before LW recognized it was not a useful way of framing the problems). However, it serves as an explicit prediction of sorts by one of the pillars of this community, who basically did not expect instrumental rationality to result in rationalists “winning more” in the conventional sense. I believe time has mostly proven him right.
I also deeply appreciate Scott’s comment here in response to a 2018 post by Sailor Vulcan. Relevant parts:
Jacob Falkovich’s classic post on “Is Rationalist Self-Improvement Real?” is also a must-read here, alongside Scott’s excellent response comment.
Thanks a lot! It’s a good comment by Scott on Sailor Vulcan’s post. I have added it and your other links to the page’s “see also” on my site.
I like this paragraph in particular. It captures the tension between the pursuit of epistemic and instrumental rationality:
I think we have an example of the first part because it has happened with the postrationalists. As a group, postrationalists are influenced by LW but embrace weaker epistemic norms for what they consider practical reasons. A major theme in “a postrationalist syllabus” is superficially irrational beliefs and behaviors that turn out to be effective, which (generalizing) postrationalists try to harness. This exacerbates the problem of schools proliferating without evidence, as reflected in this joke.
I just imagined a possible April Fools article that I am too lazy to actually write, but the idea is that it would announce a new (fictional) feature of the Less Wrong website—you can write posts and comments using two different colors: everything written in black is supposed to be epistemically rational, and everything written in blue is supposed to be instrumentally rational. The new rule is that you should upvote black texts if they are true, and downvote them if they are false, but you should upvote blue texts if they are useful to believe, and downvote them if they are harmful to believe. So it is okay to write something like “Jesus loves you and has a personal plan for you” as long as you write it in blue font. By using both colors, we can achieve the epistemic and instrumental rationality at the same time. (There is a new option in settings for the post-rationalists that sets blue as their default font color.)