A LessWrong “rationality workbook” idea

Note: This was originally posted in the discussion area, but motions to move it to the top level were made.
-----

My own desire to improve my rationality coupled with some posts criticizing LessWrong not too long ago led to an idea. For reference, the posts I mean are these:

The comment stream on that last link brought up whether or not LessWrong is fulfilling EY’s vision of a “rationality dojo” of sorts (along with what I assume is the idea that LessWong is supposed to be fulfilling that vision).
Now, I’m not the one to evaluate whether LessWrong is or is not a true “rationality dojo” (not to mention reducing the problem as not to argue over semantics), but I do have an idea for how it could perhaps do so, and an idea I personally find quite exciting.
The idea is simple: that of a LessWrong workbook.
What does that mean? For one, it means an awful lot of effort, primarily at distillation. I quite enjoy reading posts and appreciate the overall prose style used here. A lot of posts are easy reading, contain nice anecdotes, hypothetical real-world scenarios, and the like. Distillation would entail extraction of the “nuggets” that an aspiring rationalist should know. Teach me rationality in bullet points, flash card modules, and other mini-homework-sized methods (which I’ll get to).
I see the above as reducing to some form of “Rationality Overview.” What is rationality? What’s the point? What is the end result? What are the typical tools used to be rational? Finally, perhaps, what does the life of a rationalist look like? Perhaps pages containing some helpful “mantras” or questions to ask one’s self periodically, such as during some period of daily meditation? Inspirational/​helpful prose like the Twelve Virtues, The Simple Truth, and the Litanies of Gendlin and Tarski.
Secondly, I envision a literal “workbook” type of section. I was intrigued by a comment on my first post, in which JenniferRM wrote [in discussing my ongoing de-conversion]:
Unfortunately, I don’t know of any resources to help people traverse the path you’re facing in a series of small safe steps.
Why not? Shouldn’t there be some example of guidelines for evaluating beliefs and attempting to rationally adjust them according to new evidence, though processes, or trustworthy tools? Hence the birth of my workbook idea.
Most specifically, I envision the following:
  • Most interesting to me was the idea of some form of “rationality comb.” An iterative evaluation process. Again, I hardly consider myself the one to design this, but perhaps something like:

    • Take 5 minutes and brainstorm about the beliefs you think affect your actions the most

    • Focus on the first belief, set(1):belief(1)

    • Can you recall how you came to hold this belief?

    • What are some common alternative views to your belief?

    • Do you think you could provide testable justification for your current belief over the above alternatives?

    • If not, can you imagine leaving your belief for one of the alternatives?

    • And so on...

    • Then repeat with set(1):belief(2). When set(1):belief(n) is finished… re-brainstorm for 5min to come up with set(2):belief(1)...belief(n).

  • A series of “homework” problems on Bayesian Probability, perhaps including EY’s tutorial and other helpful material.

  • Brain teasers or similar items to focus on attentiveness to details, weighing evidence, knowing the limits of what you can know given certain information, etc. I think LW has already provided some good examples of neat things like this (even if they would require refinement).

  • Questions that intentionally try to deceive the reader with some form of fallacy or bias

  • Tutorials on how to have rational discussions, rules of engagement, reaching a mutual conclusion, etc.

This is just a stab at what the “meat” of the workbook could contain.
My primary interest is in the first bullet: a step-by-step guide to examining one’s life. I think it needs to be iterative, since not all beliefs in need of rational attention will be apparent on the first pass, and each subsequent step needs to be able to be done in bite sized chunks. Steps should have concrete starting and ending points that seem achievable to promote continuing effort and a sense of accomplishment. I left out quite a few steps… suggestions could be made as to what kind of information should be sought in attempt to confirm or disconfirm one’s belief/​hypothesis (and being cautious if you find yourself looking for expensive evidence in favor vs. cheap evidence against), how to know be wary of hypotheses that explain too much, and so on.
The point is to have a handy reference guide to not only becoming a rationalist, but also to have a concrete guide to applying those skills to your own life. Steps summarized from trusted methods help avoid pitfalls, help us learn from what’s already documented, and (I would suspect) help us avoid subconscious biases or shields we might be tempted to apply to cherished beliefs if left to our own devices/​methods.
By this time, I’m hoping I have the point across. I’ve tried to link heavily to show that I’m not really proposing new material—the content seems to be here. My idea consists of is rearranging this content into a compact “dojo-like” format.
So, what are your thoughts? Is this done elsewhere already (if so, my apologies for wasting your time)? Does this sound helpful? Is it plausible/​feasible? Would one workbook be able to apply to varied personalities/​learning methods? And what are your thoughts (if you like the idea) on what it would contain?