Note: please write any answers to this prompt in spoiler-tags.
Recently I set out to deliberate practice at “reasoning about confusing intellectual problems.”
Eliezer’s Class Project has a fictional group of rationality students try to find the true theory of quantum gravity in one month. This always seemed like a cool goal and test for rationality training to aspire to. If you’re not solving difficult open problems faster than science, your Art of Rationality probably isn’t complete.
Of course, our Art of Rationality isn’t complete yet. But, I think there is something promising in this area, as a way to ground out “rationality training” in something concrete. It seems like good practice to take a given physics question you don’t understand the theory behind, and try to invent the theory yourself.
I don’t think we’re anywhere close to the “definitively impressive” version of rationality practice/training. But, I think a good next step is “Solve Thinking Physics™”
Thinking Physics is a textbook teaching physics “question-first” – it presents a physics-y situation, and asks you to figure out what happens next. The questions are multiple choice, but often fairly tricky nonetheless.
I think a good rationalist-training goal is aim for a goal of “be (correctly) 95% confident in the answer”, as a rough proxy for “there were no major lingering confusions about the problem except for generic ‘maybe I missed something?’”. And, failing that, have the subgoal of at least being calibrated about how confused you. Every time you look at an answer, first log your probabilities for each of the multiple-choices in Fatebook.io (or prediction-tracking tool of your choice).
The problems are set up in a way that you can probably reason about them from some basic background knowledge, without much math background. They’re ideal for people who don’t have much physics background (since the whole point of the book is to teach you physics), although I know people with some physics education who still find it fairly hard.
I spent two weeks working on Thinking Physics problems, and hosting meetups/workshops where other people could join me. With each question, I focused on learning as much as I could about how-to-think.
My original hypothesis was that I could get significantly better at it in 6-8 weeks. I only spent two, and the result so far is I think I’m significantly better although didn’t yet hit my goal of 95% accuracy. (In my final test-set, I got 1 out of 5 questions wrong, when I was aiming for zero. I do think I have a pretty clear sense of why I got that 1 question wrong, and what I should have done differently)
After workshopping some ideas for “the Thinking Physics rationality challenge”, I now present you with three tiers of challenge.
Challenge I: Solve three problems (and learn from them)
Step 1: Do an exercise.
Spend some time trying to solve three Thinking Physics question. Aim for 95% accuracy, fully deconfusing yourself about each exercise.
Write down your probabilities for each answer.
It’s important to actually write down the probability for each answer – otherwise, you may get a vague sense of “yeah that’s probably right”, that doesn’t allow me to cleanly say “I got this one wrong.” And doing it for all the answers, not just your favorite one, gives you additional bits about whether your models made any sense. (i.e. having clearly stated “I think answer A is most likely and B is second most likely” gives you a harder update if it turns out that A and B were both wrong)
Step 2: Learn from it
Then, think about how you could have solved the problem better.
Your primary goal is to learn as much as possible from each question.
Babble as many new insights as you can about how to think. This can include explicit “strategies” (like “see if you can simplify the problem”), physiological things (like “I got tired and needed to take a break”), or psychological things (“something about this feels weirdly aversive and ughy, what’s up with that?”).
When you’re done, submit your answer on this post for “what you learned.” (Focus on your takeaways, not the object-level solution).
This is more fun with a partner, although I recommend spending a chunk of time thinking independently before sharing your answers and thought-processes with each other. You might find it helpful to get some friends together as a weekend activity.
I’ve found a fairly good default approach is to do:
20 minutes thinking about it by yourself
20 minutes thinking about it with a friend
20 minutes discussing your meta-reflections on how to solve the problem with a friend.
How to pick exercises
The exercises vary in difficulty. My recommendation is to flip to a random page, weighted towards the beginning of the book. If it feels “difficult but not impossible”, then give it a try.
If you’re pretty confident you just know the answer, still try to come up with a clear explanation for why (but err on the side of checking the answer quickly rather than spending a lot of time doublechecking).
If you end up feeling stuck, try to give it at least 10 minutes before giving up and switching to a different problem. (In most cases, I found it valuable to give it a solid 20 minutes of independent thought + 20 minutes of conversation-with-partner even if I felt really stuck).
Some particular exercises that seemed reasonably good for people I beta-tested this with (which is not to say they were easy or hard, but that I feel like I/others learned from making a good faith effort on:
The Expansion of Nothing
(Page numbers for the exercises vary between editions of the book, but you can look them up in the table of contents)
Put your answers in spoiler tags (begin each line with “>!”), although first list (unspoiler-tagged) that it was a Tier 1 challenge, the name of the exercises you did, and whether you give them each an overall thumbs up or thumbs-down as having been a good exercise.
Challenge II: Design a training regimen
After you’ve done 3 exercises and gotten a rough sense of their shape, develop a training regime that helps you significantly improve at Thinking Physics exercises.
If you started out not being able to reliably solve them at all, get to the point where you can at least semi-reliably solve them, given enough time. (Suggested target: solve 5 random questions in a row without getting any wrong, without help)
If you started out able to semi-reliably get the right answers given a lot of time, aim for speed – can you solve 10 problems in a row, relatively quickly, and only get between 0-1 question wrong?
You can submit your training regime before actually completing it (but flag whether you have actually employed it yet, and if you end up actually doing the training regimen, I suggest replying later with any updates you made).
I think it’s a fine use of this exercise to submit your training regime, then read other people’s suggested regimens to get more ideas before going off to actually do it.
Put your training description in spoiler-tags (although again list which challenge-tier you’re doing in non-spoiler tags)
(Once you actually get started with the training, I recommend adjusting your approach as you learn more)
Challenge III: Fully “Solve” Thinking Physics
After you’ve significantly improved your skill-level, develop a thorough for solving Thinking Physics exercises, in generality. Write the instructions that would have helped past-you get to the point where you could solve them reliably and/or quickly.
(It’s okay for this to include metagaming / psychologizing the author. This is “Solve ‘Thinking Physics’”, not “Solve ‘Physics’”)
Write your answer either as a spoiler-tagged comment here, or as a top-level post if it ends up feeling like a full essay (and then a quick comment here linking to it). Include a note about what concrete outcomes you achieved.
Find different sets of exercises that are as different as possible from Thinking Physics (i.e. requiring a pretty different set of skills, while still being feeling relevant to becoming a “generalist researcher”), that would make for a good followup to this exercise.