Note: please write any answers to this prompt in spoiler-tags.
Recently I set out to deliberate practice at “reasoning about confusing intellectual problems.”
Eliezer’s Class Project has a fictional group of rationality students try to find the true theory of quantum gravity in one month. This always seemed like a cool goal and test for rationality training to aspire to. If you’re not solving difficult open problems faster than science, your Art of Rationality probably isn’t complete.
Of course, our Art of Rationality isn’t complete yet. But, I think there is something promising in this area, as a way to ground out “rationality training” in something concrete. It seems like good practice to take a given physics question you don’t understand the theory behind, and try to invent the theory yourself.
I don’t think we’re anywhere close to the “definitively impressive” version of rationality practice/training. But, I think a good next step is “Solve Thinking Physics™”
Thinking Physics is a textbook teaching physics “question-first” – it presents a physics-y situation, and asks you to figure out what happens next. The questions are multiple choice, but often fairly tricky nonetheless.
I think a good rationalist-training goal is aim for a goal of “be (correctly) 95% confident in the answer”, as a rough proxy for “there were no major lingering confusions about the problem except for generic ‘maybe I missed something?’”. And, failing that, have the subgoal of at least being calibrated about how confused you. Every time you look at an answer, first log your probabilities for each of the multiple-choices in Fatebook.io (or prediction-tracking tool of your choice).
The problems are set up in a way that you can probably reason about them from some basic background knowledge, without much math background. They’re ideal for people who don’t have much physics background (since the whole point of the book is to teach you physics), although I know people with some physics education who still find it fairly hard.
I spent two weeks working on Thinking Physics problems, and hosting meetups/workshops where other people could join me. With each question, I focused on learning as much as I could about how-to-think.
My original hypothesis was that I could get significantly better at it in 6-8 weeks. I only spent two, and the result so far is I think I’m significantly better although didn’t yet hit my goal of 95% accuracy. (In my final test-set, I got 1 out of 5 questions wrong, when I was aiming for zero. I do think I have a pretty clear sense of why I got that 1 question wrong, and what I should have done differently)
After workshopping some ideas for “the Thinking Physics rationality challenge”, I now present you with three tiers of challenge.
Challenge I: Solve three problems (and learn from them)
Step 1: Do an exercise.
Spend some time trying to solve three Thinking Physics question. Aim for 95% accuracy, fully deconfusing yourself about each exercise.
Write down your probabilities for each answer.
It’s important to actually write down the probability for each answer – otherwise, you may get a vague sense of “yeah that’s probably right”, that doesn’t allow me to cleanly say “I got this one wrong.” And doing it for all the answers, not just your favorite one, gives you additional bits about whether your models made any sense. (i.e. having clearly stated “I think answer A is most likely and B is second most likely” gives you a harder update if it turns out that A and B were both wrong)
Step 2: Learn from it
Then, think about how you could have solved the problem better.
Your primary goal is to learn as much as possible from each question.
Babble as many new insights as you can about how to think. This can include explicit “strategies” (like “see if you can simplify the problem”), physiological things (like “I got tired and needed to take a break”), or psychological things (“something about this feels weirdly aversive and ughy, what’s up with that?”).
When you’re done, submit your answer on this post for “what you learned.” (Focus on your takeaways, not the object-level solution).
Overall structure
This is more fun with a partner, although I recommend spending a chunk of time thinking independently before sharing your answers and thought-processes with each other. You might find it helpful to get some friends together as a weekend activity.
I’ve found a fairly good default approach is to do:
20 minutes thinking about it by yourself
20 minutes thinking about it with a friend
20 minutes discussing your meta-reflections on how to solve the problem with a friend.
How to pick exercises
The exercises vary in difficulty. My recommendation is to flip to a random page, weighted towards the beginning of the book. If it feels “difficult but not impossible”, then give it a try.
If you’re pretty confident you just know the answer, still try to come up with a clear explanation for why (but err on the side of checking the answer quickly rather than spending a lot of time doublechecking).
If you end up feeling stuck, try to give it at least 10 minutes before giving up and switching to a different problem. (In most cases, I found it valuable to give it a solid 20 minutes of independent thought + 20 minutes of conversation-with-partner even if I felt really stuck).
Some particular exercises that seemed reasonably good for people I beta-tested this with (which is not to say they were easy or hard, but that I feel like I/others learned from making a good faith effort on:
Steam Locomotive
Cold Bath
Rare Air
The Expansion of Nothing
Landscape
(Page numbers for the exercises vary between editions of the book, but you can look them up in the table of contents)
Submission guidelines
Put your answers in spoiler tags (begin each line with “>!”), although first list (unspoiler-tagged) that it was a Tier 1 challenge, the name of the exercises you did, and whether you give them each an overall thumbs up or thumbs-down as having been a good exercise.
Challenge II: Design a training regimen
After you’ve done 3 exercises and gotten a rough sense of their shape, develop a training regime that helps you significantly improve at Thinking Physics exercises.
If you started out not being able to reliably solve them at all, get to the point where you can at least semi-reliably solve them, given enough time. (Suggested target: solve 5 random questions in a row without getting any wrong, without help)
If you started out able to semi-reliably get the right answers given a lot of time, aim for speed – can you solve 10 problems in a row, relatively quickly, and only get between 0-1 question wrong?
Submission guidelines
You can submit your training regime before actually completing it (but flag whether you have actually employed it yet, and if you end up actually doing the training regimen, I suggest replying later with any updates you made).
I think it’s a fine use of this exercise to submit your training regime, then read other people’s suggested regimens to get more ideas before going off to actually do it.
Put your training description in spoiler-tags (although again list which challenge-tier you’re doing in non-spoiler tags)
(Once you actually get started with the training, I recommend adjusting your approach as you learn more)
Challenge III: Fully “Solve” Thinking Physics
After you’ve significantly improved your skill-level, develop a thorough for solving Thinking Physics exercises, in generality. Write the instructions that would have helped past-you get to the point where you could solve them reliably and/or quickly.
(It’s okay for this to include metagaming / psychologizing the author. This is “Solve ‘Thinking Physics’”, not “Solve ‘Physics’”)
Write your answer either as a spoiler-tagged comment here, or as a top-level post if it ends up feeling like a full essay (and then a quick comment here linking to it). Include a note about what concrete outcomes you achieved.
Bonus Challenge:
Find different sets of exercises that are as different as possible from Thinking Physics (i.e. requiring a pretty different set of skills, while still being feeling relevant to becoming a “generalist researcher”), that would make for a good followup to this exercise.
Challenge I
Exercises:
steam engine :-1:
cold bath :+1:
expansion of nothing :+1:
tldr (long thing contains all the babble, only included because seemed low cost, don’t recommend reading):
Did exercises alone as didn’t feel like setting up something with a partner. Felt I was excited enough that should work.
steered away from 95% for all exercises where I hadn’t seen the puzzle before as was afraid that there’s a trick.
I mostly noticed how extremely FUN I found this! Just today when reflecting that studying for university courses has kind of killed some of my enthusiasm and I didn’t really remember last time being really excited while studying or in free time. Tried playing games (like chess), but even that felt more like doing the motions and got me more addicted and not actually in a flow state of mind. Somehow this just clicked.
Training regimen:
Probably opting for speed. This seemed on the easier side.
I will maybe try 5 minutes per question and see how that goes for 10 of them.
Journal
steam locomotive
Intuition: It seems like bigger wheels would be better for higher speed, but might be more wasteful.
abstract
This is in the momentum collum. It seems like one difference would be that the train with lower load, but speed, needs to efficiently maintain that speed. The one for freight needs better breaks.
I also am not sure if I am supposed to use other evidence. On the other hand, why would I have received.
Overall it seems the higher train is made in a way that is designed for slower speed (higher, less big Schornstein.) the thing stopping things in the front
There is something unintuitive about Gänge here as well! Having a big wheel means on rotation of the engine is covering more ground. This definitely seems like the thing that you want for the fast train with less load! On the other hand I think there is a high chance I have the direction backwards there.
I really don’t like the trick of having two options that they are both trains of the same kind! I feel like that makes it hard for me to become confident!
Anything left confusing? I don’t know lots about time or engines! Not 100% sure on direction of thing! Not sure how tricky questions are I think I want to not get overconfident (gpt-4 thing got me!)! I also haven’t spent 20 minutes!
Tracks as hints? What about the stangen thing attached to wheels?
How does number of wheels matter?
I feel like I got most of the evidence I know how to interpret.
I am not sure if I should treat this as an exercise in not being too impatient, or in moving at the appropriate speed. I think I want to go with taking the appropriate time?
When practicing I am also not sure how well this went. I felt this exercise really didn’t give me that much to work on?
Result:
b) (90%)
a) (5%)
c,d) (5%) passenger speed, freight load.
Looking back
I was right!
Gives me more confidence that this book is trying to be straightforward and not trying to trick me.
I could have explicitly thought about if the locomotive thing would have been recommended if it seemed like this thing wouldn’t have been super object level.
I feel like I could easily taken this one on in 5 minutes if I had not expected really hard stuff.
I got the thing correct for exactly the right intuition. Nice. I want to check that in the future.
I want more books that are like that! I feel like I want to take Bryan caplan’s exam that he gave gpt-3 like that (since gpt-4 still failed and I feel like I would remember the questions.)
I like how they didn’t spoiler me by telling me whether this was an easy or hard one though.
What else do we learn from this? Not sure? I’d be interested how hard people thought this one was?
More reflection after more exercises?
I had a hard time figuring out how to feel about gaming. It feels like I could be more efficient. Something is chasing me. On the other hand, I have time!
I’d be interested to know how many other
I still feel a bit impulsive
It is fun to babble all of my notes in this document
at the same time i feel anxiety about later pruning to decide what to post on the forum
i feel i will either dump everything there, or i will just decide later! babble!
I feel in general I have a bit of a hard time balancing meta and object level. Maybe an adhd thing? Maybe I just have the separating babble and prune as too much of a doctrine in my head that I don’t actually follow?
I notice that I love doing these artificial exercises. Fun! I feel way more motivated.
I think in general with adhd and everything I might be steering to much into not giving myself the artificial structure I need to really thrive by giving myself challenges that actually make me achieve great things!
I think I will switch to the next exercise before too much philosophizing.
Report:
I didn’t feel like finding a partner and just wanted to start with 3 problems for now.
Cold Bath
before
Archimedian principle thing (knowing density of the ice not actually required ha!).
I know the answer. The density of ice is lower than the density of water (or at least 4 degrees is the most dense I believe.)
Thing I might be a bit confused about:
if it is getting hotter than 4 degrees, at some point we could reach a temperature again where the thing spills over. My assumption (given this book has been reasonable so far.)
This seems really unfair to get confident
Final answer: Will stay exactly brim full. Ice displaced exactly as much mass as there is water in the thing (confusing stuff about air and everything else is negligable.).
prediction
a) 3%
b) 2%
c) 95%
after
right!
I don’t give myself too much credit as I had already encountered this.
Apparently I might have been eposed to too much of this. Probably lots of stuff out of this book was used by content creators I know.
I did end up needing to precisify my answer and I also didn’t notice even without knowing ice density, you can solve this with archimedian principle.
I think I also want to prod internal physics simulation engine more (not only the verbal one.)
Rare Air
Dang! I ended up exactly on the wrong page and spoilering myself! I had thought a tad before that I hadn’t written down how annoying it is to not spoiler yourself on the other exercises!
Lesson: stay careful to not get a page too far! (maybe precompute page disparity!) 9 pages!
The expansion of nothing.
Analysis
This one feels really interesting!
Intuitive model is very confused. I can see arguments for all three.
Slightly more intuitive if it would get smaller though.
Seems coincidental to just stay same (but eh… toy physics problems sometimes do this)
Intuition-pumps
What if we had the rod without the circle?
What happens without the circle?
What happens if we repeat?
What is the mechanism behind the expansion in the first place? I guess we have electrons in higher states. Everything is in higher energy and pushing away from each other?
Is there an analogy with other stuff that has force like this?
What if I imagine concrete points?
Making the thing really thin gives me the strong intuition that everything within the same radius is going to push each other apart, resulting in hole being bigger! Not what internal physics engine said!
I am also not sure if there is going to be some strain because the balance of material is not working out anymore?
Reminds me of the orange thing, that no matter if you have an orange or the earth, increasing your circumference is going to do the same to your radius. Means the shape would just stay the same. Everything just gets a bigger radius.
How to resolve remaining confusion?
I could try to dig deeper into how the stretching apart might work.
I could dig a bit deeper into ..j
I have learned some stuff about mechanics and lagrangians/Hamiltonians and going from normal to radial coordintees. Is that stuff any help here?
I feel if I would hit it from the top, it would still give me a different answer
Not sure if principiled to give 95% when I am still into other models? How confident in meta thing?
Noting that I have “SO” much fun doing this!
I remember just a few hours earlier feeling like I miss this feeling of just being really enthusiastic! Not sure if that was just me being not really reflective, or if that is really the case and I should attend to this. All the generic advice out there kind of tells me that I should perhaps not stop myself and just continue riding the wave for now?
For: follow your interests, there’s this guy who just for fun did all the problems sets in one go. (Paul graham) I find them effortful
Overenthusiuams seems the only real way people with adhd operate
Against:
People who work on this not just for one evening but over extended periods might actually form longterm differences with their brainz.
Noting that I feel like actually applying the finding portends for and against thing explicitly so strongly since I have not made predictions for sometime.
takeaway
I also just notice that with this exercise I just felt entitled to start
With research on AI safety stuff I feel like I am waiting for this gatekeeper to tell me that I’ll not be wasting peoples time by working on xyz. Not sure that is an actual problem in general. Specifically doesn’t seem super productive compared to just getting excited and started on things though!
I was still using slightly more sketchy analysis this time! I did realize that you could take the ring apart, but then I threw this thought away before thinking about what would actually happen if take apart, heat, take back together.
In my mind I took things not really apart, but kept them in the same place when heating. I would not have expected to still get the same answer!
I did not come up with something close to the taking a photo and expanding the whole photo analogy
I do feel like I had something close to that!
I feel great because deliberation actually got me closer from my initial first guess. Kind of suspicion though, that I was in kinda modest mode and I took more interesting intuition, but if pressed I would have gone with the expansion. (Could be hindsight bias)
All in all I really liked this challenge! Very fun!
Prediction:
a) 80%
b) 10%
c) 10%
(The >! thing didn’t work in markdown I just did a quick edit to change the comment to a LessWrong Docs comment where I was confident the spoiler tags would work, will look up how to do markdown later)
and, thanks and congrats!
I couldn’t quite tell where you journaling vs end takeaways started and stopped. I’m happy to read through the whole thing but you might want to edit for clarity for benefit of others.
I was copying it from my notes (with syntax for spoiler tag already in) and I belief that the lesswrong-docs mode didn’t work for that reason. Took some time because I got confused because I looked in the “welcome&faq”-post instead of the actual faq for the markdown way.
OK, a shot at Challenge I, with Poof and Foop, Steam Locomotive, and Expansion of Nothing. Felt like all three are in the sweet spot. I personally dislike Expansion of Nothing.
Poof and Foop:
The problem statement is a bit leading: there’s some kind of inversion symmetry relationship between the two cases, so it should go the opposite direction, right?
Initially, definitely. The puncture means that there’s less pressure on the right side—instead of colliding with the can, some particles go inside.
But those particles end up colliding with the interior left side anyway. So it seems like it should even out, and at the end the can won’t be moving.
So my guess is (c). Can I make myself more confident?
Why doesn’t an inversion argument go through? Well, the compressed air can is drawn in vacuum, but the vacuum can doesn’t empty the environment.
So it’s not simply time reversal. If the compressed air can were in air, then we might have some kind of symmetry between air particle and absence of air particle,
but then the compressed air can would slow down due to drag and stop in the limit. So that still points to (c). That also works as a thermodynamic argument—the first can isn’t equilibrating with anything, so none of the work goes to heat. 95% confidence feels good.
*checks* OK, looks like I was thinking about it right, and my explanation for why the naive inversion is wrong is equivalent to sketch II.
Reflection: The main interesting thing here is the fake symmetry argument. My favorite problems have tempting solutions that don’t work for subtle reasons. I think it’s important not to count problems solved until you can pinpoint why those solutions fail.
What did I use here? If you’re dealing with pressure, you can probably get an answer with forces or with thermodynamics. A net force can be thought of as a single force or as lack of a balancing force. That’s the symmetry idea.
I’m not very good at babbling. I’m basically looking over what I wrote and renarrating it. Words to words.
Steam Locomotive:
We might want to think about torque and the height of the axle.
Or maybe it’s about wheel radius. One cycle takes you further with bigger wheels.
I think these both point to (b).
I’m a little confused because thinking about the wheel heights of sports cars and trucks would push me towards (a). But cars have gears. Directly driving small wheels is basically low gear.
Not sure how I’d know if the answer were (c) or (d). Seems like you’d need background knowledge not in the question.
I should think about actual forces to get to 95% confidence.
Let’s say the engine puts out the same force in both cases. Then, in II, each wheel sees half as much force from the engine,
but the ground exerts force on twice as many wheels, so that part’s a wash. But because the wheels are smaller, the ground
needs to exert more force per unit engine force to keep the wheel from slipping (same torque).
So for the same engine, II seems to give more accelerating force, while I gives higher top speed. I’d put 95% on (b).
*checks* OK, seems like I had the right thought. Could I have been as confident from the distance-per-cycle argument alone? Rather than look at forces,
the author’s answer argues that we know the locomotive that goes a shorter distance in the same number of engine cycles must
be putting more energy into each mile it travels. I considered that, but I wasn’t sure it was a valid step.
Why couldn’t you just be getting less work from the engine? Well, it’s the same piston with the same motion.
My force calculation already needs that assumption, it just makes the final connection with the acceleration.
Reflection: I feel like I don’t know much about automotives. (Is a locomotive an automotive, by the way? I think so, it’s just locomotives involve a track.) I can describe transmission and gears and engines and so on if I think about it, but I don’t have much intuition. Like, I can’t explain why it’s one way and not another, or how different cars solve different problems.
I just feel like I should have been able to answer the question immediately. If I could drive stick, would that help? Probably not. I already ride a bike and didn’t immediately see the analogy.
What did I use? Qualitative personal experience. I picked a misleading experience but reasonably didn’t weight it above thinking through the problem. Identifying relevant considerations. Didn’t stop at the first idea.
Expansion of Nothing:
Oh, this one’s nasty. It has to expand, right?
If you took an iron disk and drew a circle where the hole is, the circle would expand.
If you cut that disk out and heat up the cutout, the disk expands the same amount.
So everything outside the circle can’t be exerting any net force at the boundary, and the hole has to stay the same size as the disk.
I don’t see any problems with this argument, but can I explain why other arguments don’t work? Why can’t thermal expansion generate stress instead of allowing uniform expansion? I guess in a sense I just gave the reason, but why does the gap shrink if you cut a gap in a rod instead? Well, when you have only one piece, it’s like applying a magnification transformation, which requires an origin. But the origin is arbitrary—you can just recenter. With two separate pieces, the two origins of magnification are no longer arbitrary.
*checks* Yeah, the author’s answer doesn’t go there, unfortunately.
Reflection: This problem feels really annoying to me. Maybe I saw it a long time ago and got it wrong? Or maybe it’s that you never have anything that’s free to expand uniformly. It’s braced against something, or it’s sitting on something with a different coefficient of thermal expansion, and you do get stress and it does matter how the thing is patterned.
This feels like a problem where you’re supposed to think about limiting cases. Like, if you have an atomic ring, obviously it expands. I don’t know if you can justify jumping to the right answer from that, though. If the disk is thick and the cutout doesn’t go all the way through, it expands. Ehh. You still need an argument that it expands the same.
I actually created a doc where people can add their own confusions and answers for Expansion of Nothing: https://docs.google.com/document/d/1cleM-QuO9R9_jRqDZMMKzobpWcf-k9KHBe91fUMWhuQ/edit
I’ll edit it into the OP.
I’d also add: a TODO item on my list is to make my own followup question for Expansion of Nothing that presents rings of different materials (i.e. something like a ring of water, a ring of jelly, a ring of concrete, something like that), and asks “in any of these cases, do you get a different answer than the Iron Ring?
I don’t actually know currently whether there exists a material that gets you a different answer, so this is a bit of a research question. Crafting the final question such that it somehow gives you enough information feels like part of the exercise. [by which I meant the meta-exercise of “creating an exercise”, which I think is also a useful thing to learn]
I’m also generally curious how you found the exercise, whether it seemed worthwhile to you.
I enjoyed it, although I’m already the sort of person who thinks Thinking Physics is fun—both the problem solving and the nitpicking about what constitutes a correct explanation. It seems worth doing at least a handful of problems this way, and more broadly deliberately practicing problem solving and metacognition about problem solving. Thinking Physics could be a good complement to Problem Solving Through Problems or How To Solve It, since in my (limited) experience you get quickly diminishing returns to anything but competition math with collections like that.
My answer to “Designing a training regimen.”
I recently spent ~2 weeks on this. I iterated on the approach over time, and didn’t really try to this “design training” exercise at the beginning.
My starting approach was the “aim for 95% confidence” (now listed as a requirement in the OP), based on receiving that advice from a friend and finding the general idea compelling. Initially I aimed at always giving myself at least a full day to answer a question. I eventually came back to this, but pretty quickly decided it wasn’t actually the right approach.
I ended up with a separation between “training” from “testing.” During training, I’m optimizing for learning quickly. This can include erring more on looking up answers, working with partners, etc.
During testing, I focused on evaluating whether I-specifically-learned-things, so I didn’t talk to friends about my thought process much to avoid spoilers. And I gave myself a very long time (sometimes spending more than a full day on each question).
I was experimenting with workshops throughout this time, and a lot of my effort ended up going towards managing other people and making sure they were having a good time. One of the things I’d go back-in-time and tell myself is “don’t try to mix large workshops and doing-it-myself. Invite friends to partner with, but focus on a few people you know well.”
One major update was I shouldn’t just be trying to get the right answer, I should be trying to identify the explanation the author was primarily aiming at. (Sometimes the author’s explanations are confusing or incomplete, but I think “generate lots of relevant explanations, at least one of which was the one the author generated” still seems useful for making sure I actually modeled the situation well)
I figured out partway through the process that I should be optimizing for “learning as much as I could from each question”, and that suggested a followup strategy of “choose problems that feel like I will learn a lot from”. (With the most obvious implication being ‘not too easy or too hard’, and a trickier implication being ‘requires skills that I’d still benefit from focusing on improving’)
One of the biggest problems was setting aside time to do it at all. This is a lot of cognitive work. I ultimately found I could only do this for a few hours a day and still was pretty exhausted in the evening. I think it’s relatively achievable to set aside one weekend for this but the amount of time necessary to vet “you have meaningfully improved” is pretty expensive.
I was lucky to be able to take a 2 week break where I was professionally focused on this. I think if rationality wasn’t part of my Day Job, and I couldn’t take a vacation for it, I think my approach would be to allocate one weekend-day each week towards this for a few weekends (aiming to look up the answer after an hour per question). And then, for testing… well, this feels fairly tricky. An obvious answer is just… keep allocating weekend time to it. This feels like it’d take a long time. Hrmm.
It’d be easier if “people’s ability to solve Thinking Physics problems” was better studied, and it was, say, known that some given exercises generally take an average undergrad 2 hours to deconfuse themselves on. (Then, you set yourself a 2 hour timer and submit your best answer when you’re done, rather than potentially spending days on it doublechecking yourself).
I think, for the immediate future, “take as long as you want to thoroughly understand the scenario” is a better test of thinking-skill for people doing openended research, and the fact it is it mostly makes sense to do this if you’re actually already planning to invest years in openended research with poor feedback loops.
I tried doing these exercises in my rationality group this week with 5 other people. Since we did this as part of our regular meetup, doing 1h for a single question would have taken too long (we could have done 2 questions max). Instead, we did 4 exercises in ~90 min (steam locomotive, poof and foop, expansion of nothing, rare air). We started out with relatively strong physics background (everyone knowing mechanics), so I think that wasn’t too hasty, except for the reflection part, perhaps. I gave people the first 5 minutes to think for themselves and to record their first probabilities. Then we discussed probabilities (there ended up to always be strong disagreements. Our physicist was twice >90% confident in the wrong answer).
I think because our meetups are often just more of a social meetup, there was not as big of a buy-in to go full munchkin on the exercises. Since I had already done the puzzles, I was also not participating in the discussion, as I didn’t want to leak information. I feel like that was a mistake, since I feel like by participating in the discussion I could transfer my enthusiasm and people would have had more fun and tried harder on the exercises. Next time, I am going to pick problems that I haven’t solved yet. I also forgot to do the reflections as a discussion, instead I told everyone to think about how they could have done better on their own, which was definitely worse. I then just ended up making the reflection part really short (3 min) for the first easy exercises because people didn’t seem enthusiastic.
Once we got to the rare air exercise everyone seemed to be really involved though since the exercise was obviously hard and people actually started thinking. At the end, they still converged on the wrong answer. I had a hard time reading the room for how this went. But people actually brought up whether we can try this again at our next meetup, so I guess it went well.
One of the takeaways was that people weren’t double-checking their models enough with settings they know (for example, they got rare air wrong because their definition of pressure was incorrect: particles per volume * speed)
It also took more time than I expected, where people were just trying to grok the solution (especially for poof and foop).
Is part of your hesitance that your “dataset” of thinking physics type of questions is not super large? I’d expect just doing 5 of the exercises in 50 minutes every day as a “test set” is going to get you more reliable feedback whether your daily training regime is working, but then you need to find new exercises once you run out of thinking physics questions.