It’s my impression that various rationalist groups formed what they called dojos with the intend of having a structure that’s similar to a martial arts dojo.
One of the key aspects of that institution is the practice of katas. Katas are a series of moves that can be practiced together to learn a particular art.
As far as I can see we didn’t find anything that fits into that spot.
Why didn’t we? Is it because rationality inherently can’t be broken down into that form? Is it because we approached the problem wrongly?
Possibly because Katas aren’t a very good framework for practice? Most functional martial arts that work in e.g. the UFC will suggest at the very least practicing with a partner, and gradually working up to live resistance. Martials arts that emphasize katas over live resistance tend to not be great at self-defense.
My guess is there’s something similar going on with rationality. Any sort of kata that doesn’t have a heavy component of “gradually working up to interacting with the real world” probably won’t be very effective.
While it’s true that live resistance is an essential component, it’s important to note that effective martial arts do katas as well under different names. In wrestling, it’s “drilling”. In boxing, it’s hitting the bag or pads.
The main difference between Katas and Drilling (at least most of the time) is that Katas are usually long chains of moves, and drills are a repetitive practice of a single, short move.
Another difference is drills are more often practiced with another person rather than on the air (even hitting a bag is still better than hitting air).
Krav Maga style defenses are in the middle, they don’t have you do lots of moves out chained together of context in the air, but are also not just one move, they’re usually ~5 moves intended for a specific situation practiced on another person with slight resistance.
The gap isn’t nearly so clear cut. In wrestling, we’d do solo drills of shots/sprawls/sit outs/etc every day, and sometimes combine them into chains like sit out->granby->stand up-> shot. The karate guys next to us did their partner katas all the time.
The difference is that when karate dude hits air, it looks like a joke. For all you can tell watching it, not only has this dude never been in a real fight, neither have any of his instructors or his instructors instructors. When Mike Tyson hits air, it’s terrifying, and you can tell just from watching it that this guy is very in touch with what is required for knocking out skilled opponents. It’s more about whether the connection with the ultimate test is still there and informing how the katas are done than whether you’re doing it alone or for multiple moves in a row or anything.
So katas are more ‘exercise’ than mastery?
Well, they’re mastery of a different thing. They’re great for entertainment and cool looking demonstrations.
I actually feel like a bunch of forecasting stuff got surprisingly close to being katas for rationality. Like “predicting one Metaculus question a day” seems like a good kata to me, though I might be misunderstanding the concept.
A kata contains a bunch of individual moves that are usually done in the same order every time and practiced together.
The idea is by practicing them that way the individual moves will go into muscle memory and executed in a fast way.
Predicting a metaculus question would be a general task that can be accomplished in a lot of different ways.
This is very true, but I think it misses a key point in what makes katas useful for actually learning a martial art in the first place. As noted in Matt Goldenberg’s answer, partner work is much more important for actually learning to use a martial art. Just practicing a kata by rote may look pretty, but it won’t tell you anything about how to use it. My own best teachers would teach moves and combinations and katas by rote at first, then very quickly move on to exercises that require creative application. Things like:
Mushin practice—get attacked and respond quickly, and make it work
Randori practice—get attacked repeatedly by multiple opponents, and make it work
Sparring and grappling practice
All of these can be modified with constraints to make them easier or harder. Easier might be “Every attack against you will be this kind of punch.” Harder can be something like “Choose one part of one kata. Come up with a way to use it effectively against whatever your opponent decides to throw at you.” Or “Figure out X different ways to use that same sequence of moves, with only minor variations, in different situations, then execute it and see how well it works.” Hopefully you’ve been practicing smart all along, and visualizing opponents while practicing your katas in the air, and developing your understanding of the moves and body mechanics so your interpretations make sense!
To bring this back to the original question: what are we thinking of as the fundamental “moves” we’re stringing together to make a “kata”? And is a rationality “kata” practice a one-person or two-person activity? Maybe a rationality kata is made up of moves like “1) Figure out what question/problem you’re trying to solve. 2) Spend five minutes thinking about what you know that may be at all relevant. 3) Make a list of unknowns that would be useful to know, and estimate how hard it would be to get the answers. 4) Make a list of strategies and techniques you might try to find a solution, and estimate their odds of success and time required. 5) Rank the results of #4 however you like, and work your way down the list until the problem is solved. 6) Go back to #2 and repeat until you have a good enough solution.” Or they can be more specific—katas for calibration, katas for probability estimation, and so on. “Practicing” katas, then, would be a matter of repeating the words of the steps, to commit them to the mental equivalent of muscle memory. This will be partly crystallizing useful concepts into short handles that leap to mind when needed, a kind of mental Miyagi-ing.
Also, in martial arts, katas can be practiced many different ways. For my own black belt test, some of the kata portion included things like “Precisely execute every kata you know, in order, in less than X minutes total,” and “choose one kata you know, and execute is as slowly as possible, minimum Y minutes, while maintaining precision and focus on every part of every movement,” and “perform this kata correctly, but make it fit completely in a box no more than Z feet across.” I think I’ve also been asked to do the mirror image of a kata a few times, though not as part of a rank test. Similarly, analyzing and verbalizing why a kata is a certain way, and what it’s trying to explore and teach, can be helpful for guiding further practice.
Then, more applied practice might look more like randori, where you get random problems thrown at you repeatedly and you have to use a specific kata to solve them. Or like mushin, where you have to practice quickly figuring out which kata to use for a problem that gets thrown at you, and then make that one work.
And yes, I realize I’ve just accidentally come very close to describing some of Jeffreysai’s methods, which is obviously not a coincidence, I’m sure.
Second this. This is where the katas are.
Also seconding Matt Goldenberg’s point on katas. A kata is akin to learning by rote memorization for an academic test instead of learning to approximate causal models on the fly for use in practical projects.
I think it’s an issue of “inside the box” vs “open-ended” fields, that we don’t have really good vocabulary to talk about. ‘Katas’ work great for sports that are very much inside the box. You can innovate new strategies, but the rules of the sport set up an unchanging microworld that you must stay inside of. Coincidentally, these are also areas where even current-day AIs often dominate. Established scientific disciplines with research programs are sort of half and half. You can train people in them, but they can also benefit from serious paradigm shifts and there aren’t any a priori hard and fast rules about things that absolutely can’t be done, like the rules of chess for the chess-playing domain.
Then there’s proto-science when things haven’t coalesced into a discipline yet, philosophy when it hasn’t been professionalized to death, Kegan’s stage 5. This is raw pattern matching, flashes of insight, original seeing, very open ended exploration of an unknown landscape. I don’t think anyone has had much of an idea for how to systematically train people for this. This place is also where a lot of the actually efficient rationality practice lives.
I did wind up with some personal katas.
Calibration using search: anytime I am searching for something with a quantitative answer I have the chance to do a fermi estimate/reference class forecasting and getting feedback on how I did.
Selection effects/Straussian readings: trying to figure out what incentives drove a particular piece of information to be in front of me in this moment.
Stack trace: finding the provenance of internal maps and noticing that they are often predicated on extremely sparse data which is then overgeneralized.
Schematic thinking: An extension of narrative fallacy bias. Noticing when alternatives would be equally valid when replacing parts of arguments. The implied degrees of freedom make the proposed explanation weaker than it might otherwise seem.
Do you have an explicit process for this?
Same as posts on fermi estimates. I just make a guess to whatever level of effort seems appropriate for the query (often pretty casual, but will take a bit more time if it’s about something I feel is important or especially uncertain about). Then when I am getting the piece of info I can reflect on reasons I might have been off. This often helps structure my inquiry into the piece of info also as I map out model differences that are why it surprised me.
Do you write the guess down somewhere or just keep it in your head?
Written down if I need to multiply a few values to get a ballpark. In head if it’s just a direct guess.
Why aren’t there katas for diet? Because diet is about not caving to strong temptations inherent in human nature, and it’s hard to practice not doing something. Maybe rationality is the same, but instead of eating bad foods, the temptation is allowing non-truth-tracking factors to influence your beliefs.
People like having superpowers and don’t like obeying duties, so those who try to spread rationality are pressured to present it as a superpower instead of a duty.
I could imagine an exercise consisting of walking amidst hundreds of delicious cakes, where you know that if you start eating them, no one will stop you...
A more cruel exercise would be having dozens of unknown meals (for example, spheres of blended matter, colored by random food colors) that you are supposed to taste, eat the healthy ones, and spit out the unhealthy ones. (To make it simple, the healthy ones are vegetables, dairy, and meat; the unhealthy ones all contain lots of sugar and/or salt.)
The idea in both cases is that if you repeatedly succeed in the arena, you will become more resistant against temptations in real life, perhaps to the point that you will stop perceiving cakes as food options, and even if you accidentally taste one, you will automatically spit it out.
But even this is ultimately a mindless activity. Rationality is about judgment, seeing things in perspective, etc. Though some parts of it could be made “automatical”, such as probability estimates (like calibration game, except that you are interrupted by phone at random moments of your day, and asked to quickly estimate a probability of something), murphy-jitsu (but again somehow being interrupted and asked to do this in real-life situations). Maybe a browser plugin that would detect you writing a general statement, and open a popup window asking you to provide three specific examples?
EDIT: An interesting and immediately useful exercise would be to establish a daily routine where in the morning you list your most important short-term and long-term tasks, and provide a probability estimate that you will do that thing today (the entire short-term thing, or the specified “next step” of the long-term thing). This should help you focus on the important stuff, and also become realistic about your abilities to do it.
Math problems are like “katas” for rationality. The difference is that, once you’ve solved a problem once with rationality, you can solve it again much more easily from memory without engaging your rational facilities again. Therefore you don’t get the benefit from repeating the same exercises again and again.
0. You assume there’s one optimal solution, and that it has already been found, i.e. no room for improvement, and that nothing changes that would affect the solution.
1. You can change the math problem. (Coming up with or finding novel new problems is also useful, but with work, an old problem can be extended. You’ve heard of the Monty Hall problem, but how does it change if you have more doors, and more time with doors being opened? If another person was playing a similar game, given the option, would you gain in expected value if switched with them?)
2. You may find your (old) solution is wrong. (A ‘solution’ you find online can be wrong.)
3. You can try to solve a problem using a different technique. (Try to find 1-the probability first. Exact solution instead of approximate, or the other. Numeric solution, functional solution (in terms of parameters or variables). Often, the set of parameters can be extended.)
4.
If you’ve forgotten something, then you can learn it again. Likewise, review (though with decreasing frequency) allows retention.
5. (See below.)
Varying the problem helps, as does varying your approach to the problem. Studying math generally involves many years of working progressively complex problems. But this is different from a “kata”, which is a set of moves rigorously repeated in a specific order and invariant manner*.
Psychologically speaking, a kata functions to take a set of moves that the student consciously understands and build muscle memories that can execute the moves effectively at the sub-second timescale of a fight. Reasoning uses different cognitive systems, although once a behavior is consciously understood it is often useful to practice until mental subroutines are developed that enable quick, unreflective execution of the behavior.
*There will be some variation as the student goes from not being good at the kata to being very good at the kata, but that variation is unwanted. The better you are at the kata the less variation there is.
I wonder, though—maybe there are some rational skills that do benefit from repetitive practice? Overcoming bias comes to mind—even after you recognize the bias, sometimes it still takes mental energy to resist its temptation. Maybe katas could help there?
It’s also helpful to practice when you’re first learning something.
I can think of a few skills that, while not “rationality” in themselves, make it much easier to reason effectively. Numeracy is one. The innumerate can’t really see the difference between a million, a billion, a trillion, and a godzillion.
It helps to have, in memory, a set of references to compare to. For example, there are about a third of a billion people in the United States. Therefore a billion dollars is roughly $3 each, a trillion dollars is roughly $3,000 each, and a million dollars is roughly nothing (.3 cents) each.
A working knowledge of history is also helpful, as is a rough understanding of manufacturing.
I do sudokus. These are computer-generated, and of consistent difficulty. so I can’t solve them from memory. Perhaps something similar could be done for math or logic problems, or story problems where cognitive biases work against the solutions.
Perhaps, but it would surprise me if you don’t have hundreds of common sudoku patterns in your memory. Not entire puzzles, but heuristics for solving limited parts of the puzzle. That’s how humans learn. We do pattern recognition whenever possible and fall back on reason when we’re stumped. “Learning” substantially consists of developing the heuristics that allow you to perform without reason (which is slow and error-prone).
Right, and if doing computer-generated sudokus is a kata for developing the heuristics for doing sudokus, then perhaps solving computer-generated logic problems could be a kata for developing the heuristics for rationality.
The problem is that to be good at rationality you need to be good at interacting with the real-world with all it’s uncertainty.
I think we need to distinguish between some related things here:
Rote learning is the stuff of katas, multiplication tables, etc. It’s not rationality in itself, but reason works best if you have a lot of reliable premises.
Developing heuristics is the stuff of everyday education. Most people get years of this stuff, and it’s what makes most people as rational as they are.
Crystallized intelligence it the ability to reason by applying heuristics. Most people aren’t very good at it, which is the main limitation on education. AFAIK, we don’t know how to give people more.
Fluid intelligence is the ability to reason creatively without heuristics. It’s the closest to what I mean by “rationality”, but also the hardest to train.
Executive function includes some basic cognitive processes that govern people’s behavior. Unfortunately, it is almost entirely (86-92%) genetic.
Interesting. I think of heuristics as being almost the same as cognitive biases. If it helps System 1, it’s a heuristic. If it gets in the way of System 2, it’s a cognitive bias.
Not a disagreement, just an observation that we are using language differently.
I basically agree. A heuristic lets System 1 function without invoking (the much slower) System 2. We need heuristics to get through the day; we couldn’t function if we had to reason out every single behavior we implement. A bias is a heuristic when it’s dysfunctional, resulting in a poorly-chosen System 1 behavior when System 2 could give a significantly better outcome.
One barrier to rationality is that updating one’s heuristics is effortful and often kind of annoying, so we always have some outdated heuristics. The quicker things change, the worse it gets. Too much trust in one’s heuristics risks biased behavior; too little yields indecisiveness.
John Wentworth and asjayan’s framing exercises seem to fit the bill, but I’m the only one consistently participating as of late…
It’s my impression that John regularly provides different exercises and that there’s no continues practice of the same task.
In my opinion, what we focus on is sort of like SCUBA diving. Some overall training is helpful, so that you can understand what the equipment is for and how to use it. It does decompose into parts. But it’s not a high-pressure sport, and skill comes from mainly going through the overall process and dealing with the random challenges that come up in individual circumstances. So you could teach it as “katas,” but that would be a relatively inefficient way to learn.