Doing actual mini-RCTs can be pretty simple. You only need 3 things:
1. A spreadsheet
2. A digital coin for randomization
3. A way to measure the variable that you care about
I think one of practically powerful “techniques” of rationality is doing simple empirical experiments like this. You want to get something? You don’t know how to get it? Try out some ideas and check which ones work!
There are other applications of empiricism that are not as formal, and sometimes faster. Those are also awesome. But at the very least, I’ve found that doing mini-RCTs is pretty enlightening.
On the object level, you can learn what actually works for hitting your goals.
On the process level, this trains some good epistemic norms and priors.
For one thing, I now have a much stronger intuition for the likelihood that an impressive effect is just noise. And getting into the habit of doing quantified hypothesis testing, such that you can cleanly falsify your hypotheses, teaches you to hold hypotheses lightly while inclining you to generate hypotheses in the first place.
Theorizing methods can enhance and accelerate this process, but if you have a quantified empirical feedback loop, your theorizing will be grounded. Science is hard, and most of our guesses are wrong. But that’s fine, so long as we actually check.
Doing actual mini-RCTs can be pretty simple. You only need 3 things:
1. A spreadsheet
2. A digital coin for randomization
3. A way to measure the variable that you care about
I think one of practically powerful “techniques” of rationality is doing simple empirical experiments like this. You want to get something? You don’t know how to get it? Try out some ideas and check which ones work!
There are other applications of empiricism that are not as formal, and sometimes faster. Those are also awesome. But at the very least, I’ve found that doing mini-RCTs is pretty enlightening.
On the object level, you can learn what actually works for hitting your goals.
On the process level, this trains some good epistemic norms and priors.
For one thing, I now have a much stronger intuition for the likelihood that an impressive effect is just noise. And getting into the habit of doing quantified hypothesis testing, such that you can cleanly falsify your hypotheses, teaches you to hold hypotheses lightly while inclining you to generate hypotheses in the first place.
Theorizing methods can enhance and accelerate this process, but if you have a quantified empirical feedback loop, your theorizing will be grounded. Science is hard, and most of our guesses are wrong. But that’s fine, so long as we actually check.