Okay, yeah this should have been dealt with in the OP. I have thoughts about this but I did write the essay in a bit of a rush. I agree this is one of the strongest objections.
I had someone do some review of the transfer learning literature. There was nonzero stuff there seemed to demonstrably work. But mostly it just seemed like we just don’t really have good experiments on the stuff I’d have expected to work. (And, the sorts of experiments that I’d expect to work are quite expensive)
But I don’t think “universal transfer learning” is quite the phrase here.
If you learn arithmetic, you (probably) don’t get better at arbitrary other skills. But, you get to use arithmetic wherever it’s relevant. You do have to separately practice “notice places where arithmetic is relevant.” (like it may not occur to you that many problems you face are actually math problem. Or, you might need an additional skill like Fermi Estimation to turn them into math problems)
The claim here is more like: “Noticing confusion”, “having more than 1 hypothesis”, “noticing yourself flinching away from a thought that’d be inconvenient” are skills that show up in multiple domains.
My “c’mon guys” here is not “c’mon the empirical evidence here is overwhelming.” It’s more like “look, which world do you actually expect to result in you making better decisions faster: the one where you spend >0 days on testing and reflecting on your thinking in areas where there is real feedback, or the one where you just spend all your time on ‘object level work’ that doesn’t really have the ability to tell you you were wrong?”.
(and, a host of similar questions, with the meta question is “do you really expect the optimal thing here to be zero effort on metacognition practice of some kind?”)
Obviously there is a question of how much time to spend on this is optimal, and it’s definitely possible (and perhaps common) to go overboard. But I also think it’s not too hard to figure out how to navigate that.
My “c’mon guys” here is not “c’mon the empirical evidence here is overwhelming.” It’s more like “look, which world do you actually expect to result in you making better decisions faster: the one where you spend >0 days on testing and reflecting on your thinking in areas where there is real feedback, or the one where you just spend all your time on ‘object level work’ that doesn’t really have the ability to tell you you were wrong?”.
(and, a host of similar questions, with the meta question is “do you really expect the optimal thing here to be zero effort on metacognition practice of some kind?”)
I mostly agree in general and I feel ya on the “c’mon guys” thing, yet I don’t do my own separate “rationality practice”.
For me, it’s basically the same reason why I don’t spend much time in a weight room anymore; I prefer to keep my strength by doing things that require and use strength. I’m not against weight lifting in principle, and I’ve done a decent amount of it. It’s just that when I have a choice between “exercise muscles for the sake of exercising muscles” and “exercise muscles in the process of doing something else I want to do anyway”, the latter is a pure win if the exercise is anywhere near equivalent. Not only is it “two birds with one stone”, it also streamlines the process of making sure you’re training the right muscles for the uses you actually have, and streamlines the process of maintaining motivation with proof that it is concretely useful.
The option isn’t always available, obviously. If your object level work doesn’t have good feedback, or you’re not strong enough to do your job, then specific training absolutely makes sense. Personally though, I find more than enough opportunities to work on meta cognition as applied to actual things I am doing for object level reasons.
The thing that seems more important to me isn’t whether you’re doing a separate practice for the sake of learning, but whether you’re reflecting on your thinking in areas where there’s real feedback, and you’re noticing that feedback. I do think there’s a place for working on artificial problems, but I also think there’s an under recognized place for picking the right real world problems for your current ability level with an expectation of learning to level up. And an underappreciated skill in finding feedback on less legible problems.
Okay, yeah this should have been dealt with in the OP. I have thoughts about this but I did write the essay in a bit of a rush. I agree this is one of the strongest objections.
I had someone do some review of the transfer learning literature. There was nonzero stuff there seemed to demonstrably work. But mostly it just seemed like we just don’t really have good experiments on the stuff I’d have expected to work. (And, the sorts of experiments that I’d expect to work are quite expensive)
But I don’t think “universal transfer learning” is quite the phrase here.
If you learn arithmetic, you (probably) don’t get better at arbitrary other skills. But, you get to use arithmetic wherever it’s relevant. You do have to separately practice “notice places where arithmetic is relevant.” (like it may not occur to you that many problems you face are actually math problem. Or, you might need an additional skill like Fermi Estimation to turn them into math problems)
The claim here is more like: “Noticing confusion”, “having more than 1 hypothesis”, “noticing yourself flinching away from a thought that’d be inconvenient” are skills that show up in multiple domains.
My “c’mon guys” here is not “c’mon the empirical evidence here is overwhelming.” It’s more like “look, which world do you actually expect to result in you making better decisions faster: the one where you spend >0 days on testing and reflecting on your thinking in areas where there is real feedback, or the one where you just spend all your time on ‘object level work’ that doesn’t really have the ability to tell you you were wrong?”.
(and, a host of similar questions, with the meta question is “do you really expect the optimal thing here to be zero effort on metacognition practice of some kind?”)
Obviously there is a question of how much time to spend on this is optimal, and it’s definitely possible (and perhaps common) to go overboard. But I also think it’s not too hard to figure out how to navigate that.
I mostly agree in general and I feel ya on the “c’mon guys” thing, yet I don’t do my own separate “rationality practice”.
For me, it’s basically the same reason why I don’t spend much time in a weight room anymore; I prefer to keep my strength by doing things that require and use strength. I’m not against weight lifting in principle, and I’ve done a decent amount of it. It’s just that when I have a choice between “exercise muscles for the sake of exercising muscles” and “exercise muscles in the process of doing something else I want to do anyway”, the latter is a pure win if the exercise is anywhere near equivalent. Not only is it “two birds with one stone”, it also streamlines the process of making sure you’re training the right muscles for the uses you actually have, and streamlines the process of maintaining motivation with proof that it is concretely useful.
The option isn’t always available, obviously. If your object level work doesn’t have good feedback, or you’re not strong enough to do your job, then specific training absolutely makes sense. Personally though, I find more than enough opportunities to work on meta cognition as applied to actual things I am doing for object level reasons.
The thing that seems more important to me isn’t whether you’re doing a separate practice for the sake of learning, but whether you’re reflecting on your thinking in areas where there’s real feedback, and you’re noticing that feedback. I do think there’s a place for working on artificial problems, but I also think there’s an under recognized place for picking the right real world problems for your current ability level with an expectation of learning to level up. And an underappreciated skill in finding feedback on less legible problems.