If humans are mostly a kludge of impulses, including the humans you are training, then… what exactly are you hoping to empower using “rationality training”? I mean, what wants-or-whatever will they act on after your training? What about your “rationality training” will lead them to take actions as though they want things? What will the results be?
To give a straight answer to this, if I was doing rationality training (if I was agenty enough to do something like that), I’d have the goal that the trainees finish the training with the realization that they don’t know what they want or don’t currently want anything, but they may eventually figure out what they want or want something, and therefore in the interim they should accumulate resources/optionality, avoid doing harm (things that eventually might be considered irreversibly harmful), and push towards eventually figuring out what they want. And I’d probably also teach a bunch of things to mitigate the risk that the trainees too easily convince themselves that they’ve figured out what they want.
There’s something about this framing that feels off to me and makes me worry that it could be counterproductive. I think my main concerns are something like:
1) People often figure out what they want by pursuing things they think they want and then updating on the outcomes. So making them less certain about their wants might prevent them from pursuing the things that would give them the information for actually figuring it out.
2) I think that people’s wants are often underdetermined and they could end up wanting many different things based on their choices. E.g. most people could probably be happy in many different kinds of careers that were almost entirely unlike each other, if they just picked one that offered decent working conditions and committed to it. I think this is true for a lot of things that people might potentially want, but to me the framing of “figure out what you want” implies that people’s wants are a lot more static than this.
I think this 80K article expresses these kinds of ideas pretty well in the context of career choice:
The third problem [with the advice of “follow your passion”] is that it makes it sound like you can work out the right career for you in a flash of insight. Just think deeply about what truly most motivates you, and you’ll realise your “true calling”. However, research shows we’re bad at predicting what will make us happiest ahead of time, and where we’ll perform best. When it comes to career decisions, our gut is often unreliable. Rather than reflecting on your passions, if you want to find a great career, you need to go and try lots of things.
The fourth problem is that it can make people needlessly limit their options. If you’re interested in literature, it’s easy to think you must become a writer to have a satisfying career, and ignore other options.
But in fact, you can start a career in a new area. If your work helps others, you practice to get good at it, you work on engaging tasks, and you work with people you like, then you’ll become passionate about it. The ingredients of a dream job we’ve found are most supported by the evidence, are all about the context of the work, not the content. Ten years ago, we would have never imagined being passionate about giving career advice, but here we are, writing this article.
Many successful people are passionate, but often their passion developed alongside their success, rather than coming first. Steve Jobs started out passionate about zen buddhism. He got into technology as a way to make some quick cash. But as he became successful, his passion grew, until he became the most famous advocate of “doing what you love”.
To give a straight answer to this, if I was doing rationality training (if I was agenty enough to do something like that), I’d have the goal that the trainees finish the training with the realization that they don’t know what they want or don’t currently want anything, but they may eventually figure out what they want or want something, and therefore in the interim they should accumulate resources/optionality, avoid doing harm (things that eventually might be considered irreversibly harmful), and push towards eventually figuring out what they want. And I’d probably also teach a bunch of things to mitigate the risk that the trainees too easily convince themselves that they’ve figured out what they want.
There’s something about this framing that feels off to me and makes me worry that it could be counterproductive. I think my main concerns are something like:
1) People often figure out what they want by pursuing things they think they want and then updating on the outcomes. So making them less certain about their wants might prevent them from pursuing the things that would give them the information for actually figuring it out.
2) I think that people’s wants are often underdetermined and they could end up wanting many different things based on their choices. E.g. most people could probably be happy in many different kinds of careers that were almost entirely unlike each other, if they just picked one that offered decent working conditions and committed to it. I think this is true for a lot of things that people might potentially want, but to me the framing of “figure out what you want” implies that people’s wants are a lot more static than this.
I think this 80K article expresses these kinds of ideas pretty well in the context of career choice: