One problem with winning, is that you need to be more specific: “winning at what?” And if you try to write down the list (for a human being, let’s ignore the AI for a moment), it turns out to be quite long.
To win at life, you probably want to be rich, but you also want to be fit, you want to be smart… but the time you spend earning money is the time you can’t spend exercising, and the time you spend exercising is the time you can’t spend learning, and you should also spend some time socializing, thinking strategically about your plans, maybe meditating, you should definitely get enough sleep, if you want to eat healthy food that is not too expensive that probably means you should learn to cook… and soon the list is too long. The day only has 24 hours, so it takes a lot of discipline to accomplish all of this without burning out, even under optional conditions (physical health, mental health, supportive family, some safety network).
It is much easier to win at one specific thing, for example to be an excellent student, while your parents take care of the money and food and strategic planning. Then you can spend 8 hours on the project, and the remaining 8 hours having fun (which is important for your mental health, and makes it all sustainable).
Some people get great at one thing by sacrificing everything else. For example, they create big successful companies and make tons of money… but their partner divorces them, their kids don’t talk to them, and at some moment their health collapses and they die. Or they spend their life in poverty, focusing obsessively on their art that will enter the textbooks one day… but again, their family suffers, etc.
Alternatively, you can try the middle way, where you try to get good-but-not-great at everything. That’s kinda where I am: somewhat above average in most things, excellent in nothing. I am not even sure how I feel about it: when I look at all kinds of problems that people around me have, I am happy that I am not them; when I think about my ambitions, I feel like I wasted my entire life.
Now, instead of an individual human, consider a group. By the level of seriousness, there are two basic kinds of groups: hobbies and jobs. Hobbies are what people do in their free time, after they have spent most of their energy on their jobs, families, etc. Some people are obsessed with their hobbies, but that doesn’t necessarily translate into quality; people who have both the obsession and the quality are rare. People with priorities other than their hobby often disappear from the group when something with a higher priority appears in their private life; and even before that, they often don’t have enough energy left for the group activities, so the group productivity is low.
To succeed, most groups need to become jobs: at least some members need to get paid decent money for working for the group. (Not necessarily all members, not even most of them; some groups are okay with two or three paid people who coordinate dozens of volunteers.) This gives you members who can devote 8 hours a day to advancing the group goals, sustainably. On the other hand, in addition to the intrinsic group goals now you also have a new task, to secure money for these members (also, to do the accounting, etc.), which can actually cost you a large fraction of this extra time (applying for grant money, preparing documentation for the donors, even more complex accounting, etc.). You also need to recruit new members, solve problems between the existing members, take care of your reputation (PR), etc.
And this all doesn’t happen in a vacuum: if you have goals, you probably also have enemies—people whose goals oppose yours (no matter how good and prosocial your goals are; some people probably benefit from the existing problems and they’d hate to see them fixed), or simply people who compete for the same resources (apply for the same grants, recruit members from the same population), or even people who hate you for no good reason just because something about you rubs them the wrong way. (And this all optimistically assumes that you have never done nothing wrong; no mistake ever. Otherwise, also include people who want to punish you; some of them quite disproportionately.) Also, people who see that you have resources, and would like to take them away from you, by theft or blackmail.
The goal of the group can require many different tasks to be done: research what causes the problem, research how to fix the problem, do the things that you are allowed to do, lobby for changing the rules so that you can do more, explain the situation to people so that you get them on your side (while your enemies are trying to turn them against you). Short-term tasks vs long-term strategies. Again, your time and resources are limited, the more you spend on X, the less you can spend on Y.
...oh my, I make it sound so complicated as if nothing can ever succeed. That wouldn’t be exactly true. But the filters along the way are brutal. You need to do many things right and you need to get lucky. Most projects fail. Most successful projects succeed small. Many good projects fall apart later, or get subverted.
I am trying to offer a “glass half full” or maybe even “glass 90% full” perspective here. Sure, nature doesn’t grade you on a curve. The sperm that only gets 99.99% towards the egg is wasted. From that perspective, we probably lose, and then we probably all die. But I don’t think that we are losing because we keep making obvious stupid mistakes. I think we are actually doing surprisingly many things right. It’s just that the problem is so difficult that you can do many things right and still lose at the end. :( Because no matter how many filters you have already passed, the next filter still eliminates a majority of contestants. And we still get at least three more filters ahead of us: (1) the major players need to actually care about alignment, (2) they need to find a way how to cooperate, and eliminate those who don’t, (3) and if they try to align the AI, they have to actually succeed. Each one of these alone sounds unlikely.
But also, for full perspective, let’s look back and see how many filters we have already passed. A decade and half ago, you get one smart guy called Eliezer, worrying about a thing that no one else seems to care about. And his goal is to convince the entire planet to do it right, otherwise we all die (but at that moment, he seems to be the only one who believes that). At what odds would you bet your money that starting from there, a few years later there will be a global community, a blog that publishes research on that topic (and many other things, often unrelated) almost every day, there will be books, academic courses, and organizations focused on that idea, politicians will discuss it on TV… and the “only” remaining problem will be that the most advanced tech companies on the planet will only pay lip service to his ideas instead of seriously following them? Yep, even that last point is sufficient to kill us all, but still, isn’t it impressive how we actually got here, despite the odds?
don’t spend 3+ years on a PhD (cognitive rationality) but instead get 10 other people to work on the issue (winning rationality). And that 10x s your efficiency already.
This seems to assume that there is a pool of extremely smart and conscientious and rational people out there, with sufficient mathematical and technical skills, willing to bet their careers on your idea if you explain it to them the right way… and you only need to go there and recruit 10 of them for the cause.
I think that such people are rare, and I suspect that most of them have already heard about the cause. Workshops organized by CFAR (1, 2) are at least partially about recruiting for the cause. Books like Superintelligence can reach more people than individual recruitment. (Also, HP:MoR.)
I think that the pyramid strategy (don’t work on the cause, instead recruit other people to work on the cause) would seem fishy to the people you are trying to recruit. Like, why would I bet my academic career on a field where no one wants to work… not even you, despite caring a lot and having the skills? Actually doing the PhD and writing a few papers will help to make the field seem legitimate.
To pick an extreme example, who do you think has more capacity to solve alignment, Paul Christiano, or Elon Musk?
Have you seen what Elon Musk does with Grok recently? He definitely has the resources, but I don’t know if there is a person on this planet who can make Elon Musk listen to them and take alignment seriously. Especially now that his brain is drunk with politics.
(This is like discussing that e.g. Putin has enough money so that he could feed all the starving kids in Africa. Yeah, he probably does, but it’s irrelevant, because this is never going to happen anyway.)
As far as I can tell cognitive rationality helps but winning seems to be mostly about agency and power really. So maybe LW should talk more about these (and how to use them for good)?
Sure, agency and power are good. If you think there is a low-hanging fruit we should pick, please explain more specifically. Agency, we have discussed a lot already (1, 2, 3), but maybe there is an important angle we have missed, or something that needs repeating. Power is a zero-sum game that many people want to play, so I doubt there is a low-hanging fruit.
There is a guy called SBF who seemed to try this way really hard, and although many people admired him at that moment, it didn’t end up well, and probably did a lot of harm. (Also, Zizians were quite agenty.)
tl;dr—be specific; if you think we are making trivial mistakes, you are probably wrong
Sure, agency and power are good. If you think there is a low-hanging fruit we should pick, please explain more specifically.
I cannot be more specific about winning rationality because I don’t know how to do it.
One would first have to set out to create the art, go out there, win and report back.
Agency, we have discussed a lot already (1, 2, 3),
Power is a zero-sum game
Then again I might read more of what people have published on LW and find that it’s already as good as it gets, who knows.
I cannot be more specific about winning rationality because I don’t know how to do it. One would first have to set out to create the art, go out there, win and report back.
Yes. My experience with “winning” suggests that there are three basic categories of interventions:
stop doing something harmful or wasteful
exploit an idiosyncratic opportunity
do the correct things and experience a slow and steady growth
The first category includes things like: stop taking drugs, leave your abusive partner, quit your bad job, find new friends who are not losers, stop doomscrolling, quit social networks, etc. If you are making any of these mistakes, it may dramatically improve your life when you stop doing them. But it will be unimpressive for observers, because they will be like “why were they even doing this stupid thing?”. Also, it moves you from negative to neutral, which is great, but it doesn’t move you from neutral to positive. It will make you good, but not great. And once you stop doing the obviously bad stuff, there is little progress to be gained in this category.
The second category is great and impressive, but the advice only applies to a specific kind of person in a specific situation, so most likely it is useless for you. An example would be Scott Alexander quitting his job and starting to make money on his Substack blog: it seems to have made him rich (and more importantly, financially secure, so he no longer has to worry about doxing, and can fully focus on doing the things that he wants to do); but from our perspective, the disadvantage of this strategy is that you need to be Scott Alexander first. Similarly, I met a guy in real life, who was a successful entrepreneur, but he started to hate his business, and didn’t know what else to do; we crunched some numbers and figured out that if he sells his business, he can retire early; he did that and he seems quite happy (later he started another business, but of a different kind, and most importantly, he does not depend on it: he can simply sell it the very moment it stops being fun). Again, a great solution for him, but if you don’t already have a successful company to sell, it won’t work for you. Maybe there is something approximately in this category for me, and I am just too blind to see it; but you would need to know my specific situation, and my specific strengths and weaknesses to find it; there is no general advice. There is also no guarantee that such thing exists.
The third category is completely boring for the outside observer: get enough sleep, learn new stuff, get a good job, eat healthy food, exercise regularly, put your savings in passively managed index funds, follow dozen more pieces of wisdom like this… and unless some misfortune happens, 10 years later you will be healthy and fit; 20-30 years later you will be healthy, fit, and rich; 30-50 years later you will be healthy, fit, rich, and famous if you care about that. (This works better if you start following this strategy while you still have many years of life ahead of you.) The disadvantage is that it takes a lot of time for the benefits to appear; the first few years may be completely unimpressive, and it may require a lot of willpower to stay on the right track regardless. Notice that the entire rationality community exists only slightly longer than ten years, so even in the hypothetical case if we all followed this advice religiously (which is definitely not the case) there still wouldn’t be sufficiently dramatic evidence to convince a skeptical outsider.
Some people believe that there is also a fourth category—an intervention that works perfectly for anyone, and delivers amazing results quickly. All you need is typically to give a lot of money to the guru who will teach you the secrets, or quit your job and join the pyramid / multi-level business / binary system / whatever people call it these days while it is still growing. In my experience, this usually does not work well. -- A more innocent version of this is taking an example from the “idiosyncratic opportunity” category and trying to apply it to everyone: “Hey, did you know that Scott Alexander is successful on Substack? Why don’t you start your own blog?” and it turns out that unfortunately, you are not Scott, you don’t have the stamina to write so many so good articles, and after a few years you only have dozen articles, three followers, and zero dollars.
I think it would be useful to collect success stories in the three categories above. (Of course, there is a risk of selection bias: you won’t get the stories of people who tried to do the same, and failed.) But I suspect that the general outline would be: first you get some quick boost from abandoning the harmful things, from that point on it is mostly a lot of work that produces slow progress (which looks amazing in hindsight, but boring when observed on a daily basis), with an occasional quick jump by exploring a unique opportunity. It may help a lot if you are surrounded by smart and friendly people who support you at doing the right-but-boring things, and help you brainstorm the opportunities.
There is a methodological problem of how to distinguish between “gains from rationality” vs ordinary gains from talent and hard work. Here I used Scott Alexander as an example of a famous successful rationalist… but when I imagine him in a parallel universe where the rationality community does not exist, maybe he is just as famous and successful there, too? And even if the community provided him some good ideas, a place to publish, and encouragement, maybe in that parallel universe he found a different community that provided him the same things.
...uh, no conclusion here, other than I agree that we should collect some rationality success stories. But expect that many will be disappointing in various ways: too simple, too specific, too slow, and with dubious relation to rationality.
One problem with winning, is that you need to be more specific: “winning at what?” And if you try to write down the list (for a human being, let’s ignore the AI for a moment), it turns out to be quite long.
To win at life, you probably want to be rich, but you also want to be fit, you want to be smart… but the time you spend earning money is the time you can’t spend exercising, and the time you spend exercising is the time you can’t spend learning, and you should also spend some time socializing, thinking strategically about your plans, maybe meditating, you should definitely get enough sleep, if you want to eat healthy food that is not too expensive that probably means you should learn to cook… and soon the list is too long. The day only has 24 hours, so it takes a lot of discipline to accomplish all of this without burning out, even under optional conditions (physical health, mental health, supportive family, some safety network).
It is much easier to win at one specific thing, for example to be an excellent student, while your parents take care of the money and food and strategic planning. Then you can spend 8 hours on the project, and the remaining 8 hours having fun (which is important for your mental health, and makes it all sustainable).
Some people get great at one thing by sacrificing everything else. For example, they create big successful companies and make tons of money… but their partner divorces them, their kids don’t talk to them, and at some moment their health collapses and they die. Or they spend their life in poverty, focusing obsessively on their art that will enter the textbooks one day… but again, their family suffers, etc.
Alternatively, you can try the middle way, where you try to get good-but-not-great at everything. That’s kinda where I am: somewhat above average in most things, excellent in nothing. I am not even sure how I feel about it: when I look at all kinds of problems that people around me have, I am happy that I am not them; when I think about my ambitions, I feel like I wasted my entire life.
Now, instead of an individual human, consider a group. By the level of seriousness, there are two basic kinds of groups: hobbies and jobs. Hobbies are what people do in their free time, after they have spent most of their energy on their jobs, families, etc. Some people are obsessed with their hobbies, but that doesn’t necessarily translate into quality; people who have both the obsession and the quality are rare. People with priorities other than their hobby often disappear from the group when something with a higher priority appears in their private life; and even before that, they often don’t have enough energy left for the group activities, so the group productivity is low.
To succeed, most groups need to become jobs: at least some members need to get paid decent money for working for the group. (Not necessarily all members, not even most of them; some groups are okay with two or three paid people who coordinate dozens of volunteers.) This gives you members who can devote 8 hours a day to advancing the group goals, sustainably. On the other hand, in addition to the intrinsic group goals now you also have a new task, to secure money for these members (also, to do the accounting, etc.), which can actually cost you a large fraction of this extra time (applying for grant money, preparing documentation for the donors, even more complex accounting, etc.). You also need to recruit new members, solve problems between the existing members, take care of your reputation (PR), etc.
And this all doesn’t happen in a vacuum: if you have goals, you probably also have enemies—people whose goals oppose yours (no matter how good and prosocial your goals are; some people probably benefit from the existing problems and they’d hate to see them fixed), or simply people who compete for the same resources (apply for the same grants, recruit members from the same population), or even people who hate you for no good reason just because something about you rubs them the wrong way. (And this all optimistically assumes that you have never done nothing wrong; no mistake ever. Otherwise, also include people who want to punish you; some of them quite disproportionately.) Also, people who see that you have resources, and would like to take them away from you, by theft or blackmail.
The goal of the group can require many different tasks to be done: research what causes the problem, research how to fix the problem, do the things that you are allowed to do, lobby for changing the rules so that you can do more, explain the situation to people so that you get them on your side (while your enemies are trying to turn them against you). Short-term tasks vs long-term strategies. Again, your time and resources are limited, the more you spend on X, the less you can spend on Y.
...oh my, I make it sound so complicated as if nothing can ever succeed. That wouldn’t be exactly true. But the filters along the way are brutal. You need to do many things right and you need to get lucky. Most projects fail. Most successful projects succeed small. Many good projects fall apart later, or get subverted.
I am trying to offer a “glass half full” or maybe even “glass 90% full” perspective here. Sure, nature doesn’t grade you on a curve. The sperm that only gets 99.99% towards the egg is wasted. From that perspective, we probably lose, and then we probably all die. But I don’t think that we are losing because we keep making obvious stupid mistakes. I think we are actually doing surprisingly many things right. It’s just that the problem is so difficult that you can do many things right and still lose at the end. :( Because no matter how many filters you have already passed, the next filter still eliminates a majority of contestants. And we still get at least three more filters ahead of us: (1) the major players need to actually care about alignment, (2) they need to find a way how to cooperate, and eliminate those who don’t, (3) and if they try to align the AI, they have to actually succeed. Each one of these alone sounds unlikely.
But also, for full perspective, let’s look back and see how many filters we have already passed. A decade and half ago, you get one smart guy called Eliezer, worrying about a thing that no one else seems to care about. And his goal is to convince the entire planet to do it right, otherwise we all die (but at that moment, he seems to be the only one who believes that). At what odds would you bet your money that starting from there, a few years later there will be a global community, a blog that publishes research on that topic (and many other things, often unrelated) almost every day, there will be books, academic courses, and organizations focused on that idea, politicians will discuss it on TV… and the “only” remaining problem will be that the most advanced tech companies on the planet will only pay lip service to his ideas instead of seriously following them? Yep, even that last point is sufficient to kill us all, but still, isn’t it impressive how we actually got here, despite the odds?
This seems to assume that there is a pool of extremely smart and conscientious and rational people out there, with sufficient mathematical and technical skills, willing to bet their careers on your idea if you explain it to them the right way… and you only need to go there and recruit 10 of them for the cause.
I think that such people are rare, and I suspect that most of them have already heard about the cause. Workshops organized by CFAR (1, 2) are at least partially about recruiting for the cause. Books like Superintelligence can reach more people than individual recruitment. (Also, HP:MoR.)
I think that the pyramid strategy (don’t work on the cause, instead recruit other people to work on the cause) would seem fishy to the people you are trying to recruit. Like, why would I bet my academic career on a field where no one wants to work… not even you, despite caring a lot and having the skills? Actually doing the PhD and writing a few papers will help to make the field seem legitimate.
Have you seen what Elon Musk does with Grok recently? He definitely has the resources, but I don’t know if there is a person on this planet who can make Elon Musk listen to them and take alignment seriously. Especially now that his brain is drunk with politics.
(This is like discussing that e.g. Putin has enough money so that he could feed all the starving kids in Africa. Yeah, he probably does, but it’s irrelevant, because this is never going to happen anyway.)
Sure, agency and power are good. If you think there is a low-hanging fruit we should pick, please explain more specifically. Agency, we have discussed a lot already (1, 2, 3), but maybe there is an important angle we have missed, or something that needs repeating. Power is a zero-sum game that many people want to play, so I doubt there is a low-hanging fruit.
There is a guy called SBF who seemed to try this way really hard, and although many people admired him at that moment, it didn’t end up well, and probably did a lot of harm. (Also, Zizians were quite agenty.)
tl;dr—be specific; if you think we are making trivial mistakes, you are probably wrong
I cannot be more specific about winning rationality because I don’t know how to do it. One would first have to set out to create the art, go out there, win and report back.
Then again I might read more of what people have published on LW and find that it’s already as good as it gets, who knows.
Yes. My experience with “winning” suggests that there are three basic categories of interventions:
stop doing something harmful or wasteful
exploit an idiosyncratic opportunity
do the correct things and experience a slow and steady growth
The first category includes things like: stop taking drugs, leave your abusive partner, quit your bad job, find new friends who are not losers, stop doomscrolling, quit social networks, etc. If you are making any of these mistakes, it may dramatically improve your life when you stop doing them. But it will be unimpressive for observers, because they will be like “why were they even doing this stupid thing?”. Also, it moves you from negative to neutral, which is great, but it doesn’t move you from neutral to positive. It will make you good, but not great. And once you stop doing the obviously bad stuff, there is little progress to be gained in this category.
The second category is great and impressive, but the advice only applies to a specific kind of person in a specific situation, so most likely it is useless for you. An example would be Scott Alexander quitting his job and starting to make money on his Substack blog: it seems to have made him rich (and more importantly, financially secure, so he no longer has to worry about doxing, and can fully focus on doing the things that he wants to do); but from our perspective, the disadvantage of this strategy is that you need to be Scott Alexander first. Similarly, I met a guy in real life, who was a successful entrepreneur, but he started to hate his business, and didn’t know what else to do; we crunched some numbers and figured out that if he sells his business, he can retire early; he did that and he seems quite happy (later he started another business, but of a different kind, and most importantly, he does not depend on it: he can simply sell it the very moment it stops being fun). Again, a great solution for him, but if you don’t already have a successful company to sell, it won’t work for you. Maybe there is something approximately in this category for me, and I am just too blind to see it; but you would need to know my specific situation, and my specific strengths and weaknesses to find it; there is no general advice. There is also no guarantee that such thing exists.
The third category is completely boring for the outside observer: get enough sleep, learn new stuff, get a good job, eat healthy food, exercise regularly, put your savings in passively managed index funds, follow dozen more pieces of wisdom like this… and unless some misfortune happens, 10 years later you will be healthy and fit; 20-30 years later you will be healthy, fit, and rich; 30-50 years later you will be healthy, fit, rich, and famous if you care about that. (This works better if you start following this strategy while you still have many years of life ahead of you.) The disadvantage is that it takes a lot of time for the benefits to appear; the first few years may be completely unimpressive, and it may require a lot of willpower to stay on the right track regardless. Notice that the entire rationality community exists only slightly longer than ten years, so even in the hypothetical case if we all followed this advice religiously (which is definitely not the case) there still wouldn’t be sufficiently dramatic evidence to convince a skeptical outsider.
Some people believe that there is also a fourth category—an intervention that works perfectly for anyone, and delivers amazing results quickly. All you need is typically to give a lot of money to the guru who will teach you the secrets, or quit your job and join the pyramid / multi-level business / binary system / whatever people call it these days while it is still growing. In my experience, this usually does not work well. -- A more innocent version of this is taking an example from the “idiosyncratic opportunity” category and trying to apply it to everyone: “Hey, did you know that Scott Alexander is successful on Substack? Why don’t you start your own blog?” and it turns out that unfortunately, you are not Scott, you don’t have the stamina to write so many so good articles, and after a few years you only have dozen articles, three followers, and zero dollars.
I think it would be useful to collect success stories in the three categories above. (Of course, there is a risk of selection bias: you won’t get the stories of people who tried to do the same, and failed.) But I suspect that the general outline would be: first you get some quick boost from abandoning the harmful things, from that point on it is mostly a lot of work that produces slow progress (which looks amazing in hindsight, but boring when observed on a daily basis), with an occasional quick jump by exploring a unique opportunity. It may help a lot if you are surrounded by smart and friendly people who support you at doing the right-but-boring things, and help you brainstorm the opportunities.
There is a methodological problem of how to distinguish between “gains from rationality” vs ordinary gains from talent and hard work. Here I used Scott Alexander as an example of a famous successful rationalist… but when I imagine him in a parallel universe where the rationality community does not exist, maybe he is just as famous and successful there, too? And even if the community provided him some good ideas, a place to publish, and encouragement, maybe in that parallel universe he found a different community that provided him the same things.
...uh, no conclusion here, other than I agree that we should collect some rationality success stories. But expect that many will be disappointing in various ways: too simple, too specific, too slow, and with dubious relation to rationality.