Can you (or anyone else) try to abstract away the personal details and explain in general terms what separates EAs from rationalists? Evidently (given Luke and Carl) familiarity with rationalist philosophy isn’t it, nor is it (self-proclaimed) altruism or scope sensitivity given that Eliezer wrote “shut up and multiply”. Like in what way is Eliezer himself not an EA given that he’s been doing the highest impact things he can think of for good for most of his life?
There are many important differences in the culture that I don’t know that I can capture in this small margin.
One frame that I have on the difference, is that rationalists run a lot more on virtue ethics, and EAs lean much more consequentialist. This is not to say that rationalists don’t do tons and tons of consequentialist calculations, they obviously do, but they ultimately still strongly care about virtues (honor, integrity, dignity, honest), whereas EAs are very ready to trade away some abstract concept of virtue for a direct and measurable local outcome. Some of the best people central to the EA cluster have the virtue of being able to ‘go hard’ in ways that people more central to the rat cluster are typically more restrained about. One story I have is that EAs historically have been more willing to say “the project happening this week is the most important thing that has ever happened in my life, and I will sacrifice anything and everything in order to do marginally better on it”, which is kind of cool and also kind of cursed when “everything” includes “treating people in my life with respect” and “not misleading people”.
Perhaps more importantly, there’s also a lot of hubris that comes through in EA’s confidence about having found the most important thing, that allows EAs to behave in extremely condescending and paternalistic ways toward the rest of the world (i.e. a lot more attempted narrative control, a lot more coordination behind the scenes about what stories to tell journalists and the public). Whereas rationalists are much more in favor of blurting things out (“speak the truth even if your voice trembles”), and in respecting that other people in the world can make better choices with access to more information (e.g. investing much more in honest public discourse and argument).
(Those are some quick glosses, probably I’d think of like 5 more on the same level if I tried for a few hours.)
Perhaps more importantly, there’s also a lot of hubris that comes through in EA’s confidence about having found the most important thing, that allows EAs to behave in extremely condescending and paternalistic ways toward the rest of the world (i.e. a lot more attempted narrative control, a lot more coordination behind the scenes about what stories to tell journalists and the public).
This is an extremely ironic take, given how much hubris, condescension, and paternalism early rationalists (the early SingInst era) displayed towards the rest of the world.
They regarded themselves as the approximately the only people who were even trying to be rational—”people are insane and the world is mad”—and had the explicit plan of building a recursively self-improving AI to transform the world, without consulting anyone else, even though they knew this was extremely dangerous and hard to get right.
And later MIRI talked of “pivotal acts”, and building limited AGIs to execute them. Which is absolutely a paternalistic frame.
And throughout, the prevailing meme was “don’t tell the world about AGI, we don’t want governments to catch on, they’ll probably do something dumb. Better keep this to the right kind of smart technical people.” A view that only changed in the past couple of years.
I’m not saying that any of this was necessarily mistaken. But it was absolutely paternalistic.
”World domination is such an ugly phrase. I prefer world optimization” is obviously and consciously hubristic and paternalistic. Rationalists celebrate that!
Insofar as EAs are more partial to modest epistemology compared to rationalists (which seems right to me), it seems like a stretch, or at least some kind of weird reversal to claim that the EAs are the ones who are more condescending and paternalistic. The EAs are the one intellectual subculture, that on average, think they should defer to the experts in the rest of the world!
So while I agree with these datapoints...
i.e. a lot more attempted narrative control, a lot more coordination behind the scenes about what stories to tell journalists and the public
...the story that EAs do that because they’re more condescending and more paternalistic doesn’t seem like an accurate model of the relevant sociology.
One possible alternative: the difference is that EAs (on average) implicitly believe that they can accomplish their aims via politics—making alliances with the right powerful people, and forming coalitions that comprise groups that they disagree with—than rationalists.
It’s not that rationalists are less paternalistic than EAs; it’s that they’re not savvy enough about politics or hopeful enough about politics for explicit pushes at narrative control to matter for their plans, such as they have them.
This hypothesis also doesn’t ring true to me, but it seems closer.
You don’t agree with half of these details in the sense that you dispute that the relevant people had those views / stances? Or you agree they had those views / stances, but dispute the implication of paternalism and hubris?
Do you want to state which you dispute?
I didn’t take the time to track down citations (and still might not bother, even if you pick out the ones that you think are false), but I claim all of this is documented and we can find receipts to back up all of this.
@Wei Dai I saw a comment by you sharing some LLM output that was then removed. That’s fine, but just as a matter of LLM epistemic hygiene, I trust LLM output to be fair and balanced much more when the person also shares a link to the full prompting history for the chat, and not just the output.
(Once in the past I was shared on an LLM output that had very biasing content earlier on, where the LLM was somewhat syconphantically agreeing with the author, but that part was not shared with me.)
(I’ve been wanting to advocate for this norm more broadly, so am just taking this opportunity to bring it up.)
And throughout, the prevailing meme was “don’t tell the world about AGI, we don’t want governments to catch on, they’ll probably do something dumb. Better keep this to the right kind of smart technical people.” A view that only changed in the past couple of years.
Does pushing for a lot of public fear about this kind of research, that makes all projects hard, seem hopeless?
Eliezer Yudkowsky
What does it buy us? 3 months of delay at the cost of a tremendous amount of goodwill? 2 years of delay? What’s that delay for, if we all die at the end? Even if we then got a technical miracle, would it end up impossible to run a project that could make use of an alignment miracle, because everybody was afraid of that project? Wouldn’t that fear tend to be channeled into “ah, yes, it must be a government project, they’re the good guys” and then the government is much more hopeless and much harder to improve upon than Deepmind?
Perhaps I am just distinguishing between two lizards rather than measuring absolutely, but at least Eliezer/MIRI were willing to talk about these things openly and in public. This is just them not (yet) going down the route of public advocacy, whereas (as I say in my other comment) the OpenPhil EAs required confidentiality before they’d talk seriously about timelines, I think until ChatGPT.
Yeah, there’s an odd tension here. Both of what you and Ben are saying seems true, but it’s contradictory at first glance. So I think there must be a deeper distinction here, some concept that doesn’t immediately come to me.
Hmm… what if we run with something like Ben’s virtue ethics vs consequentialist distinction, and just say both groups are arrogant (they are very elite groups, so this is relatively justified), but in ways inflected by these thinking styles. So this would make rationalists arrogant like “I’m a better thinker and a more “correct” mind than everyone, everyone else is stupid” and EAs arrogant like “I should make decisions for everyone else and manipulate them, I should have the most power”.
That feels satisfying to me, my confusion feels tentatively resolved.
(EDIT: and there’s a close connection here between power-seeking/consequentialism and modest epistemology/anti-weirdness, since being conformist and socially aware is useful for and comes with gaining power (although it’s not clear to me which way the causation goes). Yep, this is really useful, I like this.)
(EDIT 2: More accurately, it’s more “process-based” preferences vs “outcome-oriented”/backchaining”, instead of virtue ethics vs consequentialism—seems better to directly frame it in terms of cognitive patterns instead of abstract moral values.)
It confuses me though that Luke and Carl were immersed in rationalist culture (even in close proximity to Eliezer) for years, then became EAs. If rationalist vs EA is a matter of culture, and EA culture is harmful (like you, habryka, Eliezer all seem to suggest), why would people who “grew up” in rationalist culture be so susceptible to jumping ship to EA?
I am not that confused? The cultures are greatly similar to one another relative to most cultures in the world and they have a fair bit of overlap in people. While I’m talking about the differences, they have more in common than most subculutres. I’d be confused if one of them moved to being being a bodybuilder slash gym bro, or being into K-Pop fandom, or a punk rocker. But not between rationality and EA.
I think LW-rationalism has an internal conflict: it includes virtue ethics (“speak the truth even if your voice trembles” etc) but also a strong dose of consequentialism (“optimize as hard as possible”). Sometimes in some people the second part wins. Maybe it depends on personality type, e.g. I’m too lazy to “optimize as hard as possible”, but it’s fun for me to say true things and follow other LW-rationalist virtues. For others it may be the opposite.
Wow. I was very surprised to hear something like that. But actually… It quite aligns with my recent thoughts regarding myself. At least, if you consider not just virtue ethics, but virtue/deontology vs consequentialism. Because even though I totally bite the bullet of consequentialism, follow through all the logical arguments about that. It seems that by nature I am incredibly prone to deontology. Like, if you asked 10-year-old me, I would totally say how of course you should never ever lie to anyone because you should never ever lie at all. Because it is Bad, and you should never ever do Bad things. In fact, it seems to me that following through on consequentialism and such deontology are actually rather results of the same phenomena. Some trait like being unusually “logical” or “lawful” or “rule-abiding”.
Actually, I have a whole line of reasoning how all that thought correlate with being unusually good at programming and also maths and logic (I guess, if try to make predictions, it should imply that LW should have greater amount of programmers, mathematicians, physicists etc?). Systems composed of low level universal rules, Systems which very much care about Local Validity.
However I wasn’t ever as extreme as saying that you should say the truth even to a murderer with an axe, searching for your friends, Rather opposite, I would say that of course you have free pass on lying at that situation, there are just incomparable things, worry about not lying to murderers with axes in situations like that is like worry that you are late to school when you just got into a car crash. (Though, actually, now when I know about what are actual reasons to keep promises, not lie etc, from coordination perspective, idea of having free pass on lying to murderers doesn’t make sense. I clearly wasn’t reasoning about local validity, just feeling very strongly that rules are absolute and you can’t easily excuse doing bad things)
In fact, I thought about how it may be a bad thing. Since I was very deontologically inclined by nature and just thought that I shouldn’t ever do any Bad things, I never had a chance to develop understanding of why you shouldn’t do bad things even as consequentialist. Like, EY once said something like that if you can’t understand why utilitarist shouldn’t rob banks, then you shouldn’t try to use “cold calculations” and just give up on consequentialism and don’t rob banks because of usual reasons not to do it. A bit about “cold calculations” didn’t seem right to me, at least I personally don’t at all affected by idea how you should do unethical things because it is cold and logical. I just can’t see reasons why you shoudn’t rob banks. My best idea is that it would create utilitarists terrible reputation (if you fail and get caught). But my normal reason why to not rob banks would be “because stealing is Bad and so you should never steel”. So I guess maybe such primitive reasoning about how you should Abide The Rules displaced any possible more complex reasoning why you shouldn’t steal from banks to give poor anyway. Well, at least in comparison with people otherwise my level of intelligence. It seems, usually people just wouldn’t quite understand why you shouldn’t just lie and steel every time you have an excuse (or even not have), even if it doesn’t come to giving poor.
Can you (or anyone else) try to abstract away the personal details and explain in general terms what separates EAs from rationalists? Evidently (given Luke and Carl) familiarity with rationalist philosophy isn’t it, nor is it (self-proclaimed) altruism or scope sensitivity given that Eliezer wrote “shut up and multiply”. Like in what way is Eliezer himself not an EA given that he’s been doing the highest impact things he can think of for good for most of his life?
There are many important differences in the culture that I don’t know that I can capture in this small margin.
One frame that I have on the difference, is that rationalists run a lot more on virtue ethics, and EAs lean much more consequentialist. This is not to say that rationalists don’t do tons and tons of consequentialist calculations, they obviously do, but they ultimately still strongly care about virtues (honor, integrity, dignity, honest), whereas EAs are very ready to trade away some abstract concept of virtue for a direct and measurable local outcome. Some of the best people central to the EA cluster have the virtue of being able to ‘go hard’ in ways that people more central to the rat cluster are typically more restrained about. One story I have is that EAs historically have been more willing to say “the project happening this week is the most important thing that has ever happened in my life, and I will sacrifice anything and everything in order to do marginally better on it”, which is kind of cool and also kind of cursed when “everything” includes “treating people in my life with respect” and “not misleading people”.
Perhaps more importantly, there’s also a lot of hubris that comes through in EA’s confidence about having found the most important thing, that allows EAs to behave in extremely condescending and paternalistic ways toward the rest of the world (i.e. a lot more attempted narrative control, a lot more coordination behind the scenes about what stories to tell journalists and the public). Whereas rationalists are much more in favor of blurting things out (“speak the truth even if your voice trembles”), and in respecting that other people in the world can make better choices with access to more information (e.g. investing much more in honest public discourse and argument).
(Those are some quick glosses, probably I’d think of like 5 more on the same level if I tried for a few hours.)
This is an extremely ironic take, given how much hubris, condescension, and paternalism early rationalists (the early SingInst era) displayed towards the rest of the world.
They regarded themselves as the approximately the only people who were even trying to be rational—”people are insane and the world is mad”—and had the explicit plan of building a recursively self-improving AI to transform the world, without consulting anyone else, even though they knew this was extremely dangerous and hard to get right.
And later MIRI talked of “pivotal acts”, and building limited AGIs to execute them. Which is absolutely a paternalistic frame.
And throughout, the prevailing meme was “don’t tell the world about AGI, we don’t want governments to catch on, they’ll probably do something dumb. Better keep this to the right kind of smart technical people.” A view that only changed in the past couple of years.
I’m not saying that any of this was necessarily mistaken. But it was absolutely paternalistic.
”World domination is such an ugly phrase. I prefer world optimization” is obviously and consciously hubristic and paternalistic. Rationalists celebrate that!
And respond more specifically about EA:
Insofar as EAs are more partial to modest epistemology compared to rationalists (which seems right to me), it seems like a stretch, or at least some kind of weird reversal to claim that the EAs are the ones who are more condescending and paternalistic. The EAs are the one intellectual subculture, that on average, think they should defer to the experts in the rest of the world!
So while I agree with these datapoints...
...the story that EAs do that because they’re more condescending and more paternalistic doesn’t seem like an accurate model of the relevant sociology.
One possible alternative: the difference is that EAs (on average) implicitly believe that they can accomplish their aims via politics—making alliances with the right powerful people, and forming coalitions that comprise groups that they disagree with—than rationalists.
It’s not that rationalists are less paternalistic than EAs; it’s that they’re not savvy enough about politics or hopeful enough about politics for explicit pushes at narrative control to matter for their plans, such as they have them.
This hypothesis also doesn’t ring true to me, but it seems closer.
I don’t agree with half of these details, but overall yeah that’s fair.
I still think that EA did a lot of naive consequentialist stuff in response to believing it had important Thielian!secrets, that rats didn’t.
You don’t agree with half of these details in the sense that you dispute that the relevant people had those views / stances? Or you agree they had those views / stances, but dispute the implication of paternalism and hubris?
Do you want to state which you dispute?
I didn’t take the time to track down citations (and still might not bother, even if you pick out the ones that you think are false), but I claim all of this is documented and we can find receipts to back up all of this.
@Wei Dai I saw a comment by you sharing some LLM output that was then removed. That’s fine, but just as a matter of LLM epistemic hygiene, I trust LLM output to be fair and balanced much more when the person also shares a link to the full prompting history for the chat, and not just the output.
(Once in the past I was shared on an LLM output that had very biasing content earlier on, where the LLM was somewhat syconphantically agreeing with the author, but that part was not shared with me.)
(I’ve been wanting to advocate for this norm more broadly, so am just taking this opportunity to bring it up.)
This one seems inaccurate to me:
Eliezer was the main person I was able to get useful public thinking about timelines from, writing in 2017 about both AlphGo Zero and the Foom Debate and There’s No Fire Alarm for Artificial General Intelligence, the latter of which remains one of the most influential things on my own decision-making. Whereas the OpenPhil EAs were the ones who I was explicitly required to agree to confidentiality, and told not to write about timelines, as it would cause the rest of the world to act stupidly and race.
from https://intelligence.org/2021/11/11/discussion-with-eliezer-yudkowsky-on-agi-interventions/ (this is the closet thing Perplexity found)
Anonymous
Does pushing for a lot of public fear about this kind of research, that makes all projects hard, seem hopeless?
Eliezer Yudkowsky
What does it buy us? 3 months of delay at the cost of a tremendous amount of goodwill? 2 years of delay? What’s that delay for, if we all die at the end? Even if we then got a technical miracle, would it end up impossible to run a project that could make use of an alignment miracle, because everybody was afraid of that project? Wouldn’t that fear tend to be channeled into “ah, yes, it must be a government project, they’re the good guys” and then the government is much more hopeless and much harder to improve upon than Deepmind?
Perhaps I am just distinguishing between two lizards rather than measuring absolutely, but at least Eliezer/MIRI were willing to talk about these things openly and in public. This is just them not (yet) going down the route of public advocacy, whereas (as I say in my other comment) the OpenPhil EAs required confidentiality before they’d talk seriously about timelines, I think until ChatGPT.
Yeah, there’s an odd tension here. Both of what you and Ben are saying seems true, but it’s contradictory at first glance. So I think there must be a deeper distinction here, some concept that doesn’t immediately come to me.
Hmm… what if we run with something like Ben’s virtue ethics vs consequentialist distinction, and just say both groups are arrogant (they are very elite groups, so this is relatively justified), but in ways inflected by these thinking styles. So this would make rationalists arrogant like “I’m a better thinker and a more “correct” mind than everyone, everyone else is stupid” and EAs arrogant like “I should make decisions for everyone else and manipulate them, I should have the most power”.
That feels satisfying to me, my confusion feels tentatively resolved.
(EDIT: and there’s a close connection here between power-seeking/consequentialism and modest epistemology/anti-weirdness, since being conformist and socially aware is useful for and comes with gaining power (although it’s not clear to me which way the causation goes). Yep, this is really useful, I like this.)
(EDIT 2: More accurately, it’s more “process-based” preferences vs “outcome-oriented”/backchaining”, instead of virtue ethics vs consequentialism—seems better to directly frame it in terms of cognitive patterns instead of abstract moral values.)
It confuses me though that Luke and Carl were immersed in rationalist culture (even in close proximity to Eliezer) for years, then became EAs. If rationalist vs EA is a matter of culture, and EA culture is harmful (like you, habryka, Eliezer all seem to suggest), why would people who “grew up” in rationalist culture be so susceptible to jumping ship to EA?
I am not that confused? The cultures are greatly similar to one another relative to most cultures in the world and they have a fair bit of overlap in people. While I’m talking about the differences, they have more in common than most subculutres. I’d be confused if one of them moved to being being a bodybuilder slash gym bro, or being into K-Pop fandom, or a punk rocker. But not between rationality and EA.
I think LW-rationalism has an internal conflict: it includes virtue ethics (“speak the truth even if your voice trembles” etc) but also a strong dose of consequentialism (“optimize as hard as possible”). Sometimes in some people the second part wins. Maybe it depends on personality type, e.g. I’m too lazy to “optimize as hard as possible”, but it’s fun for me to say true things and follow other LW-rationalist virtues. For others it may be the opposite.
Wow. I was very surprised to hear something like that. But actually… It quite aligns with my recent thoughts regarding myself. At least, if you consider not just virtue ethics, but virtue/deontology vs consequentialism. Because even though I totally bite the bullet of consequentialism, follow through all the logical arguments about that. It seems that by nature I am incredibly prone to deontology. Like, if you asked 10-year-old me, I would totally say how of course you should never ever lie to anyone because you should never ever lie at all. Because it is Bad, and you should never ever do Bad things. In fact, it seems to me that following through on consequentialism and such deontology are actually rather results of the same phenomena. Some trait like being unusually “logical” or “lawful” or “rule-abiding”.
Actually, I have a whole line of reasoning how all that thought correlate with being unusually good at programming and also maths and logic (I guess, if try to make predictions, it should imply that LW should have greater amount of programmers, mathematicians, physicists etc?). Systems composed of low level universal rules, Systems which very much care about Local Validity.
However I wasn’t ever as extreme as saying that you should say the truth even to a murderer with an axe, searching for your friends, Rather opposite, I would say that of course you have free pass on lying at that situation, there are just incomparable things, worry about not lying to murderers with axes in situations like that is like worry that you are late to school when you just got into a car crash. (Though, actually, now when I know about what are actual reasons to keep promises, not lie etc, from coordination perspective, idea of having free pass on lying to murderers doesn’t make sense. I clearly wasn’t reasoning about local validity, just feeling very strongly that rules are absolute and you can’t easily excuse doing bad things)
In fact, I thought about how it may be a bad thing. Since I was very deontologically inclined by nature and just thought that I shouldn’t ever do any Bad things, I never had a chance to develop understanding of why you shouldn’t do bad things even as consequentialist. Like, EY once said something like that if you can’t understand why utilitarist shouldn’t rob banks, then you shouldn’t try to use “cold calculations” and just give up on consequentialism and don’t rob banks because of usual reasons not to do it. A bit about “cold calculations” didn’t seem right to me, at least I personally don’t at all affected by idea how you should do unethical things because it is cold and logical. I just can’t see reasons why you shoudn’t rob banks. My best idea is that it would create utilitarists terrible reputation (if you fail and get caught). But my normal reason why to not rob banks would be “because stealing is Bad and so you should never steel”. So I guess maybe such primitive reasoning about how you should Abide The Rules displaced any possible more complex reasoning why you shouldn’t steal from banks to give poor anyway. Well, at least in comparison with people otherwise my level of intelligence. It seems, usually people just wouldn’t quite understand why you shouldn’t just lie and steel every time you have an excuse (or even not have), even if it doesn’t come to giving poor.