Over a decade ago I read this 17 year old passage from Eliezer
When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said “John Conway—I met him at a summer math camp.” Darn, I thought, he thought of someone, and worse, it’s some ultra-famous old guy I can’t grab. I inquired how Marcello had arrived at the judgment. Marcello said, “He just struck me as having a tremendous amount of mental horsepower,” and started to explain a math problem he’d had a chance to work on with Conway.
Not what I wanted to hear.
Perhaps, relative to Marcello’s experience of Conway and his experience of me, I haven’t had a chance to show off on any subject that I’ve mastered as thoroughly as Conway had mastered his many fields of mathematics.
Or it might be that Conway’s brain is specialized off in a different direction from mine, and that I could never approach Conway’s level on math, yet Conway wouldn’t do so well on AI research.
Or...
...or I’m strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-Conway and tell them the basics, they would blaze right past me, solve the problems that have weighed on me for years, and zip off to places I can’t follow.
Is it damaging to my ego to confess that last possibility? Yes. It would be futile to deny that.
Have I really accepted that awful possibility, or am I only pretending to myself to have accepted it? Here I will say: “No, I think I have accepted it.” Why do I dare give myself so much credit? Because I’ve invested specific effort into that awful possibility. I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me. It might happen, it might not.
Or sadder: Maybe I just wasted too much time on setting up the resources to support me, instead of studying math full-time through my whole youth; or I wasted too much youth on non-mathy ideas. And this choice, my past, is irrevocable. I’ll hit a brick wall at 40, and there won’t be anything left but to pass on the resources to another mind with the potential I wasted, still young enough to learn. So to save them time, I should leave a trail to my successes, and post warning signs on my mistakes.
and idly wondered when that proto-Conway was going to show up and “blaze right past to places he couldn’t follow”.
I was reminded of this passage when reading the following exchange between Eliezer and Dwarkesh; his 15-year update was “nope that proto-Conway never showed up”:
Dwarkesh Patel1:58:57
Do you think that if you weren’t around, somebody else would have independently discovered this sort of field of alignment?
Eliezer Yudkowsky1:59:04
That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people. I’ve seen no evidence, but who knows what the alternate Everett branches of Earth are like?
Dwarkesh Patel1:59:27
But there are other kids who grew up on science fiction, so that can’t be the only part of the answer.
Eliezer Yudkowsky1:59:31
Well I sure am not surrounded by a cloud of people who are nearly Eliezer outputting 90% of the work output. And also this is not actually how things play out in a lot of places. Steve Jobs is dead, Apple apparently couldn’t find anyone else to be the next Steve Jobs of Apple, despite having really quite a lot of money with which to theoretically pay them. Maybe he didn’t really want a successor. Maybe he wanted to be irreplaceable.
I don’t actually buy that based on how this has played out in a number of places. There was a person once who I met when I was younger who had built something, had built an organization, and he was like — “Hey, Eliezer. Do you want this to take this thing over?” And I thought he was joking. And it didn’t dawn on me until years and years later, after trying hard and failing hard to replace myself, that — “Oh, yeah. I could have maybe taken a shot at doing this person’s job, and he’d probably just never found anyone else who could take over his organization and maybe asked some other people and nobody was willing.” And that’s his tragedy, that he built something and now can’t find anyone else to take it over. And if I’d known that at the time, I would have at least apologized to him.
To me it looks like people are not dense in the incredibly multidimensional space of people. There are too many dimensions and only 8 billion people on the planet. The world is full of people who have no immediate neighbors and problems that only one person can solve and other people cannot solve in quite the same way. I don’t think I’m unusual in looking around myself in that highly multidimensional space and not finding a ton of neighbors ready to take over. And if I had four people, any one of whom could do 99% of what I do, I might retire. I am tired. I probably wouldn’t. Probably the marginal contribution of that fifth person is still pretty large. I don’t know.
There’s the question of — Did you occupy a place in mind space? Did you occupy a place in social space? Did people not try to become Eliezer because they thought Eliezer already existed? My answer to that is — “Man, I don’t think Eliezer already existing would have stopped me from trying to become Eliezer.” But maybe you just look at the next Everett Branch over and there’s just some kind of empty space that someone steps up to fill, even though then they don’t end up with a lot of obvious neighbors. Maybe the world where I died in childbirth is pretty much like this one. If somehow we live to hear about that sort of thing from someone or something that can calculate it, that’s not the way I bet but if it’s true, it’d be funny. When I said no drama, that did include the concept of trying to make the story of your planet be the story of you. If it all would have played out the same way and somehow I survived to be told that. I’ll laugh and I’ll cry, and that will be the reality.
Dwarkesh Patel2:03:46
What I find interesting though, is that in your particular case, your output was so public. For example, your sequences, your science fiction and fan fiction. I’m sure hundreds of thousands of 18 year olds read it, or even younger, and presumably some of them reached out to you. I think this way I would love to learn more.
Eliezer Yudkowsky2:04:13
Part of why I’m a little bit skeptical of the story where people are just infinitely replaceable is that I tried really, really hard to create a new crop of people who could do all the stuff I could do to take over because I knew my health was not great and getting worse. I tried really, really hard to replace myself. I’m not sure where you look to find somebody else who tried that hard to replace himself. I tried. I really, really tried.
That’s what the Less wrong sequences were. They had other purposes. But first and foremost, it was me looking over my history and going — Well, I see all these blind pathways and stuff that it took me a while to figure out. I feel like I had these near misses on becoming myself. If I got here, there’s got to be ten other people, and some of them are smarter than I am, and they just need these little boosts and shifts and hints, and they can go down the pathway and turn into Super Eliezer. And that’s what the sequences were like. Other people use them for other stuff but primarily they were an instruction manual to the young Eliezers that I thought must exist out there. And they are not really here.
This was sad to read.
As an aside, “people are not dense in the incredibly multidimensional space of people” is an interesting turn of phrase, it doesn’t seem nontrivially true for the vast majority of people (me included) but is very much the case at the frontier (top thinkers, entrepreneurs, athletes, etc) where value creation goes superlinear. Nobody thought about higher dimensions like Bill Thurston for instance, perhaps the best geometric thinker in the history of math, despite Bill’s realisation that “what mathematicians most wanted and needed from me was to learn my ways of thinking, and not in fact to learn my proof of the geometrization conjecture for Haken manifolds” and subsequent years of efforts to convey his ways of thinking (he didn’t completely fail obviously, I’m saying no Super Thurstons have showed up since). Ditto Grothendieck and so on. When I first read Eliezer’s post above all those years ago I thought, what were the odds that he’d be in this reference class of ~unsubstitutable thinkers, given he was one of the first few bloggers I read? I guess while system of the world pontificators are a dime a dozen (e.g. cult leaders, tangentially I actually grew up within a few minutes of one that the police eventually raided), good builders of systems of the world are just vanishingly rare.
I think Eliezer underestimates other people because he evaluates them substantially based on how much they agree with him, and, as a consequence of him having a variety of dumb takes, smart people usually disagree with him about a bunch of stuff.
That can’t be the entire answer. If the situation was merely: “There is a person much smarter than Eliezer in approximately the same dimensions, let’s call them Eliezer2, but Eliezer is incapable of understanding that Eliezer2 is smarter than him because he judges smartness by similarity of opinion”, we still could see things such as Eliezer2 creating LessWrong2, starting a Rationality2 movement, etc. But there is no such thing.
This is about multidimensionality. It is not a problem to beat Eliezer in one specific dimension. It is a problem to find someone who could replace him, i.e. who would be comparable to him in all relevant dimensions.
This is not about scale or about bragging. You can have multiple people (actually the entire point is that there are many such people) who couldn’t replace each other, like e.g. Eliezer couldn’t replace Steve Jobs at Steve-Jobs-tasks, but also Steve Jobs couldn’t replace Eliezer at Eliezer-tasks. You lose one of those people (to an accident, or simply to old age), the world loses something.
This is counter-intuitive from certain perspective, because you would expect that among the eight billions, there certainly must be someone very close to Eliezer or Steve Jobs only younger, who could easily replace them. But it doesn’t work that way, because of the multidimensionality. You can find many people who are just as impressive or even more, but you can’t find good replacements.
(Hopefully it’s not rude to state my personal impression of Eliezer as a thinker. I think he’s enough of a public figure that it’s acceptable for me to comment on it. I’d like to note that I have benefited in many important ways from Eliezer’s writing and ideas, and I’ve generally enjoyed interacting with him in person, and I’m sad that as a result of some of our disagreements our interactions are tense.)
Yeah, I agree that there’s no one who Pareto dominates Eliezer at his top four most exceptional traits. (Which I guess I’d say are: taking important weird ideas seriously, writing compelling/moving/insightful fiction (for a certain audience), writing compelling/evocative/inspiring stuff about how humans should relate to rationality (for a certain audience), being broadly knowledgeable and having clever insights about many different fields.)
(I don’t think that he’s particularly good at thinking about AI; at the very least he is nowhere near as exceptional as he is at those other things.)
I’m not trying to disagree with you. I’m just going to ruminate unstructuredly a little on this:
I know a reasonable number of exceptional people. I am involved in a bunch of conversations about what fairly special people should do. In my experience, when you’re considering two people who might try to achieve a particular goal, it’s usually the case that each has some big advantages over the other in terms of personal capabilities. So, they naturally try to approach it fairly differently. We can think about this in the case where you are hiring CEOs for a project or speculating about what will happen when companies headed by different CEOs compete.
For example, consider the differences between Sam Altman and Dario Amodei (I don’t know either that well, nor do I understand the internal workings of OpenAI/Anthropic, so I’m sort of speculating here):
Dario, unlike Sam, is a good ML researcher. This means that Sam needs to depend more on technical judgment from other people.
Sam had way more connections in Silicon Valley tech, at least when Anthropic was founded.
Dario has lots of connections to the EA community and was able to hire a bunch of EAs.
Sam is much more suave in a certain way than Dario is. This benefits each for different audiences.
Both of them have done pretty well for themselves in similar roles.
As a CEO, it does feel pretty interesting how non-interchangeable most people are. And it’s interesting how in a lot of cases, it’s possible to compensate for one weakness with a strength that seems almost unrelated.
If Eliezer had never been around, my guess is that the situation around AI safety would be somewhat but not incredibly different (though probably overall substantially worse):
Nick Bostrom and Carl Shulman and friends were talking about all this stuff,
Shulman and Holden Karnofsky would have met and talked about AI risk.
I’m pretty sure Paul Christiano would have run across all this and started thinking about it, though perhaps more slowly? He might have tried harder to write for a public audience or get other people to if Less Wrong didn’t already exist.
The early effective altruists would have run across these ideas and been persuaded by them, though somewhat more slowly?
I’m not sure whether more or less EA community building would have happened 2016-2020. It would have been less obvious that community building efforts could work in principle, but less of the low-hanging fruit would have been plucked.
EA idea-spreading work would have been more centered around the kinds of ideas that non-Eliezer people are drawn to.
My guess is that the quality of ideas in the AI safety space would probably be better at this point?
Maybe a relevant underlying belief of mine is that Eliezer is very good at coming up with terms for things and articulating why something is important, and he also had the important strength of realizing how important AI was before that many other people had done so. But I don’t think his thinking about AI is actually very good on the merits. Most of the ideas he’s spread were originally substantially proposed by other people; his contribution was IMO mostly his reframings and popularizations. And I don’t think his most original ideas actually look that good. (See here for an AI summary.)
The early effective altruists would have run across these ideas and been persuaded by them, though somewhat more slowly?
I think I doubt this particular point. That EA embraced AI risk (to the extent that it did) seem to me like a fairly contingent historical fact due to LessWrong being one of the three original proto-communities of EA.
I think early EA could have grown into several very different scenes/movements/cultures/communities, in both from and content. That we would have broadly bought into AI risk as an important cause area doesn’t seem overdetermined to me.
Without HPMOR and his sequences, many probably wouldn’t become interested in rationality (or the way it’s presented in them) quite as quickly or at all. But then, without his fascination of certain controversial ideas (like focusing on AI takeoff/risk that depend on overly sci-fi-like threat models—like grey goo, virus that make all humans just drop dead instantly, endless intelligence self-improvement etc that we don’t know to be possible, as opposed to more realistic and verifiable threat models like “normal” pandemics, cybersecurity, military robots and normal economic/physical efficiency etc; and focusing too much on moral absolutism, and either believing AGI will have some universal “correct” ethics or we should try to ensure AGI have such ethics as the main or only path to safe AI; or various weird obsessions like the idea of legalizing r*pe etc that might have alienated many women and other readers), AI safety and rationality groups in general may have been seen as less fringe and more reasonable.
various weird obsessions like the idea of legalizing r*pe etc that might have alienated many women and other readers
Sidenote: I object to calling this a weird obsession. This was a minor-to-medium plot point in one science fiction story that he wrote, and (to my knowledge) has never advocated for or even discussed beyond the relevance to the story. I don’t think that’s an obsession.
This doesn’t feel that surprising to me. I guess my model is that different skills are correlated, and then if you pick someone who’s extremely capable at a couple of skills, it’s not that surprising if no one Pareto dominates them.
I agree that my point isn’t really responding to whether it’s surprising that there’s no one who Pareto dominates him.
What this reminds me of, is the phenomenon in the history of philosophy, where someone thinks they have figured out the system of the world, on which successors will build. But instead what happens is that people recognize a new theme that the innovator has introduced, and build their own rival systems incorporating that new theme.
For example, Kant (responding to Humean skepticism) built his system of transcendental idealism, which was supposed to be a new foundation for philosophy in general. Instead, it inaugurated the era of “German Idealism”, which included Hegel’s absolute idealism, whatever Schelling and Fichte were up to, and even Schopenhauer’s pessimism (which in turn was a source of Nietzsche’s optimism).
Another example would be the different directions that psychoanalysis took after Freud; and I’m sure there are many other examples… I should note that in addition to the rebellious intellectual offspring, there were people who built on Kant and Freud, and who called themselves (neo)Kantians and Freudians.
The closest thing to an important technical successor to Eliezer that I can think of, is Paul Christiano, co-inventor of RLHF, a central alignment technique behind the birth of ChatGPT. Many other people must have found their way to AI safety because of his works, and specific ideas of his have currency (e.g. Jan Leike, formerly of OpenAI superalignment, now at Anthropic, seems to be inspired by Coherent Extrapolated Volition). He is surely a godfather of AI safety, just as Hinton, Bengio, and LeCun were dubbed godfathers of deep learning. But the field itself is not dominated by his particular visions.
Stanislaw Lem once wrote (I translated it from Russian, because I can’t find another source in the open):
...sometimes it seems incredible to me that there are so many billions of people, many thousands of whom are gifted with imagination and expertise in the field of hypothesis generation, but no one takes on what I do. How is it that in some remote corner of Slavic land, between the North Pole and the Balkans, one madman suffers in solitude? Even from my favored statistical point of view, it follows that there must be at least a few such Quasi-Lems, Anti-Lems, Para-Lems, or Proto-Lems in the world, but here there are none.
I wonder how Eliezer would describe his “moat”, i.e., what cognitive trait or combination of traits does he have, that is rarest or hardest to cultivate in others? (Would also be interested in anyone else’s take on this.)
Yeah, I agree that there’s no one who Pareto dominates Eliezer at his top four most exceptional traits. (Which I guess I’d say are: taking important weird ideas seriously, writing compelling/moving/insightful fiction (for a certain audience), writing compelling/evocative/inspiring stuff about how humans should relate to rationality (for a certain audience), being broadly knowledgeable and having clever insights about many different fields.)
This also sounds sort of like how I’d describe what Scott Alexander is among the Pareto-best in the world at, just that Scott is high-verbal while Eliezer is high-flat (to use the SMPY’s categorisation). But Scott’s style seems more different from Eliezer’s than would be explained by verbal vs flat.
Notably, I think I disagree with Eliezer on what his moat is! I think he thinks that he’s much better at coming to correct conclusions or making substantial intellectual progress than I think he is.
That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people.
I think both of those are true, but it does not follow that history is made of individuals solving individual math problems and pushing out papers which get stacked into the intellectual tower of babel. History as far as I can see is made out of systems or ensembles of people moving around in different configurations.
Yudkowsky couldn’t do what he did without ET Jaynes, who in turn relied on the progenitors of probability and rationality including Thomas Bayes and William of Ockham. But he was also influenced “sideways” by the people who he learned from and defined himself against, the people in SL4 and the people he called idiots and the venture capitalists he once idolised for their competence and Peter Thiel and Demis Hassabis and his family. They shape (at the very least) his emotional worldview, which then shapes how he takes in information and integrates it at a deep and fundamental level. This is true insofar as it is true for any human who lives in a society. When I write anything I can feel the hands of writers past and present shaping my action space. They shape both what I write about and how I choose to write.
So yes if he was gone everything would be different. But it would also be the same, people would love and fight and struggle and cooperate. The sameness of trends manifests at a higher level of coarsegraining, the level where the systemic forces and the long dreams and molochian demons live. And none of this diminishes what he did, does, will do, or could have done. It’s just the way things are, because we can’t run randomised control trials on society.
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who’ve influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
Dwarkesh Patel1:48:36
What is the thing where we can sort of establish your track record before everybody falls over dead?
Eliezer Yudkowsky1:48:41
It’s hard. It is just easier to predict the endpoint than it is to predict the path. Some people will claim that I’ve done poorly compared to others who tried to predict things. I would dispute this. I think that the Hanson-Yudkowsky foom debate was won by Gwern Branwen, but I do think that Gwern Branwen is well to the Yudkowsky side of Yudkowsky in the original foom debate.
Roughly, Hansen was like — you’re going to have all these distinct handcrafted systems that incorporate lots of human knowledge specialized for particular domains. Handcrafted to incorporate human knowledge, not just run on giant data sets. I was like — you’re going to have a carefully crafted architecture with a bunch of subsystems and that thing is going to look at the data and not be handcrafted to the particular features of the data. It’s going to learn the data. Then the actual thing is like — Ha ha. You don’t have this handcrafted system that learns, you just stack more layers. So like, Hanson here, Yudkowsky here, reality there. This would be my interpretation of what happened in the past.
And if you want to be like — Well, who did better than that? It’s people like Shane Legg and Gwern Branwen. If you look at the whole planet, you can find somebody who made better predictions than Eliezer Yudkowsky, that’s for sure. Are these people currently telling you that you’re safe? No, they are not.
and then
Dwarkesh Patel3:39:58
Yeah, I think that’s a good place to close the discussion on AIs.
Eliezer Yudkowsky3:40:03
I do kind of want to mention one last thing. In historical terms, if you look out the actual battle that was being fought on the block, it was me going like — “I expect there to be AI systems that do a whole bunch of different stuff.” And Robin Hanson being like — “I expect there to be a whole bunch of different AI systems that do a whole different bunch of stuff.”
Dwarkesh Patel3:40:27
But that was one particular debate with one particular person.
Eliezer Yudkowsky3:40:30
Yeah, but your planet, having made the strange reason, given its own widespread theories, to not invest massive resources in having a much larger version of this conversation, as it apparently deemed prudent, given the implicit model that it had of the world, such that I was investing a bunch of resources in this and kind of dragging Robin Hanson along with me. Though he did have his own separate line of investigation into topics like these.
Being there as I was, my model having led me to this important place where the rest of the world apparently thought it was fine to let it go hang, such debate was actually what we had at the time. Are we really going to see these single AI systems that do all this different stuff? Is this whole general intelligence notion meaningful at all? And I staked out the bold position for it. It actually was bold.
And people did not all say —”Oh, Robin Hansen, you fool, why do you have this exotic position?” They were going like — “Behold these two luminaries debating, or behold these two idiots debating” and not massively coming down on one side of it or other. So in historical terms, I dislike making it out like I was right about anything when I feel I’ve been wrong about so much and yet I was right about anything.
And relative to what the rest of the planet deemed it important stuff to spend its time on, given their implicit model of how it’s going to play out, what you can do with minds, where AI goes. I think I did okay. Gwern Branwen did better. Shane Legg arguably did better.
Over a decade ago I read this 17 year old passage from Eliezer
and idly wondered when that proto-Conway was going to show up and “blaze right past to places he couldn’t follow”.
I was reminded of this passage when reading the following exchange between Eliezer and Dwarkesh; his 15-year update was “nope that proto-Conway never showed up”:
This was sad to read.
As an aside, “people are not dense in the incredibly multidimensional space of people” is an interesting turn of phrase, it doesn’t seem nontrivially true for the vast majority of people (me included) but is very much the case at the frontier (top thinkers, entrepreneurs, athletes, etc) where value creation goes superlinear. Nobody thought about higher dimensions like Bill Thurston for instance, perhaps the best geometric thinker in the history of math, despite Bill’s realisation that “what mathematicians most wanted and needed from me was to learn my ways of thinking, and not in fact to learn my proof of the geometrization conjecture for Haken manifolds” and subsequent years of efforts to convey his ways of thinking (he didn’t completely fail obviously, I’m saying no Super Thurstons have showed up since). Ditto Grothendieck and so on. When I first read Eliezer’s post above all those years ago I thought, what were the odds that he’d be in this reference class of ~unsubstitutable thinkers, given he was one of the first few bloggers I read? I guess while system of the world pontificators are a dime a dozen (e.g. cult leaders, tangentially I actually grew up within a few minutes of one that the police eventually raided), good builders of systems of the world are just vanishingly rare.
I think Eliezer underestimates other people because he evaluates them substantially based on how much they agree with him, and, as a consequence of him having a variety of dumb takes, smart people usually disagree with him about a bunch of stuff.
That can’t be the entire answer. If the situation was merely: “There is a person much smarter than Eliezer in approximately the same dimensions, let’s call them Eliezer2, but Eliezer is incapable of understanding that Eliezer2 is smarter than him because he judges smartness by similarity of opinion”, we still could see things such as Eliezer2 creating LessWrong2, starting a Rationality2 movement, etc. But there is no such thing.
This is about multidimensionality. It is not a problem to beat Eliezer in one specific dimension. It is a problem to find someone who could replace him, i.e. who would be comparable to him in all relevant dimensions.
This is not about scale or about bragging. You can have multiple people (actually the entire point is that there are many such people) who couldn’t replace each other, like e.g. Eliezer couldn’t replace Steve Jobs at Steve-Jobs-tasks, but also Steve Jobs couldn’t replace Eliezer at Eliezer-tasks. You lose one of those people (to an accident, or simply to old age), the world loses something.
This is counter-intuitive from certain perspective, because you would expect that among the eight billions, there certainly must be someone very close to Eliezer or Steve Jobs only younger, who could easily replace them. But it doesn’t work that way, because of the multidimensionality. You can find many people who are just as impressive or even more, but you can’t find good replacements.
(Hopefully it’s not rude to state my personal impression of Eliezer as a thinker. I think he’s enough of a public figure that it’s acceptable for me to comment on it. I’d like to note that I have benefited in many important ways from Eliezer’s writing and ideas, and I’ve generally enjoyed interacting with him in person, and I’m sad that as a result of some of our disagreements our interactions are tense.)
Yeah, I agree that there’s no one who Pareto dominates Eliezer at his top four most exceptional traits. (Which I guess I’d say are: taking important weird ideas seriously, writing compelling/moving/insightful fiction (for a certain audience), writing compelling/evocative/inspiring stuff about how humans should relate to rationality (for a certain audience), being broadly knowledgeable and having clever insights about many different fields.)
(I don’t think that he’s particularly good at thinking about AI; at the very least he is nowhere near as exceptional as he is at those other things.)
I’m not trying to disagree with you. I’m just going to ruminate unstructuredly a little on this:
I know a reasonable number of exceptional people. I am involved in a bunch of conversations about what fairly special people should do. In my experience, when you’re considering two people who might try to achieve a particular goal, it’s usually the case that each has some big advantages over the other in terms of personal capabilities. So, they naturally try to approach it fairly differently. We can think about this in the case where you are hiring CEOs for a project or speculating about what will happen when companies headed by different CEOs compete.
For example, consider the differences between Sam Altman and Dario Amodei (I don’t know either that well, nor do I understand the internal workings of OpenAI/Anthropic, so I’m sort of speculating here):
Dario, unlike Sam, is a good ML researcher. This means that Sam needs to depend more on technical judgment from other people.
Sam had way more connections in Silicon Valley tech, at least when Anthropic was founded.
Dario has lots of connections to the EA community and was able to hire a bunch of EAs.
Sam is much more suave in a certain way than Dario is. This benefits each for different audiences.
Both of them have done pretty well for themselves in similar roles.
As a CEO, it does feel pretty interesting how non-interchangeable most people are. And it’s interesting how in a lot of cases, it’s possible to compensate for one weakness with a strength that seems almost unrelated.
If Eliezer had never been around, my guess is that the situation around AI safety would be somewhat but not incredibly different (though probably overall substantially worse):
Nick Bostrom and Carl Shulman and friends were talking about all this stuff,
Shulman and Holden Karnofsky would have met and talked about AI risk.
I’m pretty sure Paul Christiano would have run across all this and started thinking about it, though perhaps more slowly? He might have tried harder to write for a public audience or get other people to if Less Wrong didn’t already exist.
The early effective altruists would have run across these ideas and been persuaded by them, though somewhat more slowly?
I’m not sure whether more or less EA community building would have happened 2016-2020. It would have been less obvious that community building efforts could work in principle, but less of the low-hanging fruit would have been plucked.
EA idea-spreading work would have been more centered around the kinds of ideas that non-Eliezer people are drawn to.
My guess is that the quality of ideas in the AI safety space would probably be better at this point?
Maybe a relevant underlying belief of mine is that Eliezer is very good at coming up with terms for things and articulating why something is important, and he also had the important strength of realizing how important AI was before that many other people had done so. But I don’t think his thinking about AI is actually very good on the merits. Most of the ideas he’s spread were originally substantially proposed by other people; his contribution was IMO mostly his reframings and popularizations. And I don’t think his most original ideas actually look that good. (See here for an AI summary.)
I think I doubt this particular point. That EA embraced AI risk (to the extent that it did) seem to me like a fairly contingent historical fact due to LessWrong being one of the three original proto-communities of EA.
I think early EA could have grown into several very different scenes/movements/cultures/communities, in both from and content. That we would have broadly bought into AI risk as an important cause area doesn’t seem overdetermined to me.
Without HPMOR and his sequences, many probably wouldn’t become interested in rationality (or the way it’s presented in them) quite as quickly or at all. But then, without his fascination of certain controversial ideas (like focusing on AI takeoff/risk that depend on overly sci-fi-like threat models—like grey goo, virus that make all humans just drop dead instantly, endless intelligence self-improvement etc that we don’t know to be possible, as opposed to more realistic and verifiable threat models like “normal” pandemics, cybersecurity, military robots and normal economic/physical efficiency etc; and focusing too much on moral absolutism, and either believing AGI will have some universal “correct” ethics or we should try to ensure AGI have such ethics as the main or only path to safe AI; or various weird obsessions like the idea of legalizing r*pe etc that might have alienated many women and other readers), AI safety and rationality groups in general may have been seen as less fringe and more reasonable.
Sidenote: I object to calling this a weird obsession. This was a minor-to-medium plot point in one science fiction story that he wrote, and (to my knowledge) has never advocated for or even discussed beyond the relevance to the story. I don’t think that’s an obsession.
This doesn’t feel that surprising to me. I guess my model is that different skills are correlated, and then if you pick someone who’s extremely capable at a couple of skills, it’s not that surprising if no one Pareto dominates them.
I agree that my point isn’t really responding to whether it’s surprising that there’s no one who Pareto dominates him.
What this reminds me of, is the phenomenon in the history of philosophy, where someone thinks they have figured out the system of the world, on which successors will build. But instead what happens is that people recognize a new theme that the innovator has introduced, and build their own rival systems incorporating that new theme.
For example, Kant (responding to Humean skepticism) built his system of transcendental idealism, which was supposed to be a new foundation for philosophy in general. Instead, it inaugurated the era of “German Idealism”, which included Hegel’s absolute idealism, whatever Schelling and Fichte were up to, and even Schopenhauer’s pessimism (which in turn was a source of Nietzsche’s optimism).
Another example would be the different directions that psychoanalysis took after Freud; and I’m sure there are many other examples… I should note that in addition to the rebellious intellectual offspring, there were people who built on Kant and Freud, and who called themselves (neo)Kantians and Freudians.
The closest thing to an important technical successor to Eliezer that I can think of, is Paul Christiano, co-inventor of RLHF, a central alignment technique behind the birth of ChatGPT. Many other people must have found their way to AI safety because of his works, and specific ideas of his have currency (e.g. Jan Leike, formerly of OpenAI superalignment, now at Anthropic, seems to be inspired by Coherent Extrapolated Volition). He is surely a godfather of AI safety, just as Hinton, Bengio, and LeCun were dubbed godfathers of deep learning. But the field itself is not dominated by his particular visions.
What about Nate soares?
What are his most important original ideas?
Stanislaw Lem once wrote (I translated it from Russian, because I can’t find another source in the open):
Strugatsky brothers were Quasi-Lems.
I wonder how Eliezer would describe his “moat”, i.e., what cognitive trait or combination of traits does he have, that is rarest or hardest to cultivate in others? (Would also be interested in anyone else’s take on this.)
Buck’s comment upthread has a guess:
This also sounds sort of like how I’d describe what Scott Alexander is among the Pareto-best in the world at, just that Scott is high-verbal while Eliezer is high-flat (to use the SMPY’s categorisation). But Scott’s style seems more different from Eliezer’s than would be explained by verbal vs flat.
Notably, I think I disagree with Eliezer on what his moat is! I think he thinks that he’s much better at coming to correct conclusions or making substantial intellectual progress than I think he is.
I think both of those are true, but it does not follow that history is made of individuals solving individual math problems and pushing out papers which get stacked into the intellectual tower of babel. History as far as I can see is made out of systems or ensembles of people moving around in different configurations.
Yudkowsky couldn’t do what he did without ET Jaynes, who in turn relied on the progenitors of probability and rationality including Thomas Bayes and William of Ockham. But he was also influenced “sideways” by the people who he learned from and defined himself against, the people in SL4 and the people he called idiots and the venture capitalists he once idolised for their competence and Peter Thiel and Demis Hassabis and his family. They shape (at the very least) his emotional worldview, which then shapes how he takes in information and integrates it at a deep and fundamental level. This is true insofar as it is true for any human who lives in a society. When I write anything I can feel the hands of writers past and present shaping my action space. They shape both what I write about and how I choose to write.
So yes if he was gone everything would be different. But it would also be the same, people would love and fight and struggle and cooperate. The sameness of trends manifests at a higher level of coarsegraining, the level where the systemic forces and the long dreams and molochian demons live. And none of this diminishes what he did, does, will do, or could have done. It’s just the way things are, because we can’t run randomised control trials on society.
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who’ve influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
and then