Over a decade ago I read this 17 year old passage from Eliezer
When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said “John Conway—I met him at a summer math camp.” Darn, I thought, he thought of someone, and worse, it’s some ultra-famous old guy I can’t grab. I inquired how Marcello had arrived at the judgment. Marcello said, “He just struck me as having a tremendous amount of mental horsepower,” and started to explain a math problem he’d had a chance to work on with Conway.
Not what I wanted to hear.
Perhaps, relative to Marcello’s experience of Conway and his experience of me, I haven’t had a chance to show off on any subject that I’ve mastered as thoroughly as Conway had mastered his many fields of mathematics.
Or it might be that Conway’s brain is specialized off in a different direction from mine, and that I could never approach Conway’s level on math, yet Conway wouldn’t do so well on AI research.
Or...
...or I’m strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-Conway and tell them the basics, they would blaze right past me, solve the problems that have weighed on me for years, and zip off to places I can’t follow.
Is it damaging to my ego to confess that last possibility? Yes. It would be futile to deny that.
Have I really accepted that awful possibility, or am I only pretending to myself to have accepted it? Here I will say: “No, I think I have accepted it.” Why do I dare give myself so much credit? Because I’ve invested specific effort into that awful possibility. I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me. It might happen, it might not.
Or sadder: Maybe I just wasted too much time on setting up the resources to support me, instead of studying math full-time through my whole youth; or I wasted too much youth on non-mathy ideas. And this choice, my past, is irrevocable. I’ll hit a brick wall at 40, and there won’t be anything left but to pass on the resources to another mind with the potential I wasted, still young enough to learn. So to save them time, I should leave a trail to my successes, and post warning signs on my mistakes.
and idly wondered when that proto-Conway was going to show up and “blaze right past to places he couldn’t follow”.
I was reminded of this passage when reading the following exchange between Eliezer and Dwarkesh; his 15-year update was “nope that proto-Conway never showed up”:
Dwarkesh Patel1:58:57
Do you think that if you weren’t around, somebody else would have independently discovered this sort of field of alignment?
Eliezer Yudkowsky1:59:04
That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people. I’ve seen no evidence, but who knows what the alternate Everett branches of Earth are like?
Dwarkesh Patel1:59:27
But there are other kids who grew up on science fiction, so that can’t be the only part of the answer.
Eliezer Yudkowsky1:59:31
Well I sure am not surrounded by a cloud of people who are nearly Eliezer outputting 90% of the work output. And also this is not actually how things play out in a lot of places. Steve Jobs is dead, Apple apparently couldn’t find anyone else to be the next Steve Jobs of Apple, despite having really quite a lot of money with which to theoretically pay them. Maybe he didn’t really want a successor. Maybe he wanted to be irreplaceable.
I don’t actually buy that based on how this has played out in a number of places. There was a person once who I met when I was younger who had built something, had built an organization, and he was like — “Hey, Eliezer. Do you want this to take this thing over?” And I thought he was joking. And it didn’t dawn on me until years and years later, after trying hard and failing hard to replace myself, that — “Oh, yeah. I could have maybe taken a shot at doing this person’s job, and he’d probably just never found anyone else who could take over his organization and maybe asked some other people and nobody was willing.” And that’s his tragedy, that he built something and now can’t find anyone else to take it over. And if I’d known that at the time, I would have at least apologized to him.
To me it looks like people are not dense in the incredibly multidimensional space of people. There are too many dimensions and only 8 billion people on the planet. The world is full of people who have no immediate neighbors and problems that only one person can solve and other people cannot solve in quite the same way. I don’t think I’m unusual in looking around myself in that highly multidimensional space and not finding a ton of neighbors ready to take over. And if I had four people, any one of whom could do 99% of what I do, I might retire. I am tired. I probably wouldn’t. Probably the marginal contribution of that fifth person is still pretty large. I don’t know.
There’s the question of — Did you occupy a place in mind space? Did you occupy a place in social space? Did people not try to become Eliezer because they thought Eliezer already existed? My answer to that is — “Man, I don’t think Eliezer already existing would have stopped me from trying to become Eliezer.” But maybe you just look at the next Everett Branch over and there’s just some kind of empty space that someone steps up to fill, even though then they don’t end up with a lot of obvious neighbors. Maybe the world where I died in childbirth is pretty much like this one. If somehow we live to hear about that sort of thing from someone or something that can calculate it, that’s not the way I bet but if it’s true, it’d be funny. When I said no drama, that did include the concept of trying to make the story of your planet be the story of you. If it all would have played out the same way and somehow I survived to be told that. I’ll laugh and I’ll cry, and that will be the reality.
Dwarkesh Patel2:03:46
What I find interesting though, is that in your particular case, your output was so public. For example, your sequences, your science fiction and fan fiction. I’m sure hundreds of thousands of 18 year olds read it, or even younger, and presumably some of them reached out to you. I think this way I would love to learn more.
Eliezer Yudkowsky2:04:13
Part of why I’m a little bit skeptical of the story where people are just infinitely replaceable is that I tried really, really hard to create a new crop of people who could do all the stuff I could do to take over because I knew my health was not great and getting worse. I tried really, really hard to replace myself. I’m not sure where you look to find somebody else who tried that hard to replace himself. I tried. I really, really tried.
That’s what the Less wrong sequences were. They had other purposes. But first and foremost, it was me looking over my history and going — Well, I see all these blind pathways and stuff that it took me a while to figure out. I feel like I had these near misses on becoming myself. If I got here, there’s got to be ten other people, and some of them are smarter than I am, and they just need these little boosts and shifts and hints, and they can go down the pathway and turn into Super Eliezer. And that’s what the sequences were like. Other people use them for other stuff but primarily they were an instruction manual to the young Eliezers that I thought must exist out there. And they are not really here.
This was sad to read.
As an aside, “people are not dense in the incredibly multidimensional space of people” is an interesting turn of phrase, it doesn’t seem nontrivially true for the vast majority of people (me included) but is very much the case at the frontier (top thinkers, entrepreneurs, athletes, etc) where value creation goes superlinear. Nobody thought about higher dimensions like Bill Thurston for instance, perhaps the best geometric thinker in the history of math, despite Bill’s realisation that “what mathematicians most wanted and needed from me was to learn my ways of thinking, and not in fact to learn my proof of the geometrization conjecture for Haken manifolds” and subsequent years of efforts to convey his ways of thinking (he didn’t completely fail obviously, I’m saying no Super Thurstons have showed up since). Ditto Grothendieck and so on. When I first read Eliezer’s post above all those years ago I thought, what were the odds that he’d be in this reference class of ~unsubstitutable thinkers, given he was one of the first few bloggers I read? I guess while system of the world pontificators are a dime a dozen (e.g. cult leaders, tangentially I actually grew up within a few minutes of one that the police eventually raided), good builders of systems of the world are just vanishingly rare.
What this reminds me of, is the phenomenon in the history of philosophy, where someone thinks they have figured out the system of the world, on which successors will build. But instead what happens is that people recognize a new theme that the innovator has introduced, and build their own rival systems incorporating that new theme.
For example, Kant (responding to Humean skepticism) built his system of transcendental idealism, which was supposed to be a new foundation for philosophy in general. Instead, it inaugurated the era of “German Idealism”, which included Hegel’s absolute idealism, whatever Schelling and Fichte were up to, and even Schopenhauer’s pessimism (which in turn was a source of Nietzsche’s optimism).
Another example would be the different directions that psychoanalysis took after Freud; and I’m sure there are many other examples… I should note that in addition to the rebellious intellectual offspring, there were people who built on Kant and Freud, and who called themselves (neo)Kantians and Freudians.
The closest thing to an important technical successor to Eliezer that I can think of, is Paul Christiano, co-inventor of RLHF, a central alignment technique behind the birth of ChatGPT. Many other people must have found their way to AI safety because of his works, and specific ideas of his have currency (e.g. Jan Leike, formerly of OpenAI superalignment, now at Anthropic, seems to be inspired by Coherent Extrapolated Volition). He is surely a godfather of AI safety, just as Hinton, Bengio, and LeCun were dubbed godfathers of deep learning. But the field itself is not dominated by his particular visions.
I think Eliezer underestimates other people because he evaluates them substantially based on how much they agree with him, and, as a consequence of him having a variety of dumb takes, smart people usually disagree with him about a bunch of stuff.
I wonder how Eliezer would describe his “moat”, i.e., what cognitive trait or combination of traits does he have, that is rarest or hardest to cultivate in others? (Would also be interested in anyone else’s take on this.)
Stanislaw Lem once wrote (I translated it from Russian, because I can’t find another source in the open):
...sometimes it seems incredible to me that there are so many billions of people, many thousands of whom are gifted with imagination and expertise in the field of hypothesis generation, but no one takes on what I do. How is it that in some remote corner of Slavic land, between the North Pole and the Balkans, one madman suffers in solitude? Even from my favored statistical point of view, it follows that there must be at least a few such Quasi-Lems, Anti-Lems, Para-Lems, or Proto-Lems in the world, but here there are none.
That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people.
I think both of those are true, but it does not follow that history is made of individuals solving individual math problems and pushing out papers which get stacked into the intellectual tower of babel. History as far as I can see is made out of systems or ensembles of people moving around in different configurations.
Yudkowsky couldn’t do what he did without ET Jaynes, who in turn relied on the progenitors of probability and rationality including Thomas Bayes and William of Ockham. But he was also influenced “sideways” by the people who he learned from and defined himself against, the people in SL4 and the people he called idiots and the venture capitalists he once idolised for their competence and Peter Thiel and Demis Hassabis and his family. They shape (at the very least) his emotional worldview, which then shapes how he takes in information and integrates it at a deep and fundamental level. This is true insofar as it is true for any human who lives in a society. When I write anything I can feel the hands of writers past and present shaping my action space. They shape both what I write about and how I choose to right.
So yes if he was gone everything would be different. But it would also be the same, people would love and fight and struggle and cooperate. The sameness of trends manifests at a higher level of coarsegraining, the level where the systemic forces and the long dreams and molochian demons live. And none of this diminishes what he did, does, will do, or could have done. It’s just the way things are, because we can’t run randomised control trials on society.
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who’ve influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
Dwarkesh Patel1:48:36
What is the thing where we can sort of establish your track record before everybody falls over dead?
Eliezer Yudkowsky1:48:41
It’s hard. It is just easier to predict the endpoint than it is to predict the path. Some people will claim that I’ve done poorly compared to others who tried to predict things. I would dispute this. I think that the Hanson-Yudkowsky foom debate was won by Gwern Branwen, but I do think that Gwern Branwen is well to the Yudkowsky side of Yudkowsky in the original foom debate.
Roughly, Hansen was like — you’re going to have all these distinct handcrafted systems that incorporate lots of human knowledge specialized for particular domains. Handcrafted to incorporate human knowledge, not just run on giant data sets. I was like — you’re going to have a carefully crafted architecture with a bunch of subsystems and that thing is going to look at the data and not be handcrafted to the particular features of the data. It’s going to learn the data. Then the actual thing is like — Ha ha. You don’t have this handcrafted system that learns, you just stack more layers. So like, Hanson here, Yudkowsky here, reality there. This would be my interpretation of what happened in the past.
And if you want to be like — Well, who did better than that? It’s people like Shane Legg and Gwern Branwen. If you look at the whole planet, you can find somebody who made better predictions than Eliezer Yudkowsky, that’s for sure. Are these people currently telling you that you’re safe? No, they are not.
and then
Dwarkesh Patel3:39:58
Yeah, I think that’s a good place to close the discussion on AIs.
Eliezer Yudkowsky3:40:03
I do kind of want to mention one last thing. In historical terms, if you look out the actual battle that was being fought on the block, it was me going like — “I expect there to be AI systems that do a whole bunch of different stuff.” And Robin Hanson being like — “I expect there to be a whole bunch of different AI systems that do a whole different bunch of stuff.”
Dwarkesh Patel3:40:27
But that was one particular debate with one particular person.
Eliezer Yudkowsky3:40:30
Yeah, but your planet, having made the strange reason, given its own widespread theories, to not invest massive resources in having a much larger version of this conversation, as it apparently deemed prudent, given the implicit model that it had of the world, such that I was investing a bunch of resources in this and kind of dragging Robin Hanson along with me. Though he did have his own separate line of investigation into topics like these.
Being there as I was, my model having led me to this important place where the rest of the world apparently thought it was fine to let it go hang, such debate was actually what we had at the time. Are we really going to see these single AI systems that do all this different stuff? Is this whole general intelligence notion meaningful at all? And I staked out the bold position for it. It actually was bold.
And people did not all say —”Oh, Robin Hansen, you fool, why do you have this exotic position?” They were going like — “Behold these two luminaries debating, or behold these two idiots debating” and not massively coming down on one side of it or other. So in historical terms, I dislike making it out like I was right about anything when I feel I’ve been wrong about so much and yet I was right about anything.
And relative to what the rest of the planet deemed it important stuff to spend its time on, given their implicit model of how it’s going to play out, what you can do with minds, where AI goes. I think I did okay. Gwern Branwen did better. Shane Legg arguably did better.
Over a decade ago I read this 17 year old passage from Eliezer
and idly wondered when that proto-Conway was going to show up and “blaze right past to places he couldn’t follow”.
I was reminded of this passage when reading the following exchange between Eliezer and Dwarkesh; his 15-year update was “nope that proto-Conway never showed up”:
This was sad to read.
As an aside, “people are not dense in the incredibly multidimensional space of people” is an interesting turn of phrase, it doesn’t seem nontrivially true for the vast majority of people (me included) but is very much the case at the frontier (top thinkers, entrepreneurs, athletes, etc) where value creation goes superlinear. Nobody thought about higher dimensions like Bill Thurston for instance, perhaps the best geometric thinker in the history of math, despite Bill’s realisation that “what mathematicians most wanted and needed from me was to learn my ways of thinking, and not in fact to learn my proof of the geometrization conjecture for Haken manifolds” and subsequent years of efforts to convey his ways of thinking (he didn’t completely fail obviously, I’m saying no Super Thurstons have showed up since). Ditto Grothendieck and so on. When I first read Eliezer’s post above all those years ago I thought, what were the odds that he’d be in this reference class of ~unsubstitutable thinkers, given he was one of the first few bloggers I read? I guess while system of the world pontificators are a dime a dozen (e.g. cult leaders, tangentially I actually grew up within a few minutes of one that the police eventually raided), good builders of systems of the world are just vanishingly rare.
What this reminds me of, is the phenomenon in the history of philosophy, where someone thinks they have figured out the system of the world, on which successors will build. But instead what happens is that people recognize a new theme that the innovator has introduced, and build their own rival systems incorporating that new theme.
For example, Kant (responding to Humean skepticism) built his system of transcendental idealism, which was supposed to be a new foundation for philosophy in general. Instead, it inaugurated the era of “German Idealism”, which included Hegel’s absolute idealism, whatever Schelling and Fichte were up to, and even Schopenhauer’s pessimism (which in turn was a source of Nietzsche’s optimism).
Another example would be the different directions that psychoanalysis took after Freud; and I’m sure there are many other examples… I should note that in addition to the rebellious intellectual offspring, there were people who built on Kant and Freud, and who called themselves (neo)Kantians and Freudians.
The closest thing to an important technical successor to Eliezer that I can think of, is Paul Christiano, co-inventor of RLHF, a central alignment technique behind the birth of ChatGPT. Many other people must have found their way to AI safety because of his works, and specific ideas of his have currency (e.g. Jan Leike, formerly of OpenAI superalignment, now at Anthropic, seems to be inspired by Coherent Extrapolated Volition). He is surely a godfather of AI safety, just as Hinton, Bengio, and LeCun were dubbed godfathers of deep learning. But the field itself is not dominated by his particular visions.
What about Nate soares?
I think Eliezer underestimates other people because he evaluates them substantially based on how much they agree with him, and, as a consequence of him having a variety of dumb takes, smart people usually disagree with him about a bunch of stuff.
I wonder how Eliezer would describe his “moat”, i.e., what cognitive trait or combination of traits does he have, that is rarest or hardest to cultivate in others? (Would also be interested in anyone else’s take on this.)
Stanislaw Lem once wrote (I translated it from Russian, because I can’t find another source in the open):
Strugatsky brothers were Quasi-Lems.
I think both of those are true, but it does not follow that history is made of individuals solving individual math problems and pushing out papers which get stacked into the intellectual tower of babel. History as far as I can see is made out of systems or ensembles of people moving around in different configurations.
Yudkowsky couldn’t do what he did without ET Jaynes, who in turn relied on the progenitors of probability and rationality including Thomas Bayes and William of Ockham. But he was also influenced “sideways” by the people who he learned from and defined himself against, the people in SL4 and the people he called idiots and the venture capitalists he once idolised for their competence and Peter Thiel and Demis Hassabis and his family. They shape (at the very least) his emotional worldview, which then shapes how he takes in information and integrates it at a deep and fundamental level. This is true insofar as it is true for any human who lives in a society. When I write anything I can feel the hands of writers past and present shaping my action space. They shape both what I write about and how I choose to right.
So yes if he was gone everything would be different. But it would also be the same, people would love and fight and struggle and cooperate. The sameness of trends manifests at a higher level of coarsegraining, the level where the systemic forces and the long dreams and molochian demons live. And none of this diminishes what he did, does, will do, or could have done. It’s just the way things are, because we can’t run randomised control trials on society.
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who’ve influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
and then