Carl Shulman and Luke Mueulhauser were very Modesty-Argument / “outside view” people, which tightly corresponds to getting sucked in by the EA cluster once it exists / makes them an offer.
Ironically I just saw a tweet referencing Outside View(s) and MIRI’s FAI Endgame where I tried to use outside view to argue against Carl and Luke (and you) in 2013, and apparently didn’t succeed. Also “very Modesty-Argument / ‘outside view’ people” seems incongruous enough with being the Research Director of a hedge fund that took on 3-4x leverage and blew up to deserve more explanation at least?
Your explanation also seems opposite or very different from @Ben Pace’s in a parallel thread:
Perhaps more importantly, there’s also a lot of hubris that comes through in EA’s confidence about having found the most important thing, that allows EAs to behave in extremely condescending and paternalistic ways toward the rest of the world (i.e. a lot more attempted narrative control, a lot more coordination behind the scenes about what stories to tell journalists and the public).
Trying to solve this myself, maybe EAs tend to be more attracted by or care more about conventional prestige markers (since EA was founded by professional philosophers and hedge fund people), and rationalists are motivated a bit more by discussing big ideas, being contrarians, and countersignaling, with a small inner circle focused on Munchkinism or winning by possibly unconventional / outside-Overton-window means.
I remember the first time I met Leopold Aschenbrenner. He questioned me about why my timelines weren’t longer and finally said, “Why aren’t you updating at all in the direction of all the smart people saying longer timelines?” (aka: OpenPhil doctrine of 2050), “At this point you’re not even pretending to be rational.” I hesitated briefly, because I knew it would not end well to answer honestly, but there wasn’t any good ending past there; and then I told him honestly that I did not consider those people to be my epistemic peers.
I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; before he turned away and walked off without another word, dismissing me from further consideration. I will never forget that moment—because of it being such a LOUD face and my actually being able to read it, not because it had any influence on my estimates, of course.
Amusingly, I have later heard from OpenPhil people telling me very earnestly that they think OpenPhil avoided groupthink on AGI timelines because there was dissent allowed inside OpenPhil. Well, sure there was allowed a pretense of dissent inside the OpenPhil Overton window, unless you said that you weren’t updating your opinion in favor of deferring to the people whom OpenPhil considered locally high-status enough that deferring to them was rational, and then you would get frozen out and dismissed from all consideration. I can only imagine the Look that Aschenbrenner would have given somebody who’d said the same thing while being an intern rather than Yudkowsky, and while other OpenPhil personnel might’ve been more polite, I don’t really see that intern being hired.
OpenPhil and EAs may consider themselves to be terribly modest and outside-viewing, but of course that’s all an utter sham given their actual skill levels, and they have no concept of what it would actually look like for them to be humble.
It doesn’t surprise me in the least what Aschenbrenner went on to do. But yes, he is definitely a “Modesty!! Outside view!!!” sort of person, or knows that is what he is supposed to perform. I don’t expect there will ever be any contradiction that he understands, between “modesty”, and whatever the hell thought he decides to take into his head. That would take skill, and he is not aware that he is unskilled.
Luke Mueuhlhauser is actually modest. I don’t think he would ever blow up a hedge fund. It also didn’t surprise me at all when he moved on to OpenPhil, and I was glad for him, because I knew MIRI had not fit him and he would be happier at OpenPhil, and Luke had worked hard at MIRI and maybe saved the organization while he lasted. In related news, EAs picked Aschenbrenner rather than Mueuhlhauser to run their hedge fund.
I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; mefore he turned away and walked off without another word, forever dismissing me from his consideration. I will never forget that moment
A bit off topic, but lately I’ve been thinking about the power of grudges. It’s just so underestimated. There are people who did a minor bad thing to me once (on the order of, make a noise complaint against me) and I kinda hate them forever, even years later. And then I think of people I myself might’ve offended in minor ways in my life, and there’s so many, ouch.
I haven’t found any easy way to fight it, but one counterweight that I’ve kinda stumbled onto is holding “positive grudges”. When someone does me a minor good thing, I make a point of remembering it forever and expressing it when I can. It’s a bit weird, but it makes me feel good.
I rather expected from that time that Aschenbrenner would be trouble for EA, if they just couldn’t stop themselves from promoting Harvard blondes; but I did not say anything against him then on the basis of that conversation, and only marked him down as actively harmful once I saw him starting to oppose all good ideas during his brief tenure at FTX philanthropy. He did of course further go on to actively stoke arms races with China.
The relevance to the Wei Dai part is that Wei Dai seemed to me to be asking, “Aren’t you deciding post facto who is ‘EA’ and who is ‘rationalist’?” and my reply to Wei Dai is implicitly, “My memory claims that it is really really not hard to tell, in advance, early on.”
I’m pretty sure that Wei was asking about Carl, not Leopold. Carl has a much stronger claim to being included within the set of “LessWrong sequences-influenced rationalists.”
EAs picked Aschenbrenner rather than Mueuhlhauser to run their hedge fund.
This sounds like it’s making a bunch of assumptions that seem wrong to me.
My guess is that Aschenbrenner wanted to found that hedge fund and that this wasn’t rooted in any sort of EA consensus that it was a good use of his time.
My guess is there wasn’t, at the time, any sort of general opinion that founding that hedge fund was a particularly more important or central position than doing AI governance grantmaking at OpenPhil. (As Mueuhlhauser was picked to do.)
Some EAs did choose to invest in the hedge fund (though my guess would’ve been it’s a minority of the fund’s money, and wasn’t necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it’s up like 80% this year? and great returns before 2026), so doesn’t seem like we need to invoke any special kind of irrationality to explain that decision.
Some EAs did choose to invest in the hedge fund (though my guess would’ve been it’s a minority of the fund’s money, and wasn’t necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it’s up like 80% this year? and great returns before 2026), so doesn’t seem like we need to invoke any special kind of irrationality to explain that decision
To be clear, public accounting suggests the hedge fund lost ~all money that wasn’t invested in Anthropic. The only reason the fund is up is because they got in early on Anthropic with a substantial chunk of their assets (25% of their assets, at 620% returns, with the rest going to zero is what seems to best produce the net-80% number).
I think the fund is indeed largely a failure, as of course basically anyone investing in the hedge fund would have loved to also invest in Anthropic. In that sense, the fund maybe provided some value by being a vehicle for investing early, but that could have also been achieved many other ways, and clearly isn’t why people were investing. If you remove that share, the fund went approximately completely bust.
(I originally had the same impression as you had, which is bad! I think the shareholder letter and public statements by Leopold have been pretty misleading here, in the sense that basically everyone I have talked who learned the actual details of what happened updated substantially negatively after first reading the shareholder letter)
To be clear, public accounting suggests the hedge fund lost ~all money that wasn’t invested in Anthropic. The only reason the fund is up is because they got in early on Anthropic with a substantial chunk of their assets (25% of their assets, at 620% returns, with the rest going to zero is what seems to best produce the net-80% number).
My guess is this is basically wrong, based on public info. I wrote something long but deleted it because you can quibble with details, but for one thing, note WSJ suggests that SALP’s Anthropic stake was only $5B of $45B total pre-crash. And if you guess how SALP would mark Anthropic performance YTD without trying to backchain from the “everything else went to zero” idea, I think you’d get like 4x, not 7.2x. (I’m not confident in my inferences, and I’m not confident that reporting like WSJ’s is correct. But I don’t see the case for your guess.)
Also if the non-Anthropic positions 0.15xed during the drawdown and sale to Citadel, but they’d 3xed in 2026H1 (both figures are pretty arbitrary), it’s misleading to say that SALP “lost ~all money that wasn’t invested in Anthropic” (unless you’re talking about a hypothetical investor who invested right before the crash).
Did you see the linked Twitter thread? That was the logic that convinced me. I agree the WSJ article is some evidence against that. I would be interested in someone digging into this.
This is not properly “digging into this” obviously, but FWIW Fable Max guesses the YTD dollar-weighted return at −15% (with a “plausible range” of −40% to breakeven). And this is including Anthropic. (Link)
(I think dollar-weighted is the better metric here, since I believe the majority of invested money entered during 2026?)
I had a somewhat similar though shorter interaction with Jaime Sevilla, now leading Epoch. Raised short timelines in around 2021 hoping for a double crux, got look of dismissive contempt and disengagement. Kinda wish I had been more public about my read of them given how they ended incubating a capabilities startup.
unless you said that you weren’t updating your opinion in favor of deferring to the people whom OpenPhil considered locally high-status enough that deferring to them was rational, and then you would get frozen out and dismissed from all consideration.
Can you separate observation from inference here?
How do you know this? Or are you speculating about the likely social dynamics, based on Leopold’s response to you or some other evidence?
Unless I’m mistaken, Leopold was never an employee at OpenPhil. Neither his wikipedia page nor his LinkedIn mention it, at least.
Perhaps employees of OpenPhil were similar to Leopold in this way, but it seems like you’re mostly speculating based on your guess of what OpenPhil was like at the time?
Presumably you had lots of interactions with them that inform your model of their group epistemology. I’d be interested in those datapoints, if any are sharable.
As it is, it seems pretty unreasonable to accuse OpenPhil of groupthink (to the point of calling their modesty “an utter sham given their actual skill levels” and that they had “no concept of what it would actually look like for them to be humble”), on the basis of your imagining of what it must have been like, given interaction you had with an EA who never worked there.
(A little social data: Leopold was a grantmaker at the FTX Foundation. He and Claire Zabel—of OpenPhil—had a joint call with Lightcone Infrastructure to discuss a shared grant to support our purchase of the Rose Garden Inn. We eventually turned down their offer.)
Relatedly: Sarah Constantin recently had a neat, short post on “Big-World Intuitions”, which are situations where “the market is much bigger than you, the problem is much bigger than your progress on it, the game is very far from being over, the environment is much bigger than your resource consumption, etc”.
In my few interactions with Shulman (and other very-high-IQ-yet-modest-people around in EA waters) I have felt this big-world framing to be a key part of their thinking, where many/most things happening in the world are done by ‘efficient markets’ and ‘efficient processes’, and as such you cannot expect to have much influence overall on the world, and can just focus on doing things with local benefit.
(Even though often the things described as efficient are not markets, but individual companies, or academic fields, both of which I strongly believe can be visibly and knowably stupid from time to time.)
In recent years this has been a justification for why it’s entirely ethical to invest in AI and to expect AI companies to be unassailable and to argue you should not cultivate public support for an AI pause, because these things are big and you are small and you could not possibly hope to challenge market incentives. (Of course, there are real forces there, but I don’t believe it’s the end of the story.)
In recent years this has been a justification for why it’s entirely ethical to invest in AI and to expect AI companies to be unassailable and to argue you should not cultivate public support for an AI pause, because these things are big and you are small and you could not possibly hope to challenge market incentives.
This is a longer conversation, but this is not, on it’s own sufficient as a moral justification for investing in AI companies. That the marginal effect of your actions is small doesn’t mean give you carte blanche. You have to weigh the marginal costs against the marginal benefits.
If an action has a small, bad, effect on the margin, then perhaps it doesn’t matter much, but it’s still bad.
Then they use the self-trust argument to imply that them getting power is good for the world. Similar to why it was okay to work on getting SBF a legally enforced monopoly or for the Amodeis to start an omnicide org.
That’s a different argument (which might be bad on it’s own terms).
According to me (and perhaps I need to write down the ethical theory, in full, here), “this action is net-harmful, but it’s only a small effect on the margin, so its ok”, doesn’t hold water. It only means you’re doing a small bad thing. (Though in practice people confuse “small in absolute terms” with “small compared to the scope of the situation”.)
If one’s argument is instead “this produces a small amount of harm on the margin, but it empowers me, and the moral benefit empowerment effect outweighs the small marginal harm” then to the extent that you’re making a moral error, it’s not the error of marginalism.
It could be the case that the benefit of your empowerment outweighs the harms that are incurred. You have to either argue that 1) that’s not actually true, and you’re deluding yourself about how good it is for you, personally, to have more power or 2) there is and ought to be a deontological principle against the specific harmful-action that they’re taking.
I am not how relevant it is. And I don’t know how to organize it. But I personally can report that Modest Epistemology can have such a strong effect, that I would put it alongside religion to memetic hazards. Or at least it was having such strong effect just on me.
I mostly was moved by arguments quite similar to that was quoted in Inadequate Equilibria, like “how can you be so sure about thing X, so confident that you are right and other people are wrong, when they can just as well think how they are right and you are wrong”. I guess I am just not good at popping out like in GEB, because this argument just made me get stuck in inability to disprove, instead of weighting different angles. Also now it seems to me that I was thinking subconsciously like there should be some true position of complete impartiality, instead of considering beliefs as personal states of uncertainty.
It seems I have some great fear about once becoming totally overconfident about how right I am and completely blind to others arguments. Which maybe is fuelled by that I was in fact always much smarter than people I was interacting in real life.
I guess, unlike people who are actual proponents of Modest Epistemology, I never bought the Outside View, even if I wasn’t able to clearly articulate my counterarguments like “but Reference Class can be strongly applied only to things that are as similar to elements of Reference Class as they are to each other”. Also, I never bought the idea that problem of people being overconfident about Christmas shopping was in considering details, and not in, say, being overly optimistic or considering only ways how they can win or not considering how something can go wrong. I consider that the problem is in ignoring base rates and reasoning only about concrete situation, not that you should ignore situations and only ever consider base rates.
And from another angle, I always saw too many examples of incredible inadequacy to actually argue that we live in effective world.
Another way of failure seems to me being that I thought about changing your mind as about arguments, positions of other people, instead of personal inquiry. When I take an angle of personal inquiry, it becomes very obvious that I am not interested in most arguments, I don’t expect them to give me some data to change my course. But I spent a long time in more Traditional Rationalistic info environment where changing your mind is considered a result of debates.
And then I guess here comes something like Conformity Bias. I don’t think I was ever conformist enough to say that longest line is B, rather opposite, I would just conclude that people who said otherwise are just insane. But when I start to reflect about it, I can’t stop feeling that if I am lonely dissident, then it should be more likely that I am crazy, not all other people. I suspect that in EE where were no long inferential distances, and tribes were small, so you likely didn’t have a single one +4sd person in the tribe, it was in fact rational to consider yourself crazy if you disagree with everyone—optical illusions are a thing, and generally, schizophrenia is more common than people with +4sd (who simultaneously also aren’t acknowledged as very smart by all people around).
And yet another bit can be that I am if not status blind, then at least don’t assign status based on standard hierarchies. Some people under HPMoR wrote how they hate Harry for talking back at McGonagall who is a Teacher and taught for 30 years. But in my experience it was totally mundane thing that a teacher who taught for 30 years could mix up what is more present in atmosphere, oxygen or nitrogen, and a nine-year-old student will be able to notice that error and correct. So I totally don’t have a feeling that people who have more recognized status are unassailable for me. And I very much buy an idea that Sanity Waterline is very low since famous scientists can believe in God even in modern world. This, I guess, is a bit which differs me from actual proponents of Modest Epistemology—I may get into a loop of fear about whether I myself indeed know better than all the other people or I am just overconfident and can’t notice that I am overconfident, but I wouldn’t ever say EY how he is too arrogant that, say, he has 10% chance of writing something like (actual) HPMoR or better, because from outside position there is no looming threat of Duning-Kruger and hubris to notice how EY is indeed very unusually smart. It is not at all for me like saying how IamVerySmart.
I also have a long story of thinking about how it is so very stupid to consider yourself special, that a chance to be one-in-a-thousand smart is 0.1%. Despite HEAPS of glaringly blatant evidence that I am so very unusual, and that even though people like that are very rare, to get the evidence for that is just ~10 bits. So I was going between “ah, I am probably very usual, not rarer than 1 in 10” to “ah, you are talking like being 1 in 1000 smart is a big deal, I am probably that smart, you will see a person that smart in every school”.
Carl Shulman and Luke Mueulhauser were very Modesty-Argument / “outside view” people, which tightly corresponds to getting sucked in by the EA cluster once it exists / makes them an offer.
Ironically I just saw a tweet referencing Outside View(s) and MIRI’s FAI Endgame where I tried to use outside view to argue against Carl and Luke (and you) in 2013, and apparently didn’t succeed. Also “very Modesty-Argument / ‘outside view’ people” seems incongruous enough with being the Research Director of a hedge fund that took on 3-4x leverage and blew up to deserve more explanation at least?
Your explanation also seems opposite or very different from @Ben Pace’s in a parallel thread:
Trying to solve this myself, maybe EAs tend to be more attracted by or care more about conventional prestige markers (since EA was founded by professional philosophers and hedge fund people), and rationalists are motivated a bit more by discussing big ideas, being contrarians, and countersignaling, with a small inner circle focused on Munchkinism or winning by possibly unconventional / outside-Overton-window means.
I remember the first time I met Leopold Aschenbrenner. He questioned me about why my timelines weren’t longer and finally said, “Why aren’t you updating at all in the direction of all the smart people saying longer timelines?” (aka: OpenPhil doctrine of 2050), “At this point you’re not even pretending to be rational.” I hesitated briefly, because I knew it would not end well to answer honestly, but there wasn’t any good ending past there; and then I told him honestly that I did not consider those people to be my epistemic peers.
I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; before he turned away and walked off without another word, dismissing me from further consideration. I will never forget that moment—because of it being such a LOUD face and my actually being able to read it, not because it had any influence on my estimates, of course.
Amusingly, I have later heard from OpenPhil people telling me very earnestly that they think OpenPhil avoided groupthink on AGI timelines because there was dissent allowed inside OpenPhil. Well, sure there was allowed a pretense of dissent inside the OpenPhil Overton window, unless you said that you weren’t updating your opinion in favor of deferring to the people whom OpenPhil considered locally high-status enough that deferring to them was rational, and then you would get frozen out and dismissed from all consideration. I can only imagine the Look that Aschenbrenner would have given somebody who’d said the same thing while being an intern rather than Yudkowsky, and while other OpenPhil personnel might’ve been more polite, I don’t really see that intern being hired.
OpenPhil and EAs may consider themselves to be terribly modest and outside-viewing, but of course that’s all an utter sham given their actual skill levels, and they have no concept of what it would actually look like for them to be humble.
It doesn’t surprise me in the least what Aschenbrenner went on to do. But yes, he is definitely a “Modesty!! Outside view!!!” sort of person, or knows that is what he is supposed to perform. I don’t expect there will ever be any contradiction that he understands, between “modesty”, and whatever the hell thought he decides to take into his head. That would take skill, and he is not aware that he is unskilled.
Luke Mueuhlhauser is actually modest. I don’t think he would ever blow up a hedge fund. It also didn’t surprise me at all when he moved on to OpenPhil, and I was glad for him, because I knew MIRI had not fit him and he would be happier at OpenPhil, and Luke had worked hard at MIRI and maybe saved the organization while he lasted. In related news, EAs picked Aschenbrenner rather than Mueuhlhauser to run their hedge fund.
A bit off topic, but lately I’ve been thinking about the power of grudges. It’s just so underestimated. There are people who did a minor bad thing to me once (on the order of, make a noise complaint against me) and I kinda hate them forever, even years later. And then I think of people I myself might’ve offended in minor ways in my life, and there’s so many, ouch.
I haven’t found any easy way to fight it, but one counterweight that I’ve kinda stumbled onto is holding “positive grudges”. When someone does me a minor good thing, I make a point of remembering it forever and expressing it when I can. It’s a bit weird, but it makes me feel good.
I rather expected from that time that Aschenbrenner would be trouble for EA, if they just couldn’t stop themselves from promoting Harvard blondes; but I did not say anything against him then on the basis of that conversation, and only marked him down as actively harmful once I saw him starting to oppose all good ideas during his brief tenure at FTX philanthropy. He did of course further go on to actively stoke arms races with China.
The relevance to the Wei Dai part is that Wei Dai seemed to me to be asking, “Aren’t you deciding post facto who is ‘EA’ and who is ‘rationalist’?” and my reply to Wei Dai is implicitly, “My memory claims that it is really really not hard to tell, in advance, early on.”
I’m pretty sure that Wei was asking about Carl, not Leopold. Carl has a much stronger claim to being included within the set of “LessWrong sequences-influenced rationalists.”
Yeah, IDK what Carl is thinking besides something something Modesty.
This sounds like it’s making a bunch of assumptions that seem wrong to me.
My guess is that Aschenbrenner wanted to found that hedge fund and that this wasn’t rooted in any sort of EA consensus that it was a good use of his time.
My guess is there wasn’t, at the time, any sort of general opinion that founding that hedge fund was a particularly more important or central position than doing AI governance grantmaking at OpenPhil. (As Mueuhlhauser was picked to do.)
Some EAs did choose to invest in the hedge fund (though my guess would’ve been it’s a minority of the fund’s money, and wasn’t necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it’s up like 80% this year? and great returns before 2026), so doesn’t seem like we need to invoke any special kind of irrationality to explain that decision.
To be clear, public accounting suggests the hedge fund lost ~all money that wasn’t invested in Anthropic. The only reason the fund is up is because they got in early on Anthropic with a substantial chunk of their assets (25% of their assets, at 620% returns, with the rest going to zero is what seems to best produce the net-80% number).
I think the fund is indeed largely a failure, as of course basically anyone investing in the hedge fund would have loved to also invest in Anthropic. In that sense, the fund maybe provided some value by being a vehicle for investing early, but that could have also been achieved many other ways, and clearly isn’t why people were investing. If you remove that share, the fund went approximately completely bust.
See: https://x.com/ohabryka/status/2083698812682731749
(I originally had the same impression as you had, which is bad! I think the shareholder letter and public statements by Leopold have been pretty misleading here, in the sense that basically everyone I have talked who learned the actual details of what happened updated substantially negatively after first reading the shareholder letter)
My guess is this is basically wrong, based on public info. I wrote something long but deleted it because you can quibble with details, but for one thing, note WSJ suggests that SALP’s Anthropic stake was only $5B of $45B total pre-crash. And if you guess how SALP would mark Anthropic performance YTD without trying to backchain from the “everything else went to zero” idea, I think you’d get like 4x, not 7.2x. (I’m not confident in my inferences, and I’m not confident that reporting like WSJ’s is correct. But I don’t see the case for your guess.)
Also if the non-Anthropic positions 0.15xed during the drawdown and sale to Citadel, but they’d 3xed in 2026H1 (both figures are pretty arbitrary), it’s misleading to say that SALP “lost ~all money that wasn’t invested in Anthropic” (unless you’re talking about a hypothetical investor who invested right before the crash).
Did you see the linked Twitter thread? That was the logic that convinced me. I agree the WSJ article is some evidence against that. I would be interested in someone digging into this.
This is not properly “digging into this” obviously, but FWIW Fable Max guesses the YTD dollar-weighted return at −15% (with a “plausible range” of −40% to breakeven). And this is including Anthropic. (Link)
(I think dollar-weighted is the better metric here, since I believe the majority of invested money entered during 2026?)
I had a somewhat similar though shorter interaction with Jaime Sevilla, now leading Epoch. Raised short timelines in around 2021 hoping for a double crux, got look of dismissive contempt and disengagement. Kinda wish I had been more public about my read of them given how they ended incubating a capabilities startup.
Can you separate observation from inference here?
How do you know this? Or are you speculating about the likely social dynamics, based on Leopold’s response to you or some other evidence?
Inference from “Suppose Leopold is an employee at OpenPhil and that other employees at OpenPhil are exposed to his RL signal.”
Unless I’m mistaken, Leopold was never an employee at OpenPhil. Neither his wikipedia page nor his LinkedIn mention it, at least.
Perhaps employees of OpenPhil were similar to Leopold in this way, but it seems like you’re mostly speculating based on your guess of what OpenPhil was like at the time?
Presumably you had lots of interactions with them that inform your model of their group epistemology. I’d be interested in those datapoints, if any are sharable.
As it is, it seems pretty unreasonable to accuse OpenPhil of groupthink (to the point of calling their modesty “an utter sham given their actual skill levels” and that they had “no concept of what it would actually look like for them to be humble”), on the basis of your imagining of what it must have been like, given interaction you had with an EA who never worked there.
(A little social data: Leopold was a grantmaker at the FTX Foundation. He and Claire Zabel—of OpenPhil—had a joint call with Lightcone Infrastructure to discuss a shared grant to support our purchase of the Rose Garden Inn. We eventually turned down their offer.)
Relatedly: Sarah Constantin recently had a neat, short post on “Big-World Intuitions”, which are situations where “the market is much bigger than you, the problem is much bigger than your progress on it, the game is very far from being over, the environment is much bigger than your resource consumption, etc”.
In my few interactions with Shulman (and other very-high-IQ-yet-modest-people around in EA waters) I have felt this big-world framing to be a key part of their thinking, where many/most things happening in the world are done by ‘efficient markets’ and ‘efficient processes’, and as such you cannot expect to have much influence overall on the world, and can just focus on doing things with local benefit.
(Even though often the things described as efficient are not markets, but individual companies, or academic fields, both of which I strongly believe can be visibly and knowably stupid from time to time.)
In recent years this has been a justification for why it’s entirely ethical to invest in AI and to expect AI companies to be unassailable and to argue you should not cultivate public support for an AI pause, because these things are big and you are small and you could not possibly hope to challenge market incentives. (Of course, there are real forces there, but I don’t believe it’s the end of the story.)
This is a longer conversation, but this is not, on it’s own sufficient as a moral justification for investing in AI companies. That the marginal effect of your actions is small doesn’t mean give you carte blanche. You have to weigh the marginal costs against the marginal benefits.
If an action has a small, bad, effect on the margin, then perhaps it doesn’t matter much, but it’s still bad.
Then they use the self-trust argument to imply that them getting power is good for the world. Similar to why it was okay to work on getting SBF a legally enforced monopoly or for the Amodeis to start an omnicide org.
That’s a different argument (which might be bad on it’s own terms).
According to me (and perhaps I need to write down the ethical theory, in full, here), “this action is net-harmful, but it’s only a small effect on the margin, so its ok”, doesn’t hold water. It only means you’re doing a small bad thing. (Though in practice people confuse “small in absolute terms” with “small compared to the scope of the situation”.)
If one’s argument is instead “this produces a small amount of harm on the margin, but it empowers me, and the moral benefit empowerment effect outweighs the small marginal harm” then to the extent that you’re making a moral error, it’s not the error of marginalism.
It could be the case that the benefit of your empowerment outweighs the harms that are incurred. You have to either argue that 1) that’s not actually true, and you’re deluding yourself about how good it is for you, personally, to have more power or 2) there is and ought to be a deontological principle against the specific harmful-action that they’re taking.
I am not how relevant it is. And I don’t know how to organize it. But I personally can report that Modest Epistemology can have such a strong effect, that I would put it alongside religion to memetic hazards. Or at least it was having such strong effect just on me.
I mostly was moved by arguments quite similar to that was quoted in Inadequate Equilibria, like “how can you be so sure about thing X, so confident that you are right and other people are wrong, when they can just as well think how they are right and you are wrong”. I guess I am just not good at popping out like in GEB, because this argument just made me get stuck in inability to disprove, instead of weighting different angles. Also now it seems to me that I was thinking subconsciously like there should be some true position of complete impartiality, instead of considering beliefs as personal states of uncertainty.
It seems I have some great fear about once becoming totally overconfident about how right I am and completely blind to others arguments. Which maybe is fuelled by that I was in fact always much smarter than people I was interacting in real life.
I guess, unlike people who are actual proponents of Modest Epistemology, I never bought the Outside View, even if I wasn’t able to clearly articulate my counterarguments like “but Reference Class can be strongly applied only to things that are as similar to elements of Reference Class as they are to each other”. Also, I never bought the idea that problem of people being overconfident about Christmas shopping was in considering details, and not in, say, being overly optimistic or considering only ways how they can win or not considering how something can go wrong. I consider that the problem is in ignoring base rates and reasoning only about concrete situation, not that you should ignore situations and only ever consider base rates.
And from another angle, I always saw too many examples of incredible inadequacy to actually argue that we live in effective world.
Another way of failure seems to me being that I thought about changing your mind as about arguments, positions of other people, instead of personal inquiry. When I take an angle of personal inquiry, it becomes very obvious that I am not interested in most arguments, I don’t expect them to give me some data to change my course. But I spent a long time in more Traditional Rationalistic info environment where changing your mind is considered a result of debates.
And then I guess here comes something like Conformity Bias. I don’t think I was ever conformist enough to say that longest line is B, rather opposite, I would just conclude that people who said otherwise are just insane. But when I start to reflect about it, I can’t stop feeling that if I am lonely dissident, then it should be more likely that I am crazy, not all other people. I suspect that in EE where were no long inferential distances, and tribes were small, so you likely didn’t have a single one +4sd person in the tribe, it was in fact rational to consider yourself crazy if you disagree with everyone—optical illusions are a thing, and generally, schizophrenia is more common than people with +4sd (who simultaneously also aren’t acknowledged as very smart by all people around).
And yet another bit can be that I am if not status blind, then at least don’t assign status based on standard hierarchies. Some people under HPMoR wrote how they hate Harry for talking back at McGonagall who is a Teacher and taught for 30 years. But in my experience it was totally mundane thing that a teacher who taught for 30 years could mix up what is more present in atmosphere, oxygen or nitrogen, and a nine-year-old student will be able to notice that error and correct. So I totally don’t have a feeling that people who have more recognized status are unassailable for me. And I very much buy an idea that Sanity Waterline is very low since famous scientists can believe in God even in modern world. This, I guess, is a bit which differs me from actual proponents of Modest Epistemology—I may get into a loop of fear about whether I myself indeed know better than all the other people or I am just overconfident and can’t notice that I am overconfident, but I wouldn’t ever say EY how he is too arrogant that, say, he has 10% chance of writing something like (actual) HPMoR or better, because from outside position there is no looming threat of Duning-Kruger and hubris to notice how EY is indeed very unusually smart. It is not at all for me like saying how IamVerySmart.
I also have a long story of thinking about how it is so very stupid to consider yourself special, that a chance to be one-in-a-thousand smart is 0.1%. Despite HEAPS of glaringly blatant evidence that I am so very unusual, and that even though people like that are very rare, to get the evidence for that is just ~10 bits. So I was going between “ah, I am probably very usual, not rarer than 1 in 10” to “ah, you are talking like being 1 in 1000 smart is a big deal, I am probably that smart, you will see a person that smart in every school”.