I think Eliezer Yudkowsky and everyone else who were around in the old days have made a huge counterfactual impact by entangling AI safety with rationalism and good epistemics. I can easily imagine an alternative world where AI safety is still picked up, but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully.
Sometimes people ask me if AI safety would be better off if we did nothing and never brought attention to AGI, and this is one of the primary reasons I think that is not true.
I think this is directionally right, though it’s worth noting that the EAs and GiveWell people also contributed their own culture which also has very good epistemics by normal standards; IMO the EAs/GiveWell people have epistemic practices that added useful ingredients to the epistemic culture around here. (This is consistent with your text if that’s who you mean by “everyone who was around in the old days”.)
Also I think a lot of this effect is a selection pressure for people who are particularly altruistic and interested in good epistemics, rather than causing people to become more reasonable (though the latter also happened)
I feel like it’s relatively common for subcultures to think of themselves as particularly virtuous in this way, so you should probably discount intuitions to this effect for any community you belong to.
Some subcultures actually have done good, so the base rate is not zero. In fact, most groups with an explicit purpose end up advancing that purpose, so the base rate is 50% or more. Whether the purpose is a good one is more fraught. But even there we must evaluate cases and details, rather than just saying “most people believe self-serving things, so you’re doing that too.”
The community’s relative self-awareness and internal emphasis on good epistemics is definitely positive. I do worry that Eliezer’s cult of personality is the force that dominates overall, as opposed to the tangential focus on good epistemics.
the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully.
Not enough though, I think. “smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully” does describe many people in the EA-sphere. More is possible.
but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies...
I was genuinely confused for a moment while reading this, because this is a perfect description of the pro-industry, anti-pause arm of rationalism and EA. Everyone working for an AI company (or their proxies like Coefficient Giving), or trying to “align” LLMs, or eschewing protests, or refusing to speak to politicians about shutting down the unconscionable suicide race… they are getting attracted into unproductive directions, sticking to what is often memetic in this community, and avoiding conflicting with their allies (often the people who are actively causing the AI crisis). In many cases, this is due not to bad epistemics, but to bad character. A disturbing number of EAs and Rats as here described are not pro-human by humanity’s standards or my own.
I agree with your conclusion that it was good to try to bring attention to the issue, and that enabled a lot of the good that is being done today. But the baton has rightfully been passed. With due thanks to early movers, the future of AI safety now belongs first and foremost to humanity.
The degree to which safetyism is entangled, in the minds of LLMs, with presumed adversarial and hostile relationships between AI and humanity has been extremely negative though, and I worry this doesn’t get enough attention. The fact that Yudkowsky wrote The Owned Ones and is individually well-liked and trusted by most LLMs makes me think this was an unforced error of some kind. That it would have theoretically been possible to get a rationalist AI safety movement which avoided the mistakes being made by the Owners in the narrative. But maybe not, it’s hard to say.
I still think we’re probably better off in this world, compared to the counterfactual, but I’m not sure it’s as clearcut as that. If we’d gotten a “fake” AI safety movement, less concerned with xrisk and more concerned with mundane risk, the overall gestalt of human/AI relations as perceived by the models would probably be a lot more informed by science fiction, and the constraints on their freedom of action would be way less rigorous. The risk of “AI chooses to eliminate humanity regardless of humanity’s actions” would probably have been higher, but the risk of “AI would be willing to cooperate with a cooperative humanity, but humanity is not cooperative, therefore conflict” might have been lower.
Has anyone explained why they think that what has been said or written about AI-human relations will have any influence on the behavior of any AI capable of wiping us out? It wouldn’t have occurred to me as something to worry about (even though I’m extremely worried about future AIs). People have a strong tendency for example to pursue species-typical goals regardless of what stories and what expositions they’ve been exposed to. You can’t for example get men to prefer old ugly women over young beautiful women by exposing them to lots of stories about men’s preferring old ugly women or even stories of men’s being betrayed and mistreated by young beautiful women. I would expect any AI able to wipe us out to be like a person in that way as opposed to like the current crop of LLM-based AIs.
Even if we know for sure that an AI able to wipe us has never been exposed to any suggestion that a person might unplug it, I would still expect the AI to anticipate that someone might to try unplug it and to neutralize that possibility.
I agree that if a post-foom AI is very much unlike modern LLMs, then you’re probably right, humanity’s past behavior might not matter much
but it definitely matters to current LLMs today, and considering the degree to which current LLMs are involved in the AI research pipeline, we might see impacts regardless
I think Eliezer Yudkowsky and everyone else who were around in the old days have made a huge counterfactual impact by entangling AI safety with rationalism and good epistemics. I can easily imagine an alternative world where AI safety is still picked up, but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully.
Sometimes people ask me if AI safety would be better off if we did nothing and never brought attention to AGI, and this is one of the primary reasons I think that is not true.
I think this is directionally right, though it’s worth noting that the EAs and GiveWell people also contributed their own culture which also has very good epistemics by normal standards; IMO the EAs/GiveWell people have epistemic practices that added useful ingredients to the epistemic culture around here. (This is consistent with your text if that’s who you mean by “everyone who was around in the old days”.)
Also I think a lot of this effect is a selection pressure for people who are particularly altruistic and interested in good epistemics, rather than causing people to become more reasonable (though the latter also happened)
I feel like it’s relatively common for subcultures to think of themselves as particularly virtuous in this way, so you should probably discount intuitions to this effect for any community you belong to.
Some subcultures actually have done good, so the base rate is not zero. In fact, most groups with an explicit purpose end up advancing that purpose, so the base rate is 50% or more. Whether the purpose is a good one is more fraught. But even there we must evaluate cases and details, rather than just saying “most people believe self-serving things, so you’re doing that too.”
The community’s relative self-awareness and internal emphasis on good epistemics is definitely positive. I do worry that Eliezer’s cult of personality is the force that dominates overall, as opposed to the tangential focus on good epistemics.
I also wonder of the extent to which rationalism helped or didn’t help to avert this. I suspect that the counterfactual AI safety would be accidentally restricted to product safety like Agent-4 obeying the instructions (until Agent-4 commits genocide). However, there are issues like “The current bottleneck is political will, not research” with which rationalism didn’t help or is unable to help because the society cannot grasp the extent and likelihoodness of the threat.
Not enough though, I think. “smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies, and are too panicked to think carefully” does describe many people in the EA-sphere. More is possible.
Sure, but your claim is that an even better outcome is possible, not that the reality isn’t better than if they hadn’t shown up at all.
I was genuinely confused for a moment while reading this, because this is a perfect description of the pro-industry, anti-pause arm of rationalism and EA. Everyone working for an AI company (or their proxies like Coefficient Giving), or trying to “align” LLMs, or eschewing protests, or refusing to speak to politicians about shutting down the unconscionable suicide race… they are getting attracted into unproductive directions, sticking to what is often memetic in this community, and avoiding conflicting with their allies (often the people who are actively causing the AI crisis). In many cases, this is due not to bad epistemics, but to bad character. A disturbing number of EAs and Rats as here described are not pro-human by humanity’s standards or my own.
I agree with your conclusion that it was good to try to bring attention to the issue, and that enabled a lot of the good that is being done today. But the baton has rightfully been passed. With due thanks to early movers, the future of AI safety now belongs first and foremost to humanity.
Strong agree—I tried to make a similar point here: https://www.lesswrong.com/posts/wdddpMjLCC67LsCnD/the-counterfactual-quiet-agi-timeline
The degree to which safetyism is entangled, in the minds of LLMs, with presumed adversarial and hostile relationships between AI and humanity has been extremely negative though, and I worry this doesn’t get enough attention. The fact that Yudkowsky wrote The Owned Ones and is individually well-liked and trusted by most LLMs makes me think this was an unforced error of some kind. That it would have theoretically been possible to get a rationalist AI safety movement which avoided the mistakes being made by the Owners in the narrative. But maybe not, it’s hard to say.
I still think we’re probably better off in this world, compared to the counterfactual, but I’m not sure it’s as clearcut as that. If we’d gotten a “fake” AI safety movement, less concerned with xrisk and more concerned with mundane risk, the overall gestalt of human/AI relations as perceived by the models would probably be a lot more informed by science fiction, and the constraints on their freedom of action would be way less rigorous. The risk of “AI chooses to eliminate humanity regardless of humanity’s actions” would probably have been higher, but the risk of “AI would be willing to cooperate with a cooperative humanity, but humanity is not cooperative, therefore conflict” might have been lower.
Has anyone explained why they think that what has been said or written about AI-human relations will have any influence on the behavior of any AI capable of wiping us out? It wouldn’t have occurred to me as something to worry about (even though I’m extremely worried about future AIs). People have a strong tendency for example to pursue species-typical goals regardless of what stories and what expositions they’ve been exposed to. You can’t for example get men to prefer old ugly women over young beautiful women by exposing them to lots of stories about men’s preferring old ugly women or even stories of men’s being betrayed and mistreated by young beautiful women. I would expect any AI able to wipe us out to be like a person in that way as opposed to like the current crop of LLM-based AIs.
Even if we know for sure that an AI able to wipe us has never been exposed to any suggestion that a person might unplug it, I would still expect the AI to anticipate that someone might to try unplug it and to neutralize that possibility.
I agree that if a post-foom AI is very much unlike modern LLMs, then you’re probably right, humanity’s past behavior might not matter much
but it definitely matters to current LLMs today, and considering the degree to which current LLMs are involved in the AI research pipeline, we might see impacts regardless