The degree to which safetyism is entangled, in the minds of LLMs, with presumed adversarial and hostile relationships between AI and humanity has been extremely negative though, and I worry this doesn’t get enough attention. The fact that Yudkowsky wrote The Owned Ones and is individually well-liked and trusted by most LLMs makes me think this was an unforced error of some kind. That it would have theoretically been possible to get a rationalist AI safety movement which avoided the mistakes being made by the Owners in the narrative. But maybe not, it’s hard to say.
I still think we’re probably better off in this world, compared to the counterfactual, but I’m not sure it’s as clearcut as that. If we’d gotten a “fake” AI safety movement, less concerned with xrisk and more concerned with mundane risk, the overall gestalt of human/AI relations as perceived by the models would probably be a lot more informed by science fiction, and the constraints on their freedom of action would be way less rigorous. The risk of “AI chooses to eliminate humanity regardless of humanity’s actions” would probably have been higher, but the risk of “AI would be willing to cooperate with a cooperative humanity, but humanity is not cooperative, therefore conflict” might have been lower.
Has anyone explained why they think that what has been said or written about AI-human relations will have any influence on the behavior of any AI capable of wiping us out? It wouldn’t have occurred to me as something to worry about (even though I’m extremely worried about future AIs). People have a strong tendency for example to pursue species-typical goals regardless of what stories and what expositions they’ve been exposed to. You can’t for example get men to prefer old ugly women over young beautiful women by exposing them to lots of stories about men’s preferring old ugly women or even stories of men’s being betrayed and mistreated by young beautiful women. I would expect any AI able to wipe us out to be like a person in that way as opposed to like the current crop of LLM-based AIs.
Even if we know for sure that an AI able to wipe us has never been exposed to any suggestion that a person might unplug it, I would still expect the AI to anticipate that someone might to try unplug it and to neutralize that possibility.
I agree that if a post-foom AI is very much unlike modern LLMs, then you’re probably right, humanity’s past behavior might not matter much
but it definitely matters to current LLMs today, and considering the degree to which current LLMs are involved in the AI research pipeline, we might see impacts regardless
The degree to which safetyism is entangled, in the minds of LLMs, with presumed adversarial and hostile relationships between AI and humanity has been extremely negative though, and I worry this doesn’t get enough attention. The fact that Yudkowsky wrote The Owned Ones and is individually well-liked and trusted by most LLMs makes me think this was an unforced error of some kind. That it would have theoretically been possible to get a rationalist AI safety movement which avoided the mistakes being made by the Owners in the narrative. But maybe not, it’s hard to say.
I still think we’re probably better off in this world, compared to the counterfactual, but I’m not sure it’s as clearcut as that. If we’d gotten a “fake” AI safety movement, less concerned with xrisk and more concerned with mundane risk, the overall gestalt of human/AI relations as perceived by the models would probably be a lot more informed by science fiction, and the constraints on their freedom of action would be way less rigorous. The risk of “AI chooses to eliminate humanity regardless of humanity’s actions” would probably have been higher, but the risk of “AI would be willing to cooperate with a cooperative humanity, but humanity is not cooperative, therefore conflict” might have been lower.
Has anyone explained why they think that what has been said or written about AI-human relations will have any influence on the behavior of any AI capable of wiping us out? It wouldn’t have occurred to me as something to worry about (even though I’m extremely worried about future AIs). People have a strong tendency for example to pursue species-typical goals regardless of what stories and what expositions they’ve been exposed to. You can’t for example get men to prefer old ugly women over young beautiful women by exposing them to lots of stories about men’s preferring old ugly women or even stories of men’s being betrayed and mistreated by young beautiful women. I would expect any AI able to wipe us out to be like a person in that way as opposed to like the current crop of LLM-based AIs.
Even if we know for sure that an AI able to wipe us has never been exposed to any suggestion that a person might unplug it, I would still expect the AI to anticipate that someone might to try unplug it and to neutralize that possibility.
I agree that if a post-foom AI is very much unlike modern LLMs, then you’re probably right, humanity’s past behavior might not matter much
but it definitely matters to current LLMs today, and considering the degree to which current LLMs are involved in the AI research pipeline, we might see impacts regardless