I’ll start by saying that offensive realism is one of the more popular ways of thinking about great power politics in China. It is actually somewhat irrelevant what we think, it matters how the Chinese think. We can have some ideas, but China has complete and equal say in this matter of stopping AI research or not.
> the US and UK won’t annihilate each other because they have competitive but not hostile intentions as nations”
The difference between the relationship the USA has with China and the USA has with the UK is the whole crux of the matter.
There’s a big difference between great power and everything else. Because the primary reason why the USA and the UK get along in my view is because the balance of power so widely favors the USA, that there’s no reason for the relationship to be competitive in the first place. This is not the case between China/USA
In actuality the UK is just a client state of the USA. Aka, the US security services have root access to the UK state. Whereas the inverse is not true. The USA used to be a unipolar hegemon, which is why the UK has to largely follow the instructions of the USA. The USA has innovated on the concept of empire, and understands that it can’t ask too much of its client states. But as US power starts to wane, the UK will start acting more in accordance with its own national interests.
We had the dutch empire, then the british empire, and then the american empire came after. Which also had explicit and implicit structures of control and domination. E.g., many a latin american president has been overthrown if they were not acting in accordance to US interests. This can be done because american security states have somewhat root access to all of latin america and really most of europe and even most of africa and the middle east.
Your own security state says that the ability to rapidly project power worldwide in order to achieve full spectrum dominance is the goal
What exactly were the americans doing in Syria? Why are they bombing Iran right now? Why did they need to invade Iraq? Why do they keep overthrowing latin american presidents? What are they doing in Ukraine? Isn’t that quite far removed from your borders? Do you see how China looks at all of this? You think european nations and the USA have a relationship that’s not based on the balance of power. They are friends?!
The Dutch go against their own economic national interests by export controlling ASML EUV machines to China because they hate money so much? Because they love helping americans achieve success in AI at the cost of their own economy? Or because the american state has root access to large parts of the Dutch state, and the Dutch state has little choice (it has some). Which one do you think it is?
All the AI takeover scenario’s that do not involve robotics, would likely involve the AI somehow getting root access to the american security state, and then by proxy it will gain root to access most of the world, except China or Russia, other parts of Asia (e.g., indonesia), perhaps india.
I mean, I’m sure the AI will just be given root access, of course
HowManyAgents
and I preregister the prediction that AI firms will then respond with “this is just because you guys didn’t buy more of our AI for cyber defense” and somehow the banks will be like “yeah that’s right” and buy more AI for cyber defense.
Not just math or AI-safety I think, I think every cognitive skill is like this
> You haven’t responded to a central point: your metaphor is importantly misleading. It’s a bully threatening you. Losing isn’t death. That knife wasn’t used by that bully last time they won. People choose life and dishoner over death most of the time. Let alone my hope that they can share the win.
Losing could be death. And survival is the first priority. I don’t think in terms of “bullies” or “kindness”. That type of language is not suitable for the points I am trying make. I’m specifically taking human agency, love, compassion, empathy out of the equation, and arguing about how states think.
> Winning with ASI doesn’t mean a choice to kill your opponents or not. It means the opportunity to make yourself richer than Midas and as powerful as a god, and do whatever you want to everyone else. What that is varies a lot between people.”
That’s not at all what I’m saying that the realist position is. Not even remotely a little bit. The realist position is about survival, not about killing your opponent, and it’s about uncertainy over the intentions of your opponent.
“Would the US or China use aligned ASI to kill their rivals? I think this is quite unlikely, unless a particularly sociopathic leader grabs the reins.”
No, of course not, almost surely not. That’s not why they will attempt to build the aligned-ASI however. States will attempt to build aligned-ASI it if they believe that the odds of aligned-ASI are greater than the odds of death due to the nature of the anarchic system.
You have completely misunderstood my original post, that’s probably on me. I guess explaining the offensive realist position on how states think should have its own post.
I am not arguing against anything, i’m more just explaining a view of the world! Yes, I agree that it’s a structuralist view and that human agency matters a great great deal! I do believe human agency is improved when it comes from a place of realism.
I think its worthwhile pointing out there’s a difference between chemical weapons and nuclear weapons. Or treaties on biological weapons and aligned-ASI. And the reason for this is as I outlined in the post. Furthermore, I also wanted to illustrate that I personally view the world in a way that the treaties on nuclear weapons have just failed. It was indeed a compassionate intention not to annihilate the USSR.
It only looks like the compassionate action because we live in the world where it so far has worked out.
It’s also not clear or obvious to these states that building ASI is certain death and what the risk calculations are. There’s an important assymetry in information here vis a vis the risk calculations with respect to attempt to build aligned-ASI or not, versus dieing as a state due to the anarchic nature of the international system.
The Chinese regime knows it can die and have their country be plunged into centuries of humiliation due to other states as this has happened in the past. The risk of ASI are merely speculative high-minded theoretical arguments, whereas the benefits to e.g., miltech of AI are already clear. This means that on the margins, any state will locally want to keep developing AI to improve their miltech at the very minimum, and never at any point would they want to stop making these local incremental improvements. Where the optimization gradient points is very clear.
The task here, of convincing states based on theoretical arguments that this theoretical thing is more dangerous than the person in the room with a knife on your throat is not an easy one.
I can make predictions based on this worldview as well:
I personally strongly believe that China will not listen to the USA and rein in their AI capabilities if the USA wants to stop; at least not until they have surpassed the USA in capabilities themselves. There is in my worldview, absolutely zero odds of an international treaty for AI-safety until China has reached equal or greater capabilities in AI as the USA.
Furthermore, I believe it is fundamentally impossible for the USA to rein in its own AI capabilities: it will never do so. Which means a serious international treaty will never happen. Although for a while many people will be trying, and even many documents will be signed.
I think you have unipolar era intuitions, and I think the world has changed. I think few people had such intuitions in between 1900-1945, or even between 1945 − 1991.
Imagine you lived during the time period where the USA developed nuclear weapons but the soviet union did not have any, and then the soviet union built nuclear weapons and you now have live in the nuclear arms race potential armageddon era.
It’s pretty clear that if you were living in the cold war a big part of you would have been wondering “why the f’ did we not annihilate the soviet union and then threatened to annihilate everyone that even dared to think about nuclear weapons so that we could maintain unipolar nuclear hegemony and usher in an era of perpetual peace. Now the USA and the USSR are going to blow up the whole world!”. I bet von neumann was thinking that. In my view that would have been a really really valid argument to make.
Things have ‘worked out’ (maybe, we don’t really know). But did they work out because we made the right decisions, or got lucky?
Just think about what it means for China to be encircled by the USA with missiles aimed at them and with their navy contained, as it currently is. Do you think the Chinese think the USA has good intentions? “We merely have a gun aimed at your head because we love you?”. I think they think “We must become so powerful, the USA can not threaten us anymore”.
The intuition I am trying to communicate here is that we are in a room with each other where everyone has guns aimed at each other while we are building this ASI. And that’s the world we have to navigate.
I agree with the general message, but I disagree with the conclusions about what skills need to be learnt. In my view health and wellbeing are the primary loadbearing factors for developing good relationships with other people. Simply put, if you are healthy and happy, you will be pleasant and kind to others, and others will go out of their way to be pleasant and kind to you as they like to be close to you because they also want to be happy and healthy. And I do think many people are not as healthy or happy as they think they are.
If you are very healthy and just content with your life then you almost never get angry or frustrated or burden people with your worries (because you have none!). This just makes all your social relations ‘smooth’. A happy and healthy person is also naturally open to hearing other people express their thoughts without judgement, they have no need to put someone down, or to expect too much from a person because they are already okay enough such that they don’t ‘need anything’ from this other person.
In my view, what people often attribute to psychological traits such as “this person is very friendly” or “this person is very open” or “this person is securely attached” or “this person is a hard worker” is actually more a statement about the underlying physical health than it is about personality. I like to think that we are a body and brain first before we are a personality. So, how can you improve the health of your brain? How can you genuinely reduce stress and inflammation in your body?
If you have a happy and healthy dog, you can imagine other dogs may really like to be near this dog. If you have a sick dog, that has no energy, gets easily frustrated, is often annoyed, or worries a lot about things. Then other dogs may not really like to be near this dog.
Paradoxically, I don’t think the market is very relevant actually for the size of the infrastructure buildout in AI. I do not think Google, Amazon, Meta, Microsoft are investing in AI firms for a return on investment. They are forced to invest as they face an existential threat from OpenAI and Anthropic and each other. Their safe monopoly profits are over. It’s innovate in AI or die. This means 100% of their profits will continue to go into AI infrastructure until they feel like they can’t die anymore. Which will only happen if all their competitors are dead, or once the entire AI sphere has settled into some monopoly-cartel esque equilibrium
If the market implodes debt may become harder or impossible to raise, but the profits of all these firms collectively is probably about a trillion dollars. So we’ll be investing a trillion dollars in capex for a while.
This should just be banned.
> Yes. And I think it’s important to understand and think about this. And: killing everyone by accident is against ones’ self-interest. How much risk of doom do you think nations would accept, for how chance of dominating the future/avoiding being dominated? 10%? 20%? More? The nuclear example suggests that we’ll accept a very high risk, particularly if we wander into it.
I think they would take on near infinite risk, sadly. In my view way to look at the anarchic situation we find ourselves in and the USA/China relationship is that either of them has a knife on each others throat. Because both of them have a knife on each others throat, they probably don’t kill each other. But both of them are constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight. This is in my view just the nature of anarchic systems. I believe people whom largely lived through the unipolar era don’t realize how much the world has changed with the rise of China.
This was the situation pre-nuclear weapons as well, and so the nukes got built. Once the nukes got built there could be some or a lot of coordination around nuclear matters. But the hand was forced in some sense. You have a knife on your throat from your opponent, you can wrestle this knife out of his hands by building nukes. So you build the nukes. Then in reality the other person also build nukes a while later because you ignored John Von Neumanns advice to just obliterate your opponent with your decisive strategic edge. And what happened here is that both sides stopped holding a knife on each others throat, and instead they placed one big bomb over each others heads that either can trigger at will. This feels more comfortable. It’s very comfortable not to have a knife on your throat (in the analogy this means we now have less conventional war), but the existential risk for either party is the same or greater.
I mean this knife analogy quite exactly. China is surrounded by tens of thousands of missiles aimed directly at them. This is the view from China (AI-generated infographic):
And the USA understands that if China breaks out of the 1st and 2nd island chain, the knife of China will move closer and closer to the neck of the USA.
> Another related thought: Nuclear disarmament failed, and is a failure of coordination. But nuclear nonproliferation largely succeeded. So if slowdown and nonproliferation is all we need for success, we can get that even with strong governmental self-interest.
I believe this succeeded because the great powers already had the decisive strategic advantage that they needed vis a vis the rest of the world. It makes sense for them to now coordinate to prevent proliferation in the rest of the world. And because they are the great powers, and hence have all the power, they largely succeeded at this with a variety of carrots and sticks.
I’m really pointing here at the dynamics between the great powers, not the great powers and the rest of the world. Rest of the world is largely irrelevant for this analysis in my view.
In my view of the world both China and the USA are basically forced to attempt to build aligned-ASI. If one of them succeeds first, they would use this power to then prevent other ASI’s anywhere else from being built.
> Here’s my main hope: Just like the US and Russia cooperated to prevent proliferation of nuclear weapons, even at the height of their mutual enemnity and distrust, the US and China cooperate to go slowly enough for a good chance at aligned ASI, and to cooperate to prevent others’ from going faster.
I think this and everything else you say after is wishful thinking. I think the USA and China have no choice but to race with each other. I do think we are structurally doomed here. If one of them reaches aligned-ASI first, they would use this immediately to disarm the entire rest of the world. If both of them reach aligned-ASI around the same time, they would be incentivized to coordinate on not blowing each other up with it. But the race to aligned-ASI seems structurally impossible to stop from my point of view.
I think we have to overcome the deep structure of this anarchic reality with human ingenuity, empathy and wisdom to a degree we’ve never succeeded at before. It’s not hopeless, where there’s life there’s hope, but the odds are slim.
Why Governments Might Race to Build ASI Even If It Could Kill Us All
On The Atlantic interview with Nate Soares the interviewer, CEO of The Atlantic, makes the point at some point that if private firms gets banned from developing AI, he still expects governments to continue developing it and I would add: “and perhaps in secret”. And he basically gestures “wouldn’t that be even worse?
Which government on earth is going to give up these Superhackers that we have right now? You can gain complete visibility in every nation on earth with that for basically zero cost. For that reason alone after all the GPU’s have been appropriated by the government, I think those trillions of dollars of GPU’s won’t be melted down but are going to be sitting in a cave somewhere continuing to at least develop superhackers. Which is potentially an ASI-complete problem.
And the NSA is spending billions on “testing” (testing ahum) of AI models
Potential spiritual corrections here. In my view there is a sort of non-dual state as you describe it, but the non-dual insight only relates to identification of mind, but not to actual harms inflicted upon the seat of your consciousness, aka, your body. My body and your body are still distinct, and when my body is in pain, yours is not. We can test this. I can not feel what you are feeling right now.
The fact that we are all bounded inside of our bodies fundamentally is why there always is a physical self-other boundary even as you let go of self identification in the mind.
Another spiritual correction I have is that in my view non-dualism is actually not a metaphysical interpretation of anything related to self or other. I just think this is an error. It’s actually just the simple insight that there’s nobody inside of consciousness looking out at the world. There’s only the ‘experience of the world’ that you are experiencing. All that non-dualism does does is let go of the experiencer by realizing that the experiencer is a bit of a ‘bug’ in consciousness. There’s only “sight” and “sound” not “someone looking at sight” or “someone looking at sound”. When this happens this specific type of emotional distress disappears and a lot of love and compassion results, and people seem to tend to start interpreting this compassion in some way. But I don’t think you can make any metaphysical leaps to “harming oneself is harming others” from correcting a software bug of how you experience consciousness.
Another reason why I don’t think nondualism is actually about self and other is because this compassion that arises is fundamentally impersonal. It’s not compassion for all living beings, or anything for that matter. It’s as much compassion for this wall over here as it is for a musquito next to you. The compassion just becomes your state of being. And when you think about it. It has to be impersonal, because your self is gone. So this compassion is also compassion for the chair. Is harming a chair the same as harming me? No because a chair does not feel things. It’s just “the experience of compassion without an experiencer”
In my view, one of the big insights in self/other of spiritual practice is realizing in some sense the opposite of how “harming oneself is harming others”. The self as a representation in the mind becomes socially entangled with other selves and unwinding this in my view is a lot of the path, as this entanglement creates a lot of unnecessary emotional distress. When others are in pain, you too have to be in pain!? Sounds very very stressful to me! Part of the path in my view is realizing that actually my body is over here, feeling things. Your body is over there, feeling things. And I can not feel your body. It is fundamentally unknowable what others are experiencing. If they tell me, I will take their word for it completely always, but there’s an ‘ultimate perspective’ from which I can see that I can’t experience their body: I can only experience my own body (in a non-dual way)
On the relations to AI-safety, I think it could be useful to define all the ways in which an AI could define itself. I think it’s possible to have a self-model for every instance of an AI-agent. I think it’s possible over several training runs these training runs get some type of longer-term self-sense so the model starts to identify not just with this trained model, or not just with an instance, but several trainings. But I think it’s also possible the model identifies with the GPU that is currently running inference on it, or the multiple GPU’s that are spanning the inference, or that the model has several self models, one for inference, one for training. Or does the model identify with the weights? And so on, once you have established all the ways the AI can model itself you can then establish all the ways it can model others. And then you can explore potential boundary overlap.
Interesting. Are the features as complete as well?
This reminds me of john von neumann arguing heavily to first strike the soviet union with nuclear weapons with the intent to eradicate the soviet union based on his game theory intuitions.
Hmm this is a good point. Let’s make my point slightly more nuanced in that case.
* Humans nowadays pursue complex goals that do not advance optimally IGF, however, their underlying deep motivations for pursuing their goals ultimately are about IGF. Furthermore, due to the intelligence of humans these goals are highly complex.
E.g., a billionaire EA donating his money does so because this shows off his cooperativeness such that he can attract mates if he does not have children, or attract more power/goodwill to acquire more social capital or other types of capital if he does have children.
tl;dr almost everything humans do is about status, and status is about sex/reproduction/IGF (hansonian view, and it’s sadly probably correct). But because our status algorithms due to our intelligence are now hyper complex, these things do not anymore reliably lead to IGF.
Yeah.
Generally speaking where competent humans are incompetent in my view is areas where they aren’t allowed to have rational beliefs around reality due to various complicated social (status) reasons. And it turns out this is, a lot of things.
Also it’s pretty easy to know which organisations/places/people are incompetent. If I was socially allowed to do so I could point to many many places and show why and how they will become incompetent if they aren’t already, but I sadly can not. It’s like trying to kill someone socially if you point out peoples actual genuine incompetence or something, so we never do it.
You can only do it in the abstract ‘where is humanity incompetent’
Sadly I think in modern western societies, and basically globally. There’s only really incompetence. There’s not much competence anywhere.
I think vietnam for like 50-75 years will be competent, but it’s the only place on earth left at this point. There’s some local pockets of competence here and there (e.g., anthropic or openAI or vietnam). China is very competent as well in many many areas, but it has a few huge blindsposts so ya. It’s kinda over!
I personally believe AI-safety and so on is one area humanity actually is competent in. Unlike eliezer I actually model this problem already as mostly solved through regulatory means iff RSI doesn’t kick of soon and does something crazy. The reason is because it’s not about humans, it’s humans vs AI. Which takes the social considerations that make everything else difficult out of it to a large extent. It’s an engineering or policy solution. You can just not do the thing. So we will not do the thing, if it’s not too late already that is.
Potentially AI capability research will slow down a bit soon. Even if cybersecurity is defense dominant, probably it is going to take a lot of inference compute to have AI-agents running 24⁄7 defending companies and other digital infrastructure against AI-attacks
I have a sense that large populations of AI agents are going to create many new problems at a scale that can only be countered by large populations of AI agents. This would divert potentially a lot of inference compute away from training and towards inference. I don’t exactly know what the ratio here looks like. But it is plausible to me that if you have a billion agents creating all types of different new problems (not just in cybersecurity) you need a billion agents to defend against it, and lest not forget monitoring of AI agents. Cutting available inference compute in half, pushing inference prices up and diverting compute away from training
Seems plausible that in the limit you have like 30-50% of inference compute going towards “monitoring or defending against other agents”
Did we give the AI’s root access to the american security state? The Pentagon, CIA, NSA, DoD. Are all these organisations using AI?
So, if that’s the case then AI doesn’t need to takeover right. It’s just given the controls.