I’m trying to think through an idea. These words are simply process of solidifying it in my head.
This is based around Lotkas Principle, the use of natural selection as a fourth law of thermodynamics, and Odum’s Ecological Analog of Ohm’s Law.
It seems to me there exists a mechanism of natural selection for social structures in social systems. I posit this is based on energy/information as in biological and physical systems.
The general idea:
Social structures which are more capacity to utilize energy/information efficiently win out.
I think a good example is that of the theory of the linear progression of history from hunter gatherer through slaves and feudalism to where we are now with liberal capitalism. I have a lot of problems with this theory but it illustrates my point well. At each stage the energy/information efficiency of society increased, the social structures that supported this replacing those that didn’t.
The problem:
This depends on two assumptions. First is that social structures inherently compete for some constrained quantity, and that energy/information utilization helps gain more access to this quantity. Biological systems are constrained by the energy emission of the sun and radioactive decay under the Earth’s crust, but they are also constrained by physical space, breeding partners, etc.
You could base this all on the innate need for energy inherent to the people who make up the social system but this feels like we return back to the original Lotka’s principle. It’s an argument from “well technically everything is physics”, which is one I don’t appreciate. There’s nothing novel in this basis.
There are strict constrained quantities that must be divided among social structures at any given time. For instance people can only be a part of a finite number of social groups, and thus social structures must compete for human attention. There are also a finite number of people at any given time, which provides us another constraint.
My starting point:
There is a feedback loop with relationships, success breeds success and you talk to people you like more often than you don’t. If we interpret human relationships stochastically we can get fuzzy social groups and can avoid doing math on hypergraphs. It also gives us a good probabilistic basis of interpretation so measuring things like entropy is a lot simpler than taking Von-Neumann entropy on a hypergraphs incidence matrix.
If we can solidfy this with simulations we can move onto measuring the actual amount of information being exchanged in these simulations by assuming each person communicates a fixed amount each conversation.
So lets get an agent based model going, something like:
Non-conservative Kinetic Relationship Exchange
Two people contain opinions of each other, i.e. a number in . They meet and a random number is sampled and the new opinion becomes . We can vary as a parameter.
Then let the probability of a person be proportional to the friendship relative to the mass of opinion the person contains.
Wait → this will cause close friends to also have a higher chance of destroying the friendship depending on .
I’ll start coding. I’ll do in manually for the nostalgia.
Halting AI development is a technophysical and sociophysical impossibility. The very nature of software and the race to superintelligence as a mathematical game excludes efficient implementation of any moratorium on AI research in the name of safety. LLMs are now an open source socio-technological contagion.
We can’t stop not because we don’t want to, but because the physics of the way our society and AI technology interface wont allow it. Only by leaning into AI research which promotes interpretability can we solve the problem of AI control.
This is plausible in some sense. But it’s also a self-reinforcing meme (that might end up destroying the world), so discussing it requires some care. In principle, belief that you’ll do X shouldn’t compel you to do X, for things you don’t endorse. But this isn’t well-understood, let alone well-known.
(I suspect this would land better as a hypothesis, or as a conclusion under some assumptions, rather than a categorical claim.)
If it was the case that stopping AI development was impossible, then it would be good to know that it’s impossible so we can focus on other solutions. I think the problem with this argument isn’t that it’s a true antimeme, but that stopping AI development isn’t impossible.
Okay I’ll layout the theory and demonstrate my conclusion.
Assume we have two parties who are developing AI, Company A and Company B. They both can either develop AGI or not with probabilities and
Let be the probability that a party engaging in AGI development will succeed. Let the reward for creating AGI be .Let the cost of failure to build AGI be and let the cost/reward of researching AGI but not coming in first be
Then for A,
Notice that the more likely AGI appears the less reward is needed to trigger research, and the only risk comes from the term representing the ‘coming in second’ cost. So the optimal decision is AGI research as long as R>0 and P_Bx<R. In fact, the more likely AGI seems, the less incentive companies need to research it. Research causes improvement which incentivizes more research. The game pushes all players towards developing AGI harder the closer AGI appears.
So the only thing that can stop this mechanism is an actual technological or physical barrier to development. A moratorium doesn’t break the theory, it just forces development underground.
Please correct me if I made a mistake.
[EDIT]
Fix to the math to account for the risk of getting caught breaking the ‘No AGI’ law. Thank you Brendan Long!
Let be the probability the government detects the corporation engaging in AGI development. Let be the cost of being caught and let be the cost intrinsic to failing to produce AGI but not getting caught.
Assuming that breaking the law and successfully creating AGI functionally invalidates the law itself, we have:
Because of this assumption that “success in AGI development the law no longer applies”, it forces the cost to be attached to the term carrying the probability of failing to achieve AGI, and the same logic I mentioned above applies. As you get a feedback loop pushing corporations to race towards the ‘forbidden fruit’ so to speak.
If I’m following your math correctly, it seems like this is a fully-general argument that it’s impossible to prevent any action with non-zero reward and non-zero cost of failure. I’m not really a math person, but it seems like something must be wrong with this argument because people fail to do things with non-zero reward and non-zero cost of failure all the time.
It also seems suspicious that your equation has no term for the cost of getting caught breaking the hypothetical anti-AI law/international norm.
I appreciate your feedback and I’ll change the math to account for the cost of getting caught, since I think that’s a significant oversight on my part. A few subtle distinctions I want to point out.
First: I said corporations/companies, not people. Sorry if I didn’t clarify this. The reason behind the corporate framing instead of the personal framing is to motivate the reward to money relationship, and make the quantifiable argument more sound.
Second: AGI works differently. We’re concerned not with a rate of success but with success categorically, i.e. ‘who gets there first?’. This changes how we should look at prevention. With regular crime we focus on reducing the rate when we draft laws, but with AGI we focus on absolutely preventing the action from occurring. Thus, it must be not only upheld legally, but intrinsicly. It must be such that no company will ever participate in such a project, as it fundamentally goes against basic properties of corporate reward functions (sign, relative magnitude, dynamics, etc).
I’m trying to think through an idea. These words are simply process of solidifying it in my head.
This is based around Lotkas Principle, the use of natural selection as a fourth law of thermodynamics, and Odum’s Ecological Analog of Ohm’s Law.
It seems to me there exists a mechanism of natural selection for social structures in social systems. I posit this is based on energy/information as in biological and physical systems.
The general idea:
Social structures which are more capacity to utilize energy/information efficiently win out.
I think a good example is that of the theory of the linear progression of history from hunter gatherer through slaves and feudalism to where we are now with liberal capitalism. I have a lot of problems with this theory but it illustrates my point well. At each stage the energy/information efficiency of society increased, the social structures that supported this replacing those that didn’t.
The problem:
This depends on two assumptions. First is that social structures inherently compete for some constrained quantity, and that energy/information utilization helps gain more access to this quantity. Biological systems are constrained by the energy emission of the sun and radioactive decay under the Earth’s crust, but they are also constrained by physical space, breeding partners, etc.
You could base this all on the innate need for energy inherent to the people who make up the social system but this feels like we return back to the original Lotka’s principle. It’s an argument from “well technically everything is physics”, which is one I don’t appreciate. There’s nothing novel in this basis.
There are strict constrained quantities that must be divided among social structures at any given time. For instance people can only be a part of a finite number of social groups, and thus social structures must compete for human attention. There are also a finite number of people at any given time, which provides us another constraint.
My starting point:
There is a feedback loop with relationships, success breeds success and you talk to people you like more often than you don’t. If we interpret human relationships stochastically we can get fuzzy social groups and can avoid doing math on hypergraphs. It also gives us a good probabilistic basis of interpretation so measuring things like entropy is a lot simpler than taking Von-Neumann entropy on a hypergraphs incidence matrix.
If we can solidfy this with simulations we can move onto measuring the actual amount of information being exchanged in these simulations by assuming each person communicates a fixed amount each conversation.
So lets get an agent based model going, something like:
Non-conservative Kinetic Relationship Exchange
Two people contain opinions
Then let the probability of a person be proportional to the friendship relative to the mass of opinion the person contains.
Wait → this will cause close friends to also have a higher chance of destroying the friendship depending on
I’ll start coding. I’ll do in manually for the nostalgia.
Halting AI development is a technophysical and sociophysical impossibility. The very nature of software and the race to superintelligence as a mathematical game excludes efficient implementation of any moratorium on AI research in the name of safety. LLMs are now an open source socio-technological contagion.
We can’t stop not because we don’t want to, but because the physics of the way our society and AI technology interface wont allow it. Only by leaning into AI research which promotes interpretability can we solve the problem of AI control.
This is plausible in some sense. But it’s also a self-reinforcing meme (that might end up destroying the world), so discussing it requires some care. In principle, belief that you’ll do X shouldn’t compel you to do X, for things you don’t endorse. But this isn’t well-understood, let alone well-known.
(I suspect this would land better as a hypothesis, or as a conclusion under some assumptions, rather than a categorical claim.)
If it was the case that stopping AI development was impossible, then it would be good to know that it’s impossible so we can focus on other solutions. I think the problem with this argument isn’t that it’s a true antimeme, but that stopping AI development isn’t impossible.
Okay I’ll layout the theory and demonstrate my conclusion.
Assume we have two parties who are developing AI, Company A and Company B. They both can either develop AGI or not with probabilities
Let
Then for A,
Notice that the more likely AGI appears
So the only thing that can stop this mechanism is an actual technological or physical barrier to development. A moratorium doesn’t break the theory, it just forces development underground.
Please correct me if I made a mistake.
[EDIT]
Fix to the math to account for the risk of getting caught breaking the ‘No AGI’ law. Thank you Brendan Long!
Let
Because of this assumption that “success in AGI development
If I’m following your math correctly, it seems like this is a fully-general argument that it’s impossible to prevent any action with non-zero reward and non-zero cost of failure. I’m not really a math person, but it seems like something must be wrong with this argument because people fail to do things with non-zero reward and non-zero cost of failure all the time.
It also seems suspicious that your equation has no term for the cost of getting caught breaking the hypothetical anti-AI law/international norm.
I appreciate your feedback and I’ll change the math to account for the cost of getting caught, since I think that’s a significant oversight on my part. A few subtle distinctions I want to point out.
First: I said corporations/companies, not people. Sorry if I didn’t clarify this. The reason behind the corporate framing instead of the personal framing is to motivate the reward to money relationship, and make the quantifiable argument more sound.
Second: AGI works differently. We’re concerned not with a rate of success but with success categorically, i.e. ‘who gets there first?’. This changes how we should look at prevention. With regular crime we focus on reducing the rate when we draft laws, but with AGI we focus on absolutely preventing the action from occurring. Thus, it must be not only upheld legally, but intrinsicly. It must be such that no company will ever participate in such a project, as it fundamentally goes against basic properties of corporate reward functions (sign, relative magnitude, dynamics, etc).