To respond to this comment, I’ll give a view on why I think the answer to coordination might be easier for AIs than for people, and also explain why AI invention likely breaks a lot of the social rules we are used to.
For example, one big difference I think that impacts coordination for AIs is that an AI model is likely to be able to copy itself millions of times, given current inference scaling, and in particular you can distribute fine-tunes to those millions as though they were a single unit.
This is a huge change for coordination, because humans can’t copy themselves into millions of humans that share very similar values just by getting more compute, say.
Merging might also be much easier, and it is easier to merge and split two pieces of data of an AI than it is to staple two human brains.
These alone let you coordinate to an extent we haven’t really seen in history, such that it makes more sense to treat the millions or billions of AI instances as 1 unified agent than it is to treat a nation as 1 unified agent.
To answer this question:
In these discussions, I think there’s an implicit assumption that AIs would automatically operate outside the usual norms, laws, and social constraints that govern social behavior. The idea is that all the ordinary rules of society will simply stop applying, because we’re talking about AIs.
While this argument is indeed invalid if that was all there was to it, there is an actual reason why the current rules of society mostly stop working with AIs, because of one big issue:
Human economic labor no longer is very valuable, because labor is cheap compared to capital, and can even have negative economic value due to not being able to work with AIs due to being bottlenecks.
When this happens, you can’t rely on the property that the best way to make yourself well off is to make others well off, and indeed the opposite is the case if we assume that their labor is net-negative economic value.
The basic reason for this is that if your labor has 0 or negative economic value, then your value likely comes from your land and capital, and there is 0 disincentive, and at least a weak incentive to steal your capital and land to fuel their growth.
In essence, you can’t assume that violent stealing of property is not incentivized, and a lot of the foundations of comparative advantage and our society don’t work when you allow workers that are duplicable and very low cost.
This means if you survive and still have property, it will be because of alignment to your values, not economic reasons, because you cannot exclude bad outcomes like stealing property through violence via economics anymore.
(This is also why Ricardian comparative advantage won’t apply. If the AI side has a choice of trading with humans for something, vs. spending the same resources on building AIs to produce the same thing cheaper, then the latter option is more profitable. So after a certain point in capability development, the only thing AIs and AI companies will want from us is our resources, like land; not our labor. The best analogy is enclosures in England.)
Consider a scenario in which AGI and human-equivalent robotics are developed and end up owned (via e.g. controlling exclusively the infrastructure that runs it, and being closed source) by a group of, say, 10,000 people overall who have some share in this automation capital. If these people have exclusive access to it, a perfectly functional equilibrium is “they trade among peers goods produced by their automated workers and leave everyone else to fend for themselves”.
To address the human enhancement point: I agree that humans will likely be cognitively and physically enhanced to a level and pace of change that is genuinely ludicrously big compared to the pre-AI automation era.
However, there are 2 problems that arise here:
1, Most people that do work today do so because it’s necessary to have a life, not for reasons like intrinsically liking work, so by default in an AI automation future where a company can choose an AI over a human, and the human’s not necessary for AI to go well, I’d predict 80-90%+ humans would voluntarily remove themselves from the job market over the course of at most 10-20 years.
2. Unless humans mass upload and copy, which is absolutely possible but also plausibly harder than just having AIs for work, the coordination costs for humans would be a big barrier, because it’s way easier for AIs to productively coordinate than humans due to sharing basically the same weights, combined with very similar values due to copy/pasting 1 AI being quite likely as a strategy to fulfill millions of jobs.
To be clear, I’m not stating that humans will remain unchanged, they will change rapidly. Just not as fast as AI changes.
Finally, one large reason on why human laws become mostly irrelevant is that if you have AIs that are able to serve in robotic armies, and do automated work, it becomes far too easy to either slowly change the laws such that people are ultimately closer to pets in status, or to do revolts, and critically once AI controls robotic armies and does all of the economic work, then any social system that the human controlling the AI, or the AI itself opposes is very easy to destroy/remove.
To respond to this comment, I’ll give a view on why I think the answer to coordination might be easier for AIs than for people, and also explain why AI invention likely breaks a lot of the social rules we are used to.
For example, one big difference I think that impacts coordination for AIs is that an AI model is likely to be able to copy itself millions of times, given current inference scaling, and in particular you can distribute fine-tunes to those millions as though they were a single unit.
This is a huge change for coordination, because humans can’t copy themselves into millions of humans that share very similar values just by getting more compute, say.
Merging might also be much easier, and it is easier to merge and split two pieces of data of an AI than it is to staple two human brains.
These alone let you coordinate to an extent we haven’t really seen in history, such that it makes more sense to treat the millions or billions of AI instances as 1 unified agent than it is to treat a nation as 1 unified agent.
To answer this question:
While this argument is indeed invalid if that was all there was to it, there is an actual reason why the current rules of society mostly stop working with AIs, because of one big issue:
Human economic labor no longer is very valuable, because labor is cheap compared to capital, and can even have negative economic value due to not being able to work with AIs due to being bottlenecks.
When this happens, you can’t rely on the property that the best way to make yourself well off is to make others well off, and indeed the opposite is the case if we assume that their labor is net-negative economic value.
The basic reason for this is that if your labor has 0 or negative economic value, then your value likely comes from your land and capital, and there is 0 disincentive, and at least a weak incentive to steal your capital and land to fuel their growth.
In essence, you can’t assume that violent stealing of property is not incentivized, and a lot of the foundations of comparative advantage and our society don’t work when you allow workers that are duplicable and very low cost.
This means if you survive and still have property, it will be because of alignment to your values, not economic reasons, because you cannot exclude bad outcomes like stealing property through violence via economics anymore.
I like these comments on the subject:
https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher
https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher#BJk8XgpsHEF6mjXNE
To address the human enhancement point: I agree that humans will likely be cognitively and physically enhanced to a level and pace of change that is genuinely ludicrously big compared to the pre-AI automation era.
However, there are 2 problems that arise here:
1, Most people that do work today do so because it’s necessary to have a life, not for reasons like intrinsically liking work, so by default in an AI automation future where a company can choose an AI over a human, and the human’s not necessary for AI to go well, I’d predict 80-90%+ humans would voluntarily remove themselves from the job market over the course of at most 10-20 years.
2. Unless humans mass upload and copy, which is absolutely possible but also plausibly harder than just having AIs for work, the coordination costs for humans would be a big barrier, because it’s way easier for AIs to productively coordinate than humans due to sharing basically the same weights, combined with very similar values due to copy/pasting 1 AI being quite likely as a strategy to fulfill millions of jobs.
To be clear, I’m not stating that humans will remain unchanged, they will change rapidly. Just not as fast as AI changes.
Finally, one large reason on why human laws become mostly irrelevant is that if you have AIs that are able to serve in robotic armies, and do automated work, it becomes far too easy to either slowly change the laws such that people are ultimately closer to pets in status, or to do revolts, and critically once AI controls robotic armies and does all of the economic work, then any social system that the human controlling the AI, or the AI itself opposes is very easy to destroy/remove.