My biggest critique of this approach is that it takes too literally the analogy that we will eventually be to superintelligence what dogs are to humans, and extrapolates it to suggest that we will be just as helpless as dogs are today.
Thank you, that’s an interesting point. I’ll try to lay out my counterargument as clearly as I can.
I mentioned dogs not because they have a specific level of intelligence relative to humans, but because they got a relatively good deal. Chimps are a lot smarter than dogs, and they’re worse off. Homo erectus had culturally transmitted tools, some art, seafaring craft of some sort, and possibly language. And they’re extinct. The only common factor across these cases is that runners-up in the intelligence of race didn’t get to make the important decisions.
In fact, AGI wouldn’t need to be much smarter than humans to outcompete us in the long run. For example, if it’s no smarter than the average Nobel Prize researcher, if it’s able to work productively for $1/hour, and if it’s able to copy-and-paste multiple copies of itself, then it would already be our evolutionary superior. We might be able to remain in charge for a while. But that’s sort of like how a multicellular organism can survive for many decades. But in the end, if nothing else kills them first, multicellular organisms tend to die of cancer. This is a case of local Darwinian incentives gradually eroding “cellular alignement” with the larger multicellular organism. Similarly, if the world consists of slow, expensive and frankly stupid humans, who can’t even pass down learned knowledge “genetically” with a simple copy-paste (how primitive!), and also highly cost-effective and intelligent AIs, then there’s a constant danger of alignment failing somewhere, and a “cancerous” AI replicator escaping control.
So even if we somehow manage to create “aligned” AI, I don’t expect that to last. When you’re too stupid and too expensive to be allowed anywhere near the real economy, you’re in a very dangerous long-term position.
We will still be able to logically comprehend (at a much simpler level relative to the AIs) what is good to us over a long term, in a way that dogs can’t.
I’m not convinced of this. Paul Graham once described something he called the Blub paradox. He explained this in terms of programming languages, but I suspect that it applies more broadly:
Programmers get very attached to their favorite languages, and I don’t want to hurt anyone’s feelings, so to explain this point I’m going to use a hypothetical language called Blub. Blub falls right in the middle of the abstractness continuum. It is not the most powerful language, but it is more powerful than Cobol or machine language.
And in fact, our hypothetical Blub programmer wouldn’t use either of them. Of course he wouldn’t program in machine language. That’s what compilers are for. And as for Cobol, he doesn’t know how anyone can get anything done with it. It doesn’t even have x (Blub feature of your choice).
As long as our hypothetical Blub programmer is looking down the power continuum, he knows he’s looking down. Languages less powerful than Blub are obviously less powerful, because they’re missing some feature he’s used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn’t realize he’s looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.
When we switch to the point of view of a programmer using any of the languages higher up the power continuum, however, we find that he in turn looks down upon Blub. How can you get anything done in Blub? It doesn’t even have y.
When we look “down”, chimps are obviously stupider than we are. They don’t have spoken language! They don’t have books! They can’t do real math! The can make “tools”, sure, but they’re basically pointy sticks, not factories, Space Shuttles, or computers. Their “economy” is based on family relationships and some individual reciprocity, and they don’t have even one joint stock company. Their idea of military strategy is to gang up in a band and go murder some other chimps, without understanding the role of non-commissioned officers or combined arms!
Chimps, to put it politely, have no clue.
But let’s trying looking “up” the intelligence spectrum? What do we see? Well, it looks sort of like funny humans with some weird extra stuff. The AIs can’t be that much smarter than we are, right? And if we ask nicely, I’m sure they can explain everything important to us.
But when the AIs look “down” towards Homo sapiens, they just shake their heads. Why, humans can’t even understand Z! Even if you take something really simple, like how isomorphisms between topoi and subsets of the lambda calculus make it trivial to design powerful custom programming languages for specific tasks, their eyes just glaze over! Even primitive baby AIs like Opus 4.5 could understand that. Can you imagine trying to explain to a human what replaced the econony, lol?
So here are some things which I expect to be true:
AI that was in the top 0.01% of human intelligence, that worked for a dollar an hour, and that could be replicated by copying a hard drive would already be enough to jeopardize human control of our futures.
Basic Darwinism suggests that highly resource-efficient replicators with a high rate of replication will ultimately tend to replicate.
Even weakly superintelligent AIs will have a broad range of powerful ideas and skills that humans are poorly equipped to understand, in much the same way that chimps don’t understand joint stock companies or combined arms warfare, or the way that Homo erectus doesn’t seem to have understood long-distance trade. This will make checking up on what the AIs are doing vastly harder.
My argument here is really just basic economics, politics and evolutionary biology. If you create something that renders human intellectual and physical labor economically worthless and evolutionarily uncompetitive, then the odds are excellent that you’re going to lose control. Maybe the AI will like keeping humans around as glorified pets! But that will be the AI’s decision, not ours.
Well, an aligned AI would do whatever the humans want.
If asked to not replicate even with the ability to, it wouldn’t. Or maybe you can tell it to replicate just enough to help you root out the actual AI replicators being built elsewhere, then stop at that point.
I think your argument does show how hard and fragile it is to deeply align AI in this way, though.
Thank you, that’s an interesting point. I’ll try to lay out my counterargument as clearly as I can.
I mentioned dogs not because they have a specific level of intelligence relative to humans, but because they got a relatively good deal. Chimps are a lot smarter than dogs, and they’re worse off. Homo erectus had culturally transmitted tools, some art, seafaring craft of some sort, and possibly language. And they’re extinct. The only common factor across these cases is that runners-up in the intelligence of race didn’t get to make the important decisions.
In fact, AGI wouldn’t need to be much smarter than humans to outcompete us in the long run. For example, if it’s no smarter than the average Nobel Prize researcher, if it’s able to work productively for $1/hour, and if it’s able to copy-and-paste multiple copies of itself, then it would already be our evolutionary superior. We might be able to remain in charge for a while. But that’s sort of like how a multicellular organism can survive for many decades. But in the end, if nothing else kills them first, multicellular organisms tend to die of cancer. This is a case of local Darwinian incentives gradually eroding “cellular alignement” with the larger multicellular organism. Similarly, if the world consists of slow, expensive and frankly stupid humans, who can’t even pass down learned knowledge “genetically” with a simple copy-paste (how primitive!), and also highly cost-effective and intelligent AIs, then there’s a constant danger of alignment failing somewhere, and a “cancerous” AI replicator escaping control.
So even if we somehow manage to create “aligned” AI, I don’t expect that to last. When you’re too stupid and too expensive to be allowed anywhere near the real economy, you’re in a very dangerous long-term position.
I’m not convinced of this. Paul Graham once described something he called the Blub paradox. He explained this in terms of programming languages, but I suspect that it applies more broadly:
When we look “down”, chimps are obviously stupider than we are. They don’t have spoken language! They don’t have books! They can’t do real math! The can make “tools”, sure, but they’re basically pointy sticks, not factories, Space Shuttles, or computers. Their “economy” is based on family relationships and some individual reciprocity, and they don’t have even one joint stock company. Their idea of military strategy is to gang up in a band and go murder some other chimps, without understanding the role of non-commissioned officers or combined arms!
Chimps, to put it politely, have no clue.
But let’s trying looking “up” the intelligence spectrum? What do we see? Well, it looks sort of like funny humans with some weird extra stuff. The AIs can’t be that much smarter than we are, right? And if we ask nicely, I’m sure they can explain everything important to us.
But when the AIs look “down” towards Homo sapiens, they just shake their heads. Why, humans can’t even understand Z! Even if you take something really simple, like how isomorphisms between topoi and subsets of the lambda calculus make it trivial to design powerful custom programming languages for specific tasks, their eyes just glaze over! Even primitive baby AIs like Opus 4.5 could understand that. Can you imagine trying to explain to a human what replaced the econony, lol?
So here are some things which I expect to be true:
AI that was in the top 0.01% of human intelligence, that worked for a dollar an hour, and that could be replicated by copying a hard drive would already be enough to jeopardize human control of our futures.
Basic Darwinism suggests that highly resource-efficient replicators with a high rate of replication will ultimately tend to replicate.
Even weakly superintelligent AIs will have a broad range of powerful ideas and skills that humans are poorly equipped to understand, in much the same way that chimps don’t understand joint stock companies or combined arms warfare, or the way that Homo erectus doesn’t seem to have understood long-distance trade. This will make checking up on what the AIs are doing vastly harder.
My argument here is really just basic economics, politics and evolutionary biology. If you create something that renders human intellectual and physical labor economically worthless and evolutionarily uncompetitive, then the odds are excellent that you’re going to lose control. Maybe the AI will like keeping humans around as glorified pets! But that will be the AI’s decision, not ours.
Well, an aligned AI would do whatever the humans want.
If asked to not replicate even with the ability to, it wouldn’t. Or maybe you can tell it to replicate just enough to help you root out the actual AI replicators being built elsewhere, then stop at that point.
I think your argument does show how hard and fragile it is to deeply align AI in this way, though.