… solving the AI control problem merely delays the inevitable. If we get our house in order prior to the development of AI then it won’t matter whether we solved the control problem in advance. And if we never get our house in order, then it doesn’t matter whether or not we solved the control problem, we are doomed anyway
I am not at all sure how ‘getting our house in order’ prior to AI development assures that the Control Problem will not be necessary to solve. A system that can reliably ensure that no agent could do something harmful, when the agents have human-level capabilities, can fail when dealing with something superhuman. A house that is in order for my family would not continue to be if Magneto moved in.
Also, in the paragraph before the quoted one, you used ‘are’ where you meant ‘our’.
If your house is totally in order, then no one ever gets to the stage of building a powerful unsafe AI. Society would be safe even if building a superhuman AI simply immediately destroyed the world. Indeed, we may some day uncover similarly simple/important technologies that do simply destroy the world—the point of having your house in order is to be robust to even the worst technological developments.
There are a lot of assumptions here, some of which I think don’t hold. But the concept, an institutional variation of the Bayesian Conspiracy, is interesting and new.
One of such assumption is that a powerful AI will require a mature, difficult to bootstrap industrial process, instead of say, something that can be created by anyone having access to a developement environment (this is where I think the analogy with nuclear weapons starts to break).
Also, the existence of a barrier controlling future technology will create powerful incentives to competing agencies and hackers.
A singleton could also shields us from Basilisks, although it would also shields from the opposite kind of acausal trade (which I’ve called Phoenixes).
This should make you agree more with the overall takeaway from the post; on your perspective AI is extremely useful for this kind of coordination.
That said, I do disagree with your claim. I think it’s easy to imagine a world that is like ours but in which e.g. there is a single decision-making agency (call it a singleton-lite, imagine something like a modern state’s effective monopoly on violence) with exclusive access to post-1950 technology.
No one can run an AI because the singleton-lite controls all of the powerful computers. No one can build powerful computers because the singleton-lite controls all semiconductor manufacturing. No one can make a semiconductor fab because the singleton-lite controls all powerful manufacturing and chemical synthesis.
The singleton-lite mostly needs to defend factories, semi-conductor fabs, datacenters, and so forth. It must also use force to prevent criminals from e.g. building up parallel manufacturing. But the scale required to build up a parallel industry is quite large, and it doesn’t seem at all clear that it would require especially invasive surveillance to prevent it (you only have to surveil groups of hundreds of criminals working together—something we already do).
Going another route (closer to the one you seem to have in mind) you could do this with intrusive surveillance but no special control over technology. It’s not clear that this require sophisticated AI either though. Certainly cheap human-level AI is sufficient to surveil all living humans but you could presumably get by with much dumber AI doing the vast majority of monitoring.
I think you are much better off in the regime where no humans have access to powerful computers, then the regime where they have access to powerful computers but surveillance is sufficiently tight that they can’t use them to do harm. And the “no one has powerful computers” option doesn’t really involve AI in a critical way.
I agree that powerful AI makes all of this significantly easier (that was one of my main claims in the post). For example, it’s not clear that we could even run a stable singleton-lite without significant progress on AI control. But I don’t think the issue is straightforward.
That said, I do disagree with your claim. I think it’s easy to imagine a world that is like ours but in which e.g. there is a single decision-making agency (call it a singleton-lite, imagine something like a modern state’s effective monopoly on violence) with exclusive access to post-1950 technology.
What a dystopic nightmare. I’ll take destruction by AI over that, please.
I’m not trolling, I’m serious. Not even existential risk is justification enough to deny basic liberties.
What exactly is the relevant distinction with state control of nuclear weapons, from the perspective of liberty?
We won’t ever wind back to a singleton-lite controlling controlling all post-1950 technology (it’s just an analogy) but we could end up in a situation where the singleton-lite controls e.g. all post-2040 technology.
So, that “singleton-lite”, is it going to be a human? A group of humans? In this case I’m curious how do you think that singleton-lite will evolve in time (“power corrupts” comes to mind).
Also, shouldn’t we call it a “global dictatorship enforcing its monopoly on high technology”, then?
One, what is the mechanism by which this singleton-lite can be removed from power and dismantled? In contemporary democracies governments come and go; constitutions (and even EU treaties) can be changed and amended by the will of voters. How will that work in your case?
Two, the singleton-lite will have a huge power advantage: it will have sole access to high technology and the rest of the world will not. What will be there to counterbalance its reach and its power?
The level of control you are giving to this singleton entity is nightmarish. You are talking about an all-powerful autocratic entity that has absolute authoritarian control over the entire human populace. For relevant examples of how well this has fared in the past, including humanitarian costs, look at communist states of the 20th century, in particular the early Soviet Union and revolutionary China. Fun times.
What you describe is seriously one of the most evil state of affairs I can imagine being constructed today. Denial of access to technology, particularly the development of new technologies, will retard progress in medical science, stagnate our economy, and restrict basic freedoms to such a degree as would be required to maintain the police state.
Sure, you are free to ignore me. But you should expect exactly this sort of response anytime you advocate for draconian authoritarian states. Freedom ain’t free; it is bought at the cost of eternal vigilance and a willingness to defend basic liberties from those who would seek to deny them. People who have lived through these terrors in the 20th century know just how real and non-academc this is.
I said “we can imagine X,” not “we should strive to create X.”
I objected to descriptions like “dictatorship,” “autocracy,” and “authoritarian” because those are words with actual meanings which don’t necessarily apply to the institutions under discussion. For example, a democracy may pass and enforce laws without becoming a dictatorship.
I’m not trolling, I’m serious. Not even existential risk is justification enough to deny basic liberties.
We already live in a world where this is roughly the case for nuclear technology. It’s not obvious that this is a huge infringement on basic liberties, or that it would be if that were extended to include powerful computers.
(I am assuming that, like we have regulated power plants that produce energy for civilians, we would also have regulated software companies and research labs that can use powerful computers, but have to do so in a way that makes clear they’re not building AI. We also may have different definitions of ‘powerful computers.’)
We already live in a world where this is roughly the case for nuclear technology
And how effective was this at denying rogue, outcast states (like North Korea) access to nuclear weapons?
Also, I don’t think the OP meant just regulations on the supercomputer use. He explicitly mentioned “a single decision-making agency … with exclusive access to post-1950 technology”. So basically everything that involves a transistor is locked up in a government building and you need a security clearance to touch it. That doesn’t look reasonable (never mind desirable) to me.
It’s only roughly the case for nuclear weapons, we have prevented most but not all actors from developing nuclear weapons.
Forming a singleton-lite in 1935 to control access to post-1950 technology was a metaphor for gradually forming a singleton-lite at some point in the future which controls access to further-future technology.
I don’t see the qualitative distinction between “any computer” and “giant computers,” it seems like the same discussion but shifted 60 years forward in time.
I don’t see the qualitative distinction between “any computer” and “giant computers,”
Is there a qualitative distinction between “any bomb” and “nuclear bombs”?
In any case, you are basically arguing for enforced stagnation, for entering a sort of technological stasis, to be enforced by a world government which does have access to high tech. Presumably there will be some sort of a high priest caste to tend to this high tech. Does that sound about right?
This looks like a pretty standard sci-fi dystopia to me. Why do you like it?
We already live in a world where this is roughly the case for nuclear technology. It’s not obvious that this is a huge infringement on basic liberties, or that it would be if that were extended to include powerful computers.
What he said was “exclusive access to post-1950 technology.” That’s a heck of a lot more than “powerful computers.” It also includes, e.g., the entirety of modern transportation technology, communications (especially the Internet), logistics, agricultural automation, and nearly every medical technology.
It would be something like Maoist China in the height of the Great Leap Forward and Cultural Revolution. What a nightmare.
Also, as mentioned by Lumifer such nuclear restrictions haven’t worked. States that want nuclear weapons have obtained them, and about all that has been accomplished is the retardation of peaceful nuclear technology in the hands of civilians :(
In Handling Destructive Technology,
I am not at all sure how ‘getting our house in order’ prior to AI development assures that the Control Problem will not be necessary to solve. A system that can reliably ensure that no agent could do something harmful, when the agents have human-level capabilities, can fail when dealing with something superhuman. A house that is in order for my family would not continue to be if Magneto moved in.
Also, in the paragraph before the quoted one, you used ‘are’ where you meant ‘our’.
If your house is totally in order, then no one ever gets to the stage of building a powerful unsafe AI. Society would be safe even if building a superhuman AI simply immediately destroyed the world. Indeed, we may some day uncover similarly simple/important technologies that do simply destroy the world—the point of having your house in order is to be robust to even the worst technological developments.
There are a lot of assumptions here, some of which I think don’t hold. But the concept, an institutional variation of the Bayesian Conspiracy, is interesting and new.
One of such assumption is that a powerful AI will require a mature, difficult to bootstrap industrial process, instead of say, something that can be created by anyone having access to a developement environment (this is where I think the analogy with nuclear weapons starts to break). Also, the existence of a barrier controlling future technology will create powerful incentives to competing agencies and hackers.
A singleton could also shields us from Basilisks, although it would also shields from the opposite kind of acausal trade (which I’ve called Phoenixes).
That seems basically impossible without AI doing incredibly intrusive work all over the place.
This should make you agree more with the overall takeaway from the post; on your perspective AI is extremely useful for this kind of coordination.
That said, I do disagree with your claim. I think it’s easy to imagine a world that is like ours but in which e.g. there is a single decision-making agency (call it a singleton-lite, imagine something like a modern state’s effective monopoly on violence) with exclusive access to post-1950 technology.
No one can run an AI because the singleton-lite controls all of the powerful computers. No one can build powerful computers because the singleton-lite controls all semiconductor manufacturing. No one can make a semiconductor fab because the singleton-lite controls all powerful manufacturing and chemical synthesis.
The singleton-lite mostly needs to defend factories, semi-conductor fabs, datacenters, and so forth. It must also use force to prevent criminals from e.g. building up parallel manufacturing. But the scale required to build up a parallel industry is quite large, and it doesn’t seem at all clear that it would require especially invasive surveillance to prevent it (you only have to surveil groups of hundreds of criminals working together—something we already do).
Going another route (closer to the one you seem to have in mind) you could do this with intrusive surveillance but no special control over technology. It’s not clear that this require sophisticated AI either though. Certainly cheap human-level AI is sufficient to surveil all living humans but you could presumably get by with much dumber AI doing the vast majority of monitoring.
I think you are much better off in the regime where no humans have access to powerful computers, then the regime where they have access to powerful computers but surveillance is sufficiently tight that they can’t use them to do harm. And the “no one has powerful computers” option doesn’t really involve AI in a critical way.
I agree that powerful AI makes all of this significantly easier (that was one of my main claims in the post). For example, it’s not clear that we could even run a stable singleton-lite without significant progress on AI control. But I don’t think the issue is straightforward.
What a dystopic nightmare. I’ll take destruction by AI over that, please.
I’m not trolling, I’m serious. Not even existential risk is justification enough to deny basic liberties.
What exactly is the relevant distinction with state control of nuclear weapons, from the perspective of liberty?
We won’t ever wind back to a singleton-lite controlling controlling all post-1950 technology (it’s just an analogy) but we could end up in a situation where the singleton-lite controls e.g. all post-2040 technology.
So, that “singleton-lite”, is it going to be a human? A group of humans? In this case I’m curious how do you think that singleton-lite will evolve in time (“power corrupts” comes to mind).
Also, shouldn’t we call it a “global dictatorship enforcing its monopoly on high technology”, then?
I don’t see why it would be a dictatorship any more than any government is a dictatorship.
Two questions, then.
One, what is the mechanism by which this singleton-lite can be removed from power and dismantled? In contemporary democracies governments come and go; constitutions (and even EU treaties) can be changed and amended by the will of voters. How will that work in your case?
Two, the singleton-lite will have a huge power advantage: it will have sole access to high technology and the rest of the world will not. What will be there to counterbalance its reach and its power?
The level of control you are giving to this singleton entity is nightmarish. You are talking about an all-powerful autocratic entity that has absolute authoritarian control over the entire human populace. For relevant examples of how well this has fared in the past, including humanitarian costs, look at communist states of the 20th century, in particular the early Soviet Union and revolutionary China. Fun times.
What you describe is seriously one of the most evil state of affairs I can imagine being constructed today. Denial of access to technology, particularly the development of new technologies, will retard progress in medical science, stagnate our economy, and restrict basic freedoms to such a degree as would be required to maintain the police state.
I have objections to this comment, but I don’t think that continuing this conversation in this medium is likely to be the best use of our time.
Sure, you are free to ignore me. But you should expect exactly this sort of response anytime you advocate for draconian authoritarian states. Freedom ain’t free; it is bought at the cost of eternal vigilance and a willingness to defend basic liberties from those who would seek to deny them. People who have lived through these terrors in the 20th century know just how real and non-academc this is.
I said “we can imagine X,” not “we should strive to create X.”
I objected to descriptions like “dictatorship,” “autocracy,” and “authoritarian” because those are words with actual meanings which don’t necessarily apply to the institutions under discussion. For example, a democracy may pass and enforce laws without becoming a dictatorship.
We already live in a world where this is roughly the case for nuclear technology. It’s not obvious that this is a huge infringement on basic liberties, or that it would be if that were extended to include powerful computers.
(I am assuming that, like we have regulated power plants that produce energy for civilians, we would also have regulated software companies and research labs that can use powerful computers, but have to do so in a way that makes clear they’re not building AI. We also may have different definitions of ‘powerful computers.’)
And how effective was this at denying rogue, outcast states (like North Korea) access to nuclear weapons?
Also, I don’t think the OP meant just regulations on the supercomputer use. He explicitly mentioned “a single decision-making agency … with exclusive access to post-1950 technology”. So basically everything that involves a transistor is locked up in a government building and you need a security clearance to touch it. That doesn’t look reasonable (never mind desirable) to me.
It’s only roughly the case for nuclear weapons, we have prevented most but not all actors from developing nuclear weapons.
Forming a singleton-lite in 1935 to control access to post-1950 technology was a metaphor for gradually forming a singleton-lite at some point in the future which controls access to further-future technology.
I don’t see the qualitative distinction between “any computer” and “giant computers,” it seems like the same discussion but shifted 60 years forward in time.
Is there a qualitative distinction between “any bomb” and “nuclear bombs”?
In any case, you are basically arguing for enforced stagnation, for entering a sort of technological stasis, to be enforced by a world government which does have access to high tech. Presumably there will be some sort of a high priest caste to tend to this high tech. Does that sound about right?
This looks like a pretty standard sci-fi dystopia to me. Why do you like it?
I have objections to this comment, but I don’t think that continuing this conversation in this medium is likely to be the best use of our time.
What he said was “exclusive access to post-1950 technology.” That’s a heck of a lot more than “powerful computers.” It also includes, e.g., the entirety of modern transportation technology, communications (especially the Internet), logistics, agricultural automation, and nearly every medical technology.
It would be something like Maoist China in the height of the Great Leap Forward and Cultural Revolution. What a nightmare.
Also, as mentioned by Lumifer such nuclear restrictions haven’t worked. States that want nuclear weapons have obtained them, and about all that has been accomplished is the retardation of peaceful nuclear technology in the hands of civilians :(