In the context of AI x-risk, I’m mainly interested in
(1) can an AI use nanotech as a central ingredient of a plan to wipe out humanity, and
(2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?
[(2) is obviously possible once you have a few billion human-level-intelligent robots, but the question is “can nanotech dramatically reduce the amount of time that the AI is relying on human help, compared to that baseline?”. Presumably “being able to make arbitrarily more chips or chip-equivalents” would be the most difficult ingredient.]
In both cases it seems to me that the answer is “obviously yes”:
super-plagues / crop diseases / etc. are an existence proof for (1),
human brains are an existence proof for (2).
Therefore grey goo as defined in this post doesn’t seem too relevant for my AI-related questions. Like, if the AI doesn’t have a plan to make nanotech things that can exterminate / outcompete microbes living in rocks deep under the seafloor—man, I just don’t care.
None of this is meant to be a criticism of this post, which I’m glad exists, even if I’m not in a position to evaluate it. Indeed, I’m not even sure OP would disagree with my comment here (based on their main AI post).
The merit of this post is to taboo nanotech. Practical bottom-up nanotech is simply synthetic biology, and practical top-down nanotech is simply modern chip lithography.
So:
1.) can an AI use synthetic bio as a central ingredient of a plan to wipe out humanity?
Sure.
2.) can an AI use synthetic bio or chip litho a central ingredient of a plan to operate perpetually in a world without humans?
Another merit of the OP might be in pointing out bullshit by Eliezer Yudkowsky/Eric Drexler?
It’s kind of unfortunate if key early figures in the rationalist community introduce some bullshit to the memespace and we never get around to purging it and end up tanking our reputation by regularly appealing to it. Having this sort of post around helps get rid of it.
(3) could an AI that is developing nanotech without paying attention to the full range of consequences accidentally develop a form of nanotech that is devastating to humanity
(Imagine if e.g. there is some nanotech that does something useful but also creates long-lasting poisonous pollution as a side-effect, for instance.)
I.e. is it sufficient safety that the AI isn’t trying to kill us with nanotech? Or must it also be trying to not kill us?
Also worth noting w.r.t this that an AI that is leaning on bio-like nano is not one that can reliably maintain control over its own goals—it will have to gamble a lot more with evolutionary dynamics than many scenarios seem to imply meaning: - instrumental goal convergence more likely - paperclippers more unlikely
So again, tabooing magical nano has a big impact on a lot of scenarios widely discussed.
I don’t understand why evolution has anything to do with what I wrote.
Evolution designed a genome, and then the genome (plus womb etc.) builds a brain.
By the same token, it’s possible that a future AI could design a genome (or genome-like thing), and then that genome could build a brain. RIght?
Hmm, I guess a related point is that an AI wanting to take over the world probably needs to be able to either make lots of (ideally exact) copies of itself or solve the alignment problem w.r.t. its successors. And the former is maybe infeasible for a bio-like brain-ish thing in a vat. But not necessarily. And anyway, it might be also infeasible for a non-bio-like computer made from self-assembling nanobots or whatever. So I still don’t really care.
(2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?
In the ‘magical nano exists’ universe, the AI can do this with well-behaved nanofactories.
In the ‘bio-like nano’ universe, ‘evolutionary dynamics’ (aka game theory among replicators under high brownian noise) will make ‘operate perpetually’ a shaky proposal for any entity that values its goals and identity. No-one ‘operates perpetually’ under high noise, goals and identity are constantly evolving.
So the answer to the question is likely ‘no’—you need to drop some constraints on ‘an AI’ or ‘operate perpetually’.
Before you say ‘I don’t care, we all die anyway’—maybe you don’t, but many people (myself included) do care rather a lot about who kills us and why and what they do afterwards.
ME: Imagine a world with chips similar to today’s chips, and robots similar to humans, and no other nano magic. With enough chips and enough robots, such a system could operate perpetually, right? Just as human society does.
THEM: OK sure that could happen but not until there are millions or even billions of human-level robots, because chips are very hard to fabricate, like you need to staff all these high-purity chemical factories and mines and thousands of companies manufacturing precision equipment for the fab etc.
ME: I don’t agree with “millions or even billions”, but I’ll concede that claim for the sake of argument. OK fine, let’s replace the “chips” (top-down nano) with “brains-in-vats” (self-assembling nano). The vats are in a big warehouse with robots supplying nutrients. Each brain-in-vat is grown via a carefully controlled process that starts with a genome (or genome-like thing) that is synthesized in a DNA-synthesis machine and quadruple-checked for errors. Now the infrastructure requirements are much smaller.
~~
OK, so now in this story, do you agree that evolution is not particularly relevant? Like, I guess a brain-in-a-vat might get cancer, if the AI can’t get DNA replication error rates dramatically lower than it is in humans (I imagine it could, because its tradeoffs are different), but I don’t think that’s what you were talking about. A brain-in-a-vat with cancer is not a risk to the AI itself, it could just dump the vat and start over.
(This story does require that the AI solves the alignment problem with respect to the brains-in-vats.)
If you construct a hypothetical wherein there is obviously no space for evolutionary dynamics, then yes, evolutionary dynamics are unlikely to play a big role.
The case I was thinking of (which would likely be part of the research process towards ‘brains in vats’—essentially a prerequisit) is larger and larger collectives of designed organisms, forming tissues etc.
It may be possible to design a functioning brain in a vat from the ground up with no evolution, but I imagine that
a) you would get there faster verifying hypotheses with in vitro experiments
b) by the time you got to brains-in-vats, you would be able to make lots of other, smaller scale designed organisms that could do interesting, useful things as large assemblies
And since you have to pay a high price for error correction, the group that is more willing to gamble with evolutionary dynamics will likely have MVOs ready to deploy sooner that the one that insists on stripping all the evolutionary dynamics out of their setup.
(1) can an AI use nanotech as a central ingredient of a plan to wipe out humanity, and
(2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?
Given the hard limitations on dry nanotech, and pretty underwhelming power of wet nanotech/biotech, both answers should be ”...Eh.”
We have no plausible evidence that any kind of efficent nanotech that could be used for a Gray Goo scenario is possible, and this post is one of the many arguments against it.
If we focus only on completely plausible versions of nanotech, the worst case scenario is a the AI creating a “blight” that could very, very, very slowly damage our agriculture, cause disease in humans, and expand the AI’s influence, on the scale of decades or centuries. There is no plausible way to make an exponentially growing nanite cloud that would wipe us out and assemble into an AI God, the worst case scenario is an upjumped artificial slime mold that slowly creeps over everything, and can be fended off with a dustpan.
If an AI arranged to release a highly-contagious deadly engineered pathogen in an international airport, it would not take “decades or centuries” to spread. Right????
a pathogen is not grey nanotech, but biotech. And while it would be very, very dangerous, there is no plausible way for it to wipe out humanity. We already have highly-contagious deadly pathogens all over the planet, and they are sluggish to spread, and their deadliness is inverse to their contagiousness for obvious reasons (dead men don’t travel very well).
I’m finding this conversation frustrating. It seems to me that your grandparent comment was specifically talking about biotech & pandemics. For example, you said “wet nanotech/biotech”. And then in that context you said “”blight” that could very, very, very slowly damage our agriculture, cause disease in humans, and expand the AI’s influence, on the scale of decades or centuries”. This sure sounds to me like a claim that a novel pandemic would spread over the course of decades or centuries. Right? And such a claim is patently absurd. It did not take decades or centuries for COVID to spread around the world. (Even before mass air travel, it did not take decades or centuries for Spanish Flu to spread around the world.) Instead of acknowledging that mistake, your response is “a pathogen is not grey nanotech, but biotech”, which is missing the point—I was disputing a claim that you made about biotech.
their deadliness is inverse to their contagiousness for obvious reasons (dead men don’t travel very well).
Famously, when you catch COVID, you can become infectious a day or two before you become symptomatic. (That’s why it was so hard to contain.) And COVID also could cause nerve-damage that presumably had nothing to do with its ability to spread. More generally, it seems perfectly possible for a disease to have a highly-contagious-but-not-too-damaging early phase and then a few days later it turns lethal, perhaps by spreading into a totally different part of the body. So I strongly disbelieve the claim that deadliness and contagiousness of engineered pathogens are inevitably inverse, let alone that this is “obvious”.
sorry if the thread of my comment got messy, I did mention somewhere that COVID-like pathogen would likely be worst case scenario, for the reasons you mentioned above (long incubation).
However, I believe that COVID pandemic actually proves that humanity is robust against such threats. Quarantine worked. Masks worked. Vaccines worked. Soap and disinfectant worked. As human response would scale up with the danger inherent in any pandemic, I think that anything significantly more deadly that COVID would be stopped even faster, due to far more draconian quarantine responses.
With those in place, I do not see how a pathogen could be used to “wipe out humanity”. Decimate, yes. Annihilate? No.
But as I agreed in another thread, we should cut that conversation now. Discussing this online is literally feeding ideas to our potential enemy (be it AI or misaligned humans).
fair enough. most countries’ responses left a lot to be desired. a few countries that are general known for having their act together overall did for covid too, but it didn’t include some critical large population countries.
In the context of AI x-risk, I’m mainly interested in
(1) can an AI use nanotech as a central ingredient of a plan to wipe out humanity, and
(2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?
[(2) is obviously possible once you have a few billion human-level-intelligent robots, but the question is “can nanotech dramatically reduce the amount of time that the AI is relying on human help, compared to that baseline?”. Presumably “being able to make arbitrarily more chips or chip-equivalents” would be the most difficult ingredient.]
In both cases it seems to me that the answer is “obviously yes”:
super-plagues / crop diseases / etc. are an existence proof for (1),
human brains are an existence proof for (2).
Therefore grey goo as defined in this post doesn’t seem too relevant for my AI-related questions. Like, if the AI doesn’t have a plan to make nanotech things that can exterminate / outcompete microbes living in rocks deep under the seafloor—man, I just don’t care.
None of this is meant to be a criticism of this post, which I’m glad exists, even if I’m not in a position to evaluate it. Indeed, I’m not even sure OP would disagree with my comment here (based on their main AI post).
The merit of this post is to taboo nanotech. Practical bottom-up nanotech is simply synthetic biology, and practical top-down nanotech is simply modern chip lithography. So:
Sure.
Sure
But doesn’t sound as exciting? Good.
Another merit of the OP might be in pointing out bullshit by Eliezer Yudkowsky/Eric Drexler?
It’s kind of unfortunate if key early figures in the rationalist community introduce some bullshit to the memespace and we never get around to purging it and end up tanking our reputation by regularly appealing to it. Having this sort of post around helps get rid of it.
I’d also be interested in:
(3) could an AI that is developing nanotech without paying attention to the full range of consequences accidentally develop a form of nanotech that is devastating to humanity
(Imagine if e.g. there is some nanotech that does something useful but also creates long-lasting poisonous pollution as a side-effect, for instance.)
I.e. is it sufficient safety that the AI isn’t trying to kill us with nanotech? Or must it also be trying to not kill us?
Also worth noting w.r.t this that an AI that is leaning on bio-like nano is not one that can reliably maintain control over its own goals—it will have to gamble a lot more with evolutionary dynamics than many scenarios seem to imply meaning:
- instrumental goal convergence more likely
- paperclippers more unlikely
So again, tabooing magical nano has a big impact on a lot of scenarios widely discussed.
I don’t understand why evolution has anything to do with what I wrote.
Evolution designed a genome, and then the genome (plus womb etc.) builds a brain.
By the same token, it’s possible that a future AI could design a genome (or genome-like thing), and then that genome could build a brain. RIght?
Hmm, I guess a related point is that an AI wanting to take over the world probably needs to be able to either make lots of (ideally exact) copies of itself or solve the alignment problem w.r.t. its successors. And the former is maybe infeasible for a bio-like brain-ish thing in a vat. But not necessarily. And anyway, it might be also infeasible for a non-bio-like computer made from self-assembling nanobots or whatever. So I still don’t really care.
In the ‘magical nano exists’ universe, the AI can do this with well-behaved nanofactories.
In the ‘bio-like nano’ universe, ‘evolutionary dynamics’ (aka game theory among replicators under high brownian noise) will make ‘operate perpetually’ a shaky proposal for any entity that values its goals and identity. No-one ‘operates perpetually’ under high noise, goals and identity are constantly evolving.
So the answer to the question is likely ‘no’—you need to drop some constraints on ‘an AI’ or ‘operate perpetually’.
Before you say ‘I don’t care, we all die anyway’—maybe you don’t, but many people (myself included) do care rather a lot about who kills us and why and what they do afterwards.
I’m imagining an exchange like this.
ME: Imagine a world with chips similar to today’s chips, and robots similar to humans, and no other nano magic. With enough chips and enough robots, such a system could operate perpetually, right? Just as human society does.
THEM: OK sure that could happen but not until there are millions or even billions of human-level robots, because chips are very hard to fabricate, like you need to staff all these high-purity chemical factories and mines and thousands of companies manufacturing precision equipment for the fab etc.
ME: I don’t agree with “millions or even billions”, but I’ll concede that claim for the sake of argument. OK fine, let’s replace the “chips” (top-down nano) with “brains-in-vats” (self-assembling nano). The vats are in a big warehouse with robots supplying nutrients. Each brain-in-vat is grown via a carefully controlled process that starts with a genome (or genome-like thing) that is synthesized in a DNA-synthesis machine and quadruple-checked for errors. Now the infrastructure requirements are much smaller.
~~
OK, so now in this story, do you agree that evolution is not particularly relevant? Like, I guess a brain-in-a-vat might get cancer, if the AI can’t get DNA replication error rates dramatically lower than it is in humans (I imagine it could, because its tradeoffs are different), but I don’t think that’s what you were talking about. A brain-in-a-vat with cancer is not a risk to the AI itself, it could just dump the vat and start over.
(This story does require that the AI solves the alignment problem with respect to the brains-in-vats.)
If you construct a hypothetical wherein there is obviously no space for evolutionary dynamics, then yes, evolutionary dynamics are unlikely to play a big role.
The case I was thinking of (which would likely be part of the research process towards ‘brains in vats’—essentially a prerequisit) is larger and larger collectives of designed organisms, forming tissues etc.
It may be possible to design a functioning brain in a vat from the ground up with no evolution, but I imagine that
a) you would get there faster verifying hypotheses with in vitro experiments
b) by the time you got to brains-in-vats, you would be able to make lots of other, smaller scale designed organisms that could do interesting, useful things as large assemblies
And since you have to pay a high price for error correction, the group that is more willing to gamble with evolutionary dynamics will likely have MVOs ready to deploy sooner that the one that insists on stripping all the evolutionary dynamics out of their setup.
(1) can an AI use nanotech as a central ingredient of a plan to wipe out humanity, and
(2) can an AI use nanotech as a central ingredient of a plan to operate perpetually in a world without humans?
Given the hard limitations on dry nanotech, and pretty underwhelming power of wet nanotech/biotech, both answers should be ”...Eh.”
We have no plausible evidence that any kind of efficent nanotech that could be used for a Gray Goo scenario is possible, and this post is one of the many arguments against it.
If we focus only on completely plausible versions of nanotech, the worst case scenario is a the AI creating a “blight” that could very, very, very slowly damage our agriculture, cause disease in humans, and expand the AI’s influence, on the scale of decades or centuries. There is no plausible way to make an exponentially growing nanite cloud that would wipe us out and assemble into an AI God, the worst case scenario is an upjumped artificial slime mold that slowly creeps over everything, and can be fended off with a dustpan.
If an AI arranged to release a highly-contagious deadly engineered pathogen in an international airport, it would not take “decades or centuries” to spread. Right????
a pathogen is not grey nanotech, but biotech. And while it would be very, very dangerous, there is no plausible way for it to wipe out humanity. We already have highly-contagious deadly pathogens all over the planet, and they are sluggish to spread, and their deadliness is inverse to their contagiousness for obvious reasons (dead men don’t travel very well).
I’m finding this conversation frustrating. It seems to me that your grandparent comment was specifically talking about biotech & pandemics. For example, you said “wet nanotech/biotech”. And then in that context you said “”blight” that could very, very, very slowly damage our agriculture, cause disease in humans, and expand the AI’s influence, on the scale of decades or centuries”. This sure sounds to me like a claim that a novel pandemic would spread over the course of decades or centuries. Right? And such a claim is patently absurd. It did not take decades or centuries for COVID to spread around the world. (Even before mass air travel, it did not take decades or centuries for Spanish Flu to spread around the world.) Instead of acknowledging that mistake, your response is “a pathogen is not grey nanotech, but biotech”, which is missing the point—I was disputing a claim that you made about biotech.
Famously, when you catch COVID, you can become infectious a day or two before you become symptomatic. (That’s why it was so hard to contain.) And COVID also could cause nerve-damage that presumably had nothing to do with its ability to spread. More generally, it seems perfectly possible for a disease to have a highly-contagious-but-not-too-damaging early phase and then a few days later it turns lethal, perhaps by spreading into a totally different part of the body. So I strongly disbelieve the claim that deadliness and contagiousness of engineered pathogens are inevitably inverse, let alone that this is “obvious”.
I also suggest reading this article.
sorry if the thread of my comment got messy, I did mention somewhere that COVID-like pathogen would likely be worst case scenario, for the reasons you mentioned above (long incubation).
However, I believe that COVID pandemic actually proves that humanity is robust against such threats. Quarantine worked. Masks worked. Vaccines worked. Soap and disinfectant worked. As human response would scale up with the danger inherent in any pandemic, I think that anything significantly more deadly that COVID would be stopped even faster, due to far more draconian quarantine responses.
With those in place, I do not see how a pathogen could be used to “wipe out humanity”. Decimate, yes. Annihilate? No.
But as I agreed in another thread, we should cut that conversation now. Discussing this online is literally feeding ideas to our potential enemy (be it AI or misaligned humans).
did we live through the same pandemic?
we very likely did not, given the span of it, and various national responses.
fair enough. most countries’ responses left a lot to be desired. a few countries that are general known for having their act together overall did for covid too, but it didn’t include some critical large population countries.