However, the notion that “avoid strength, attack weakness” is primarily a movie-plot-ism seems dubious to me.
That certainly would be dubious. Avoid strength, attack weakness is right behind ‘be a whole heap stronger’ as far as obvious universal strategies go.
Humans will perform software experiments trying to harness badly-understood technologies (ecosystems of self-modifying software agents, say). There will be some (epsilon) danger of paperclipping in this process. Humans will take precautions (lots of people have ideas for precautions that we could take). It is rational for them to take precautions, AND the precautions do not completely eliminate the chance of paperclipping, AND it is rational for them to forge ahead with the experiments despite the danger. During these experiments, people will gradually learn how the badly-understood technologies work, and transform them into much safer (and often much more effective) technologies.
If there are ways to make it possible to experiment and make small mistakes and minimise the risk of catastrophe then I am all in favour of using them. Working out which experiments are good ones to do so that people can learn from them and which ones will make everything dead is a non-trivial task that I’m quite glad to leave to someone else. Given that I suspect both caution and courage to lead to an unfortunately high probability of extinction I don’t envy them the responsibility.
AND it is rational for them to forge ahead with the experiments despite the danger.
Possibly. You can’t make that conclusion without knowing the epsilon in question and the alternatives to such experimentation. But there are times when it is rational to go ahead despite the danger.
The fate of most species is extinction. As the first intelligent agents, people can’t seriously expect our species to last for very long. Now that we have unleashed user-modifiable genetic materials on the planet, DNA’s days are surely numbered. Surely that’s a good thing. Today’s primitive and backwards biotechnology is a useless tangle of unmaintainable spaghetti code that leaves a trail of slime wherever it goes—who would want to preserve that?
That certainly would be dubious. Avoid strength, attack weakness is right behind ‘be a whole heap stronger’ as far as obvious universal strategies go.
If there are ways to make it possible to experiment and make small mistakes and minimise the risk of catastrophe then I am all in favour of using them. Working out which experiments are good ones to do so that people can learn from them and which ones will make everything dead is a non-trivial task that I’m quite glad to leave to someone else. Given that I suspect both caution and courage to lead to an unfortunately high probability of extinction I don’t envy them the responsibility.
Possibly. You can’t make that conclusion without knowing the epsilon in question and the alternatives to such experimentation. But there are times when it is rational to go ahead despite the danger.
The fate of most species is extinction. As the first intelligent agents, people can’t seriously expect our species to last for very long. Now that we have unleashed user-modifiable genetic materials on the planet, DNA’s days are surely numbered. Surely that’s a good thing. Today’s primitive and backwards biotechnology is a useless tangle of unmaintainable spaghetti code that leaves a trail of slime wherever it goes—who would want to preserve that?