and then decide to copy itself onto biological computing or nanobots or whatever else strange options it can think of.
If artificial general intelligence moves to a completely non-artificial substrate at many nested levels of configuration (meaning in this case, a substrate configured like use from the proteins to the cells), then it would not be artificial anymore.
I am talking about wetware like us, not something made out of standardised components. So these new wetware-based configurations definitely would also not have the general capacities you might think they would have. It’s definitely not a copy of the AGI’s configurations.
If they are standardised in their configuration (like hardware), the substrate-needs convergence argument above definitely still applies.
The argument is about how general artificial intelligence, as defined, would converge if they continue to exist. I can see how that was not clear from the excerpt, because I did not move over this sentence: ”This is about the introduction of self-sufficient learning machinery, and of all modified versions thereof over time, into the world we humans live in.”
Intelligent engineering can already observed to work much faster than selection effects.
I get what you are coming from. Next to the speed of the design, maybe look at the *comprehensiveness* of the ‘design’.
Something you could consider spending more time thinking about is how natural selection works through the span of all physical interactions between (parts of) the organism and their connected surroundings. And top-down design does not.
For example, Eliezer brought up before how top-down design of an ‘eye’ wouldn’t have the retina sit back behind all that fleshy stuff that distorts light. A camera was designed much faster by humans. However, does a camera self-heal when it breaks like our eye does? Does a camera clean itself? And so on – to much fine-grained functional features of the eye.
And intelligence itself can be very robust to selection effects. Homomorphic-encryption and checkums
Yesterday, Anders Sandberg had a deep productive conversation about this with my mentor.
What is missing in your description is that the unidimensionality and simple direct causality of low-level error correction methods (eg. correcting bit flips) cannot be extrapolated to higher-level and more ambiguous abstractions (eg. correcting for viruses running over software, correcting for neural network hallucinations, correcting for interactive effects across automated machine production infrastructure).
These fall outside the limits of what the AGI’s actual built-in detection and correction methods could control for. > Would it?
Yes, because of the inequalities I explained in the longer post you read. I’ll leave it to the reader to do their own thinking to understand why.
As an alternative, an aligned superintelligent AI...
This is assuming the conclusion. If we could actually have an aligned AGI (let’s make the distinction), the evolutionary feedback effects cannot be sufficiently controlled for to stay aligned with internal reference values. The longer post explains why.
which has taken over the world can simply upload humans so they don’t die when the physical conditions become too bad.
Those “emulated humans” based on lossy scans of human brains, etc, wouldn’t be human anymore. You need to understand the fine-grained biological complexity involved.
I expect that an aligned superintelligence can come up with much better solutions than I can.
If you keep repeating the word ‘aligned’, it does not make it so. Saying it also does not make it less infeasible.
If there truly is no way at all for an aligned superintelligence to exist without humans dying, then (as I’ve mentioned before), it can just notice that and shut itself down…
How about we have a few careful human thinkers presently living, like Anders, actually spend the time to understand the arguments?
How about we not wager all life on Earth on the hope that “AGI” being developed on the basis of corporate competition and other selective forces would necessarily be orienting around understanding the arguments and then acting in a coherently aligned enough way to shut themselves down?
… after spending much-less-than-500-years rearranging the world into one that is headed towards a much better direction
I know this sounds just like an intellectual debate, but you’re playing with fire.
Thanks for your thoughts
If artificial general intelligence moves to a completely non-artificial substrate at many nested levels of configuration (meaning in this case, a substrate configured like use from the proteins to the cells), then it would not be artificial anymore.
I am talking about wetware like us, not something made out of standardised components. So these new wetware-based configurations definitely would also not have the general capacities you might think they would have. It’s definitely not a copy of the AGI’s configurations.
If they are standardised in their configuration (like hardware), the substrate-needs convergence argument above definitely still applies.
The argument is about how general artificial intelligence, as defined, would converge if they continue to exist. I can see how that was not clear from the excerpt, because I did not move over this sentence:
”This is about the introduction of self-sufficient learning machinery, and of all modified versions thereof over time, into the world we humans live in.”
I get what you are coming from. Next to the speed of the design, maybe look at the *comprehensiveness* of the ‘design’.
Something you could consider spending more time thinking about is how natural selection works through the span of all physical interactions between (parts of) the organism and their connected surroundings. And top-down design does not.
For example, Eliezer brought up before how top-down design of an ‘eye’ wouldn’t have the retina sit back behind all that fleshy stuff that distorts light. A camera was designed much faster by humans. However, does a camera self-heal when it breaks like our eye does? Does a camera clean itself? And so on – to much fine-grained functional features of the eye.
Yesterday, Anders Sandberg had a deep productive conversation about this with my mentor.
What is missing in your description is that the unidimensionality and simple direct causality of low-level error correction methods (eg. correcting bit flips) cannot be extrapolated to higher-level and more ambiguous abstractions (eg. correcting for viruses running over software, correcting for neural network hallucinations, correcting for interactive effects across automated machine production infrastructure).
Yes, because of the inequalities I explained in the longer post you read. I’ll leave it to the reader to do their own thinking to understand why.
This is assuming the conclusion.
If we could actually have an aligned AGI (let’s make the distinction), the evolutionary feedback effects cannot be sufficiently controlled for to stay aligned with internal reference values. The longer post explains why.
Those “emulated humans” based on lossy scans of human brains, etc, wouldn’t be human anymore.
You need to understand the fine-grained biological complexity involved.
If you keep repeating the word ‘aligned’, it does not make it so. Saying it also does not make it less infeasible.
How about we have a few careful human thinkers presently living, like Anders, actually spend the time to understand the arguments?
How about we not wager all life on Earth on the hope that “AGI” being developed on the basis of corporate competition and other selective forces would necessarily be orienting around understanding the arguments and then acting in a coherently aligned enough way to shut themselves down?
I know this sounds just like an intellectual debate, but you’re playing with fire.