I agree that an AGI would need lots of trial and error to develop a major new technology.
I’m unsure whether an AGI would need to be as slow as humans about that trial and error. If it needs secrecy, that might be a big constraint. If it gets human cooperation, I’d expect it to improve significantly on human R&D speed.
I also see a nontrivial chance that humans will develop Drexlerian nanotech before full AGI.
Your post gets stranger toward the end.
I don’t see much value in a careful engineering analysis of how an AGI might kill us. Most likely it would involve a complex set of strategies, including plenty of manipulation, with no one attack producing certain victory by itself, but with humanity being overwhelmed by the number of different kinds of attack. There’s enough uncertainty in that kind of fight that I don’t expect to get a consensus on who would win. The uncertainty ought to be scary enough that we shouldn’t need to prove who would win.
Many kinds of uncertain attacks is not the strategy EY points at with his “diamondoid bacteria” idea. He’s worrying about a single undetectable attack with high chance of success using approaches that only an AGI can execute.
Others worry about an AGI using many attacks, but that is ultimately unbeatable.
Here’s you’re worrying about an AGI using many attacks that is beatable by humans, but not with confidence.
These are distinct arguments, and we should be clear about which one is being made and responded to.
“but with humanity being overwhelmed by the number of different kinds of attack.”
But AGI will only be able to start carrying out these sneaky attacks once its fairly convinced it can survive without human help? Otherwise humans will notice the various problems propping up and might just decide to “burn all GPUs” which is currently an unimaginable act.. So AGI will have to act sneakily behind the scenes for a very long time. This is again coming back to the argument that humans have a strong upper hand as long as we’ve got monopoly on physical world manipulation.
The beginning of this post seems fairly good.
I agree that an AGI would need lots of trial and error to develop a major new technology.
I’m unsure whether an AGI would need to be as slow as humans about that trial and error. If it needs secrecy, that might be a big constraint. If it gets human cooperation, I’d expect it to improve significantly on human R&D speed.
I also see a nontrivial chance that humans will develop Drexlerian nanotech before full AGI.
Your post gets stranger toward the end.
I don’t see much value in a careful engineering analysis of how an AGI might kill us. Most likely it would involve a complex set of strategies, including plenty of manipulation, with no one attack producing certain victory by itself, but with humanity being overwhelmed by the number of different kinds of attack. There’s enough uncertainty in that kind of fight that I don’t expect to get a consensus on who would win. The uncertainty ought to be scary enough that we shouldn’t need to prove who would win.
Many kinds of uncertain attacks is not the strategy EY points at with his “diamondoid bacteria” idea. He’s worrying about a single undetectable attack with high chance of success using approaches that only an AGI can execute.
Others worry about an AGI using many attacks, but that is ultimately unbeatable.
Here’s you’re worrying about an AGI using many attacks that is beatable by humans, but not with confidence.
These are distinct arguments, and we should be clear about which one is being made and responded to.
“but with humanity being overwhelmed by the number of different kinds of attack.”
But AGI will only be able to start carrying out these sneaky attacks once its fairly convinced it can survive without human help? Otherwise humans will notice the various problems propping up and might just decide to “burn all GPUs” which is currently an unimaginable act.. So AGI will have to act sneakily behind the scenes for a very long time. This is again coming back to the argument that humans have a strong upper hand as long as we’ve got monopoly on physical world manipulation.