An AI system replicating itself seems very unlikely because AI labs are hopefully and presumably protecting against that in particular. However, there are many other dangerous things an AI system could do that aren’t self-replication and are often worse. It also seems that if that does happen, we are doomed since AI labs are trying their hardest to prevent that, and if they fail to prevent that, we have a self-replicating non-aligned AI, and so we are screwed.
An AI system replicating itself seems very unlikely because AI labs are hopefully and presumably protecting against that in particular. However, there are many other dangerous things an AI system could do that aren’t self-replication and are often worse. It also seems that if that does happen, we are doomed since AI labs are trying their hardest to prevent that, and if they fail to prevent that, we have a self-replicating non-aligned AI, and so we are screwed.