I see your point there. But I think this discussion sort of went in an irrelevant direction, albeit probably my fault for not being clear enough. When I put “powerful enough to destroy humanity” in that criterion, I mainly meant “powerful” as in “really powerful optimization process”, mathematical optimization power, not “power” as in direct influence over the world. We’re inferring that the former will usually lead fairly easily to the latter, but they are not identical. So “powerful enough to destroy humanity” would mean something like “powerful enough to figure out a good subjunctive plan to do so given enough information about the world, even if it has no output streams and is kept in an airtight safe at the bottom of the ocean”.
I see your point there. But I think this discussion sort of went in an irrelevant direction, albeit probably my fault for not being clear enough. When I put “powerful enough to destroy humanity” in that criterion, I mainly meant “powerful” as in “really powerful optimization process”, mathematical optimization power, not “power” as in direct influence over the world. We’re inferring that the former will usually lead fairly easily to the latter, but they are not identical. So “powerful enough to destroy humanity” would mean something like “powerful enough to figure out a good subjunctive plan to do so given enough information about the world, even if it has no output streams and is kept in an airtight safe at the bottom of the ocean”.