So, the interesting part is that it’s not enough that they’re a better source of raw material (even if they were) and better for optimizing (even if they were), because travelling to those planets also costs something.
I can think of reasons why any of those might be more or less desirable than the characteristics of Earth
So, we would need specific evidence that would cut one way but not another. If we can explain AI choosing another planet over Earth as well as we can explain it choosing Earth over another planet, we have zero knowledge.
2. This is an interesting point. I thought at first that it can simply set it up to keep synchronizing the probe with itself, so that it would be a single redundantly run process, rather than another agent. But that would involve always having to shut down periodically (so that the other half could be active for a while). But it’s plausible it would be confident enough in simply creating its copy and choosing not to modify the relevant parts of its utility function without some sort of handshake or metaprocedure. It definitely doesn’t sound like something that it would have to wait to completely solve alignment for.
3. That would give us a brief window during which humans would be tricked into or forced to work for an unaligned AI, after which it would kill us all.
So, the interesting part is that it’s not enough that they’re a better source of raw material (even if they were) and better for optimizing (even if they were), because travelling to those planets also costs something.
So, we would need specific evidence that would cut one way but not another. If we can explain AI choosing another planet over Earth as well as we can explain it choosing Earth over another planet, we have zero knowledge.
2. This is an interesting point. I thought at first that it can simply set it up to keep synchronizing the probe with itself, so that it would be a single redundantly run process, rather than another agent. But that would involve always having to shut down periodically (so that the other half could be active for a while). But it’s plausible it would be confident enough in simply creating its copy and choosing not to modify the relevant parts of its utility function without some sort of handshake or metaprocedure. It definitely doesn’t sound like something that it would have to wait to completely solve alignment for.
3. That would give us a brief window during which humans would be tricked into or forced to work for an unaligned AI, after which it would kill us all.