Even granting that it is possible for cosmic rays to flip any given bit, or any sequence of bits, in a computer’s memory, it is far from clear to me that the probability of this happening approaches 1 over the lifetime of the universe. It isn’t very hard to come up with cases where an event is both completely possible, and has probability 0: for instance, if I pick a number at random with uniform distribution from the closed interval [0,1], the probability I will pick 1 is 0 even though 1 is as likely a choice as any other option on the interval. And in the concrete case you’re referring to, the universe has finite time to flip these bits before it sinks into entropy. Moreover, I wouldn’t expect the sequence of datapoints needed to convince an AI that humans are hostile (or whatever) to be invariant across time: as the AI accrued more data, it would plausibly require more data to persuade it to change its mind.
The claim was that, if there exists a bit, such that if that bit was struck by a cosmic ray, then for an agent which would be “safe” in a universe without cosmic rays, would become “unsafe” then, as cosmic rays exist, no agent may be “safe” with “probability 1″, as that would require it to not be stuck by cosmic rays with “probability 1”.
They’re saying we can’t be sure it won’t be hit by cosmic rays. This was meant not as a worry about cosmic rays, but to say they were interested in how you go about making “safe* agent/s” in a universe without inconvenient things like cosmic rays which keep the probability from being 1, but are otherwise unrelated to the work of making “safe agent/s”.
*Might be talking about things other than “safety” as well.
Even granting that it is possible for cosmic rays to flip any given bit, or any sequence of bits, in a computer’s memory, it is far from clear to me that the probability of this happening approaches 1 over the lifetime of the universe. It isn’t very hard to come up with cases where an event is both completely possible, and has probability 0: for instance, if I pick a number at random with uniform distribution from the closed interval [0,1], the probability I will pick 1 is 0 even though 1 is as likely a choice as any other option on the interval. And in the concrete case you’re referring to, the universe has finite time to flip these bits before it sinks into entropy. Moreover, I wouldn’t expect the sequence of datapoints needed to convince an AI that humans are hostile (or whatever) to be invariant across time: as the AI accrued more data, it would plausibly require more data to persuade it to change its mind.
Linking this, I meant “with probability strictly greater than 0, the agent is not safe”. Sorry for the confusion.
The claim was that, if there exists a bit, such that if that bit was struck by a cosmic ray, then for an agent which would be “safe” in a universe without cosmic rays, would become “unsafe” then, as cosmic rays exist, no agent may be “safe” with “probability 1″, as that would require it to not be stuck by cosmic rays with “probability 1”.
They’re saying we can’t be sure it won’t be hit by cosmic rays. This was meant not as a worry about cosmic rays, but to say they were interested in how you go about making “safe* agent/s” in a universe without inconvenient things like cosmic rays which keep the probability from being 1, but are otherwise unrelated to the work of making “safe agent/s”.
*Might be talking about things other than “safety” as well.