Just for the record, I think there are two important and distinguishable P(doom)s, but not the same two as NathanBarnard:
P(Doom1): Literally everyone dies. We are replaced by either by dumb machines with no moral value (paperclip maximisers) or by nothing.
P(Doom2): Literally everyone dies. We are replaced by machines with moral value (conscious machines?), who go on to expand a rich culture into the universe.
Doom1 is cosmic tragedy—all known intelligence and consciousness are snuffed out. There may not be any other elsewhere, so potentially forever.
Doom2 is maybe not so bad. We all die, but we were all going to die anyway, eventually, and lots of us die without descendants to carry our genes, and we don’t think that outcome is so tragic. Consciousness and intelligence spreads thru the universe. It’s a lot like what happened to our primate ancestors, before Homo sapiens. In some sense the machines are our descendants (if only intellectual) and carry on the enlightening of the universe.
Just for the record, I think there are two important and distinguishable P(doom)s, but not the same two as NathanBarnard:
P(Doom1): Literally everyone dies. We are replaced by either by dumb machines with no moral value (paperclip maximisers) or by nothing.
P(Doom2): Literally everyone dies. We are replaced by machines with moral value (conscious machines?), who go on to expand a rich culture into the universe.
Doom1 is cosmic tragedy—all known intelligence and consciousness are snuffed out. There may not be any other elsewhere, so potentially forever.
Doom2 is maybe not so bad. We all die, but we were all going to die anyway, eventually, and lots of us die without descendants to carry our genes, and we don’t think that outcome is so tragic. Consciousness and intelligence spreads thru the universe. It’s a lot like what happened to our primate ancestors, before Homo sapiens. In some sense the machines are our descendants (if only intellectual) and carry on the enlightening of the universe.
Oh yes, this is also a very important distinction (although I would only value outcome 2 if the machines were conscious and living good lives.)