“The level of “intelligence” (if you can call it that) you’re talking about with an AI whose able to draw up plans to destroy Earth (or the solar system), evade detection or convince humans to help it, actually enact its plans and survive the whole thing, is beyond the scope of realistic dreams for the first AI. It amounts to belief in a trickster deity, one which only FAI, the benevolent god, can save you from.”
It’s not necessarily the “first AI” as such. It’s the first AI capable of programming an AI smarter than itself that we’re worried about. Because that AI will make another, smarter one, and that one will make one smarter yet, and so on, until we end up with something that’s as smart as the laws of physics and local resources will allow.
Bit of sci-fi speculation here: What would “computronium” actually look like? It might very well look almost exactly like a star. If our sun actually was a giant computer, running some vastly complex calculation, would we here on Earth be able to tell?
It might very well look almost exactly like a star.
No, it won’t. The argument in favor of that is a strict upper bound, but there are far stricter upper bounds you can set, if you require things like the computer being capable of performing operations, or storing data.
“The level of “intelligence” (if you can call it that) you’re talking about with an AI whose able to draw up plans to destroy Earth (or the solar system), evade detection or convince humans to help it, actually enact its plans and survive the whole thing, is beyond the scope of realistic dreams for the first AI. It amounts to belief in a trickster deity, one which only FAI, the benevolent god, can save you from.”
It’s not necessarily the “first AI” as such. It’s the first AI capable of programming an AI smarter than itself that we’re worried about. Because that AI will make another, smarter one, and that one will make one smarter yet, and so on, until we end up with something that’s as smart as the laws of physics and local resources will allow.
Bit of sci-fi speculation here: What would “computronium” actually look like? It might very well look almost exactly like a star. If our sun actually was a giant computer, running some vastly complex calculation, would we here on Earth be able to tell?
No, it won’t. The argument in favor of that is a strict upper bound, but there are far stricter upper bounds you can set, if you require things like the computer being capable of performing operations, or storing data.