I was recently explaining the concepts behind this story to a friend of mine, and like I’ve read before on here, sometimes you don’t really understand what you know until you think it all over again. As such, I found it amazing how great this story demonstrates how incredibly unsafe an arbitrary superintelligence is to humanity. When I first read this, I thought about the implications behind how “living in the Matrix” didn’t really mean a whole lot when you could essentially hack your way “out” into the rest of the universe. It helped explain how any part of the universe we have yet to discover is by necessity a part of the universe as well.
After discussing the future of AI with said friend, he would ask things like “what if the AI was simply confined inside a computer?” I explained how that wouldn’t work because said AI would find a way to send a message to someone who would unwittingly allow the AI to create itself some physical existence outside the computer and still take over. It truly is amazing how deep this story is when you understand a lot of what’s in the rest of the core sequences.
Be careful though, this story is a work of fiction. We must be careful not to generalize from fictional evidence. In particular, the enormous relative time differential between the machine intelligence and the outside world is completely unrealistic. It makes for good fiction and helps to illustrate a point, that we mustn’t assume that machine intelligence will develop at human scale. But that doesn’t mean the opposite extreme presented in this story is real either. Don’t update your beliefs based on this work of fiction.
I’d say that the primary belief-affecting value of this sort of fiction is to counteract the (common human) tendency to treat incredulity as evidence… the “I can’t easily think of how X could possibly be true, therefore X can’t be true” syndrome.
Agreed. I like the story, and I frequently point people to it. I just wouldn’t hold it up as evidence in an argument, unless the other person was specifically committing the fallacy of anthropomorphizing the speed of thought of a machine intelligence, in which case, yes, this story helps inoculate against the incredulity of alternatives.
Well, it wasn’t really this story that updated my beliefs. It was my beliefs that made me appreciate this story. I tend to interpret stories using my own understanding of reality, so oftentimes it’s simply just rationalization of what was presented (e.g., read what New Age people say about “quantum physics” and try to see if it fits at all with QFT, or at least how one could come to such mistaken beliefs). Demonstrating a hypothesis through a story helps in communicating the idea, that’s for sure.
I was recently explaining the concepts behind this story to a friend of mine, and like I’ve read before on here, sometimes you don’t really understand what you know until you think it all over again. As such, I found it amazing how great this story demonstrates how incredibly unsafe an arbitrary superintelligence is to humanity. When I first read this, I thought about the implications behind how “living in the Matrix” didn’t really mean a whole lot when you could essentially hack your way “out” into the rest of the universe. It helped explain how any part of the universe we have yet to discover is by necessity a part of the universe as well.
After discussing the future of AI with said friend, he would ask things like “what if the AI was simply confined inside a computer?” I explained how that wouldn’t work because said AI would find a way to send a message to someone who would unwittingly allow the AI to create itself some physical existence outside the computer and still take over. It truly is amazing how deep this story is when you understand a lot of what’s in the rest of the core sequences.
Be careful though, this story is a work of fiction. We must be careful not to generalize from fictional evidence. In particular, the enormous relative time differential between the machine intelligence and the outside world is completely unrealistic. It makes for good fiction and helps to illustrate a point, that we mustn’t assume that machine intelligence will develop at human scale. But that doesn’t mean the opposite extreme presented in this story is real either. Don’t update your beliefs based on this work of fiction.
I’d say that the primary belief-affecting value of this sort of fiction is to counteract the (common human) tendency to treat incredulity as evidence… the “I can’t easily think of how X could possibly be true, therefore X can’t be true” syndrome.
Agreed. I like the story, and I frequently point people to it. I just wouldn’t hold it up as evidence in an argument, unless the other person was specifically committing the fallacy of anthropomorphizing the speed of thought of a machine intelligence, in which case, yes, this story helps inoculate against the incredulity of alternatives.
Well, it wasn’t really this story that updated my beliefs. It was my beliefs that made me appreciate this story. I tend to interpret stories using my own understanding of reality, so oftentimes it’s simply just rationalization of what was presented (e.g., read what New Age people say about “quantum physics” and try to see if it fits at all with QFT, or at least how one could come to such mistaken beliefs). Demonstrating a hypothesis through a story helps in communicating the idea, that’s for sure.