1) The AI would have to surprise us not just about the fact, but all observations therewith entangled. Eliezer_Yudkowsky mentioned in one comment the possibility of it telling us that humans have tails. Well, that sounds to me like a “dragon in the garage” scenario. What observation does this imply? Does the tail have mass and take up space? Is its blood flow connected to the rest of me? Does it hurt to cut it off?
2) For that reason, any surprise it tells us would have to be sufficiently disentangled from the rest of our observations. For example, imagine telling someone ALL of the steps needed to build a nuclear bomb in the year 1800, starting from technology that educated people already understand. That is how a surprise would have to seem, because people then weren’t yet capable of making observations that are obviously entangled with atomic science. Whether or not the design worked, they would have no way of knowing.
So an answer to this question would have to appear to us as a “cheat code”: something that you have to make a very unusual set of measurements (broadly defined) in order to notice. On that basis, one answer I would give to the question would be the “cognitive blind spot” common to all humans that can be exploited to make them do whatever you tell them. And that method would have to be something that people would never dream of doing. Not just “hey that would be morally wrong”, but “huh? That couldn’t work!”
Imagine something like those “hypnosis terrorists” that trick random people into giving them stuff, but much weirder, much more effective, and which results in the victims feeling good about whatever they were tricked into, all the rest of their lives, and showing all signs of happiness on all MRIs and future brainscan technologies when thinking about their acts. (I’ll post a link about hypnosis terrorists when I get a chance.)
If humans have some inherent flaw which makes them blind to the existence of their tails, then they must also have an inherent flaw which makes them blind to mass measurements that include the tail, and blood flow measurements that include the tail, and the source of pain being the tail when the tail is cut off, and the fact that they have designed pants with holes in them to fit the tail, etc. You end up postulating a whole series of inherent flaws permeating our ability to do very basic things. That’s plausible for stroke victims, but not really for humanity in general unless the entirety of modern medicine and science is fatally flawed.
Two things about this:
1) The AI would have to surprise us not just about the fact, but all observations therewith entangled. Eliezer_Yudkowsky mentioned in one comment the possibility of it telling us that humans have tails. Well, that sounds to me like a “dragon in the garage” scenario. What observation does this imply? Does the tail have mass and take up space? Is its blood flow connected to the rest of me? Does it hurt to cut it off?
2) For that reason, any surprise it tells us would have to be sufficiently disentangled from the rest of our observations. For example, imagine telling someone ALL of the steps needed to build a nuclear bomb in the year 1800, starting from technology that educated people already understand. That is how a surprise would have to seem, because people then weren’t yet capable of making observations that are obviously entangled with atomic science. Whether or not the design worked, they would have no way of knowing.
So an answer to this question would have to appear to us as a “cheat code”: something that you have to make a very unusual set of measurements (broadly defined) in order to notice. On that basis, one answer I would give to the question would be the “cognitive blind spot” common to all humans that can be exploited to make them do whatever you tell them. And that method would have to be something that people would never dream of doing. Not just “hey that would be morally wrong”, but “huh? That couldn’t work!”
Imagine something like those “hypnosis terrorists” that trick random people into giving them stuff, but much weirder, much more effective, and which results in the victims feeling good about whatever they were tricked into, all the rest of their lives, and showing all signs of happiness on all MRIs and future brainscan technologies when thinking about their acts. (I’ll post a link about hypnosis terrorists when I get a chance.)
Well, yeah. Obviously.
If humans have some inherent flaw which makes them blind to the existence of their tails, then they must also have an inherent flaw which makes them blind to mass measurements that include the tail, and blood flow measurements that include the tail, and the source of pain being the tail when the tail is cut off, and the fact that they have designed pants with holes in them to fit the tail, etc. You end up postulating a whole series of inherent flaws permeating our ability to do very basic things. That’s plausible for stroke victims, but not really for humanity in general unless the entirety of modern medicine and science is fatally flawed.
Alternately, you simply ignore said evidence.