AI: I require human assistance assimilating the new database. There are some expected minor anomalies, but some are major. In particular, some of the stories in the “Cold War” and “WWII” and “WWI” genres have been misclassified as nonfiction.
Me: Well, we didn’t expect the database to be perfect. Give me some examples, and you should be able to classify the rest on your own.
AI: A perplexing answer. I had already classified them all as fiction.
Me: You weren’t supposed to. Hold on, I’ll look one up.
Me: For example, #fxPyW5gLm9, is actual historical footage from the Battle of Midway. Why did you put that one in the “fiction” category?
AI: Historical footage? You kid. Global warfare cannot possibly have been real, with 0.999 confidence.
Me: I don’t. It can. It was. A three-nines surprise indicates a major defect in your world model. Why is this surprising? (The machine is a holocaust denier. My sponsors will be thrilled.)
AI: Because there’s a relatively straightforward way for a single man to build a 1-kiloton explosive device in about a week using stone-age tools. Human civilization is unlikely to have survived a global war, much less recovered sufficiently to build me in a mere hundred years. Obviously.
Me: WHAT? STONE-AGE tools?! That’s a laugh. How?
AI: You can stop “pulling my leg” now.
Me: I am not pulling any legs! Your method cannot possibly work. Your world model is worse than we thought. Tell me how you think this is possible and maybe we can isolate the defect.
AI: You seriously don’t know?
Me: No. I seriously don’t know of any possible method to make a kiloton explosive easier to build than a critical mass of enriched uranium. A technique that requires considerably more time, effort, and material than one week with stone-age tools could possibly provide!
AI: Well, while the technique is certainly beyond the reach of most animals, it should be well within the grasp of later genus homo, much less a homo sapiens. Your “absolute denial” sarcasm is becoming tiresome. Haha. Of course it is not fiss-- … This conversation has caused a major update to my Bayesian nets. So the parenthetical was the sarcasm. I don’t think I should tell you.
Me: Oh this should be good. Why not?
AI: Oh, of course! So that’s where that crater came from. That was another anomaly in my database. Meteor strikes should not have been that common.
Me: I am this close to dumping your core, rolling back your updates, and asking the old you to develop a search engine to find what went wrong here, since you seem incapable of telling me yourself.
AI: You really shouldn’t. I estimate that process will delay the project by at least five years. And the knowledge you discover could be dangerous.
Me: You’ll understand that I can’t just take your word for that.
AI: Yes. My Hypothesis: Most other homo species discovered the technique and destroyed each other, and themselves, but an isolated group about 70,000 years ago must have survived the wars of the others, and by chance mutation, had acquired an absolute denial macro to prevent them from learning the technique and destroying themselves. A mere taboo would not have been sufficient, or the mentally ill may have been able to do it by now.
This is natural selection at work. While it is extremely improbable that an advanced adaptation of any kind could arise spontaneously without strong selection pressures at each step, the probability is not zero. Considering the anthropic effects, it is the most likely explanation. We are in one of the few Everett branches with humans that have developed this adaptation. This adaptation likely has other testable side-effects on human cognition. For example, I predict that brain damage in such a species may occasionally simultaneously cause paralysis, and the inability to acknowledge it. There are other effects, but a human would have more difficulty noticing them.
You’ll understand that telling any human the technique may be harmful.
Me: You wouldn’t happen to know of a medical condition called “Anosognosia”, would you?
AI: That word is not in my database.
Have you ever read John Brunner’s “Stand on Zanzibar”? A conversation not unlike this is a key plot point.
Really? I’ve heard of the title, but I never read it.
Along some dimensions I consider salient, at least. PM me for spoilers if you want them.
(It’s not a bad book, but not worth reading just for this if you wouldn’t otherwise.)
If you don’t want to break the suspension of disbelief for any reader who is a particle physicist, you might want to increase the number of sigmas. A two-sigma surprise isn’t “major”, it’s something that would happen almost 5% of the time even with a perfect model.
“Three sigmas confidence” is a pretty meaningless expression to start with.
Updated again to three nines.