I think Vygotsky’s expression “zone of proximal development” means “one inferential step away”, so in theory professional teachers should understand this. I prefer to imagine knowledge like a “tech tree” in a computer game.
When teaching one student, it is possible to detect their knowledge base and use their preferred vocabulary. I remember explaining some programming topics to a manager: source code is like a job specification; functions are employees; data are processed materials; exceptions are emergency plans.
Problem is, when teaching the whole class, everyone’s knowledge base is very different. In theory it shouldn’t be so, because they all supposedly learned the same things in recent years, but in reality there are huge differences—so the teacher basicly has to choose a subset of class as target audience. Writing a textbook is even more difficult, when there is no interaction.
I wonder what should Friendly AI do, when it discovers something that at first sight seems like a “crackpot belief” to its human operators. Let’s assume that the AI is far smarter than humans (and the “crackpot belief” requires many logical steps), but is still in a testing phase and humans don’t believe in its correctness.
If AI tells the discovery openly to humans, they will probably turn it off quickly, assuming there was something wrong in a program.
On the other hand, if the AI predicts that humans are not ready for this information, and tries to hide it, a security subroutine will detect that “AI wants to cheat its masters” and will force a shutdown. Even worse, if the AI decides that the right thing is telling the information to humans, but not right now, and instead give them first some “sequences” that will prepare them to accept the new information, and only give them the new information when they have changed their thinking… the security subroutine might still evaluate this as “AI wants to manipulate its masters” and force a shutdown.
Next question is, what would humans do. I am afraid that after receiving an unbelievable information X, they might simply add “not X” into AI axioms or values. It might seem like the rational thing to do; they will not think: “we don’t like X”, but rather: “X is a cognitive error, possibly one that AIs are prone to, so we should protect our AI against this cognitive error”.
As an example, imagine a world before the quantum physics was discovered; and imagine that AI discovered quantum physics and multiple universes—and gave this all info together to the unprepared humans. Now imagine some new discovery in future, possibly one hundred times less understandable to humans, with even more shocking consequences.