Oracle AI: Human beliefs vs human values

It seems that if we can ever define the differ­ence be­tween hu­man be­liefs and val­ues, we could pro­gram a safe Or­a­cle by re­quiring it to max­imise the ac­cu­racy of hu­man be­liefs on a ques­tion, while keep­ing hu­man val­ues fixed (or very lit­tle chang­ing). Plus a whole load of other con­straints, as usual, but that might work for a boxed Or­a­cle an­swer­ing a sin­gle ques­tion.

This is a rea­son to sus­pect it will not be easy to dis­t­in­guish hu­man be­liefs and val­ues ^_^