Rather, I want the AI to be like a disagreeable nerd that wants us to have a good understanding, notices areas where we’re confused, and is brainstorming and strategizing on how to help set us straight by improving its clarity and pedagogy.
It seems like it is difficult to distinguish between the disagreeable nerd described above and the same exact type of person, but who’s trying to get you to have a good understanding of a conspiracy theory or religion (ie. something that they believe to be true, but is not). The conspiracy theorist or religious true believer is well intentioned, but ultimately wrong, and I don’t think we’d want AI to want us to have the traits of the disagreeable nerd you describe regarding topics that are ultimately false. But how do you know (from the inside) if you’re a disagreeable nerd or religious believer?
Similarly, the intent and feelings of the listener also seems important in the human-human cases—if the listener is actively seeking knowledge/confirmation of a false theory it seems at least less bad, and if the listener is actively uninterested/hostile to hearing about a true fact from the nerd, it seems more bad. So being able to include some consideration of the stance and level of interest (or lack thereof) of the listener also seems important.
Another human intuition on things like this involves how important the topic at hand is. It’s more acceptable to try really hard to convince someone of something, even if they don’t want to be convinced or even if you’re not 100% certain of it yourself, if the consequences of not changing their actions are severe enough.
It seems like it is difficult to distinguish between the disagreeable nerd described above and the same exact type of person, but who’s trying to get you to have a good understanding of a conspiracy theory or religion (ie. something that they believe to be true, but is not). The conspiracy theorist or religious true believer is well intentioned, but ultimately wrong, and I don’t think we’d want AI to want us to have the traits of the disagreeable nerd you describe regarding topics that are ultimately false. But how do you know (from the inside) if you’re a disagreeable nerd or religious believer?
Similarly, the intent and feelings of the listener also seems important in the human-human cases—if the listener is actively seeking knowledge/confirmation of a false theory it seems at least less bad, and if the listener is actively uninterested/hostile to hearing about a true fact from the nerd, it seems more bad. So being able to include some consideration of the stance and level of interest (or lack thereof) of the listener also seems important.
Another human intuition on things like this involves how important the topic at hand is. It’s more acceptable to try really hard to convince someone of something, even if they don’t want to be convinced or even if you’re not 100% certain of it yourself, if the consequences of not changing their actions are severe enough.