Idea for a psychology study with the goal of defeating confirmations bias:
I think that confirmation bias (and all the biases it encompasses) is (one) of the most evil cognitive biases because it essentially causes peoples opinions to diverge (pretty much independent of what the evidence says), which causes society to be unable to pursue the best path of action.
Early studies showed the existence of confirmation bias by providing people with the same evidence (e.g. about nuclear power accidents) and observing that most people get even more convinced of their previously held opinions.
In real life when people change their minds, it’s often because a friend get’s them too engage with the other view in a one to one conversation. It might be possible to reproduce this effect with LLM’s.
The idea of the study is to give people evidence about a topic just like the early studies then let half of those people talk to a well prompted LLM for 10 minutes afterwards. The LLM tries to get people to really engage with the other view(s) on the topic.
I could imagine that this could reduce or maybe even cancel out the effect of confirmation bias, but it’s a long shot. If true though, this would provide a scalable way for reducing polarization and improving decisionmaking (on important topics). I imagine you could have such a chat window under every online article. Also it could give whoever decides to do this a chance of publishing in a prestigious journal.
Projects like this (that improve decisionmaking / cooperation) might be helpful in Gradual Disempowerment type scenarios.
Maybe someone here knows somebody who could do such a study.
I got the idea from this post.
JMaar
Karma: 22
One mitigation that seems good to me is to train on honeypot RL environments, where reward hacking is possible (e.g. there’s a security flaw, the environment doesn’t work as it is supposed to, etc.) and models are rewarded if they report this. Giving models the ability to report these things during training would probably also help with cleaning up the non-honeypot environments. I think this could also get rid of a bunch of mundane misalignment failure modes.
Does that make sense?
The biggest counterargument I see is that RL environments might be so fucked that this would slow training a lot (models would report bugs the whole time instead of doing their tasks) and would therefore be unviable for labs. Even in that case, it would be possible to do one training run this way that progresses very slowly and leads to a lot of fixes in every environment, while still doing other training runs simultaneously.