Measuring open-mindedness

Re­cently we have opened an ex­per­i­men­tal web­site for Ra­tional Dis­cus­sion of Poli­tics. A spe­cial fea­ture of the new web­site is an au­to­mated recom­men­da­tion sys­tem which stud­ies user prefer­ences based on their vot­ing records. The pur­pose of this fea­ture is to en­hance the qual­ity of dis­cus­sion with­out us­ing any form of cen­sor­ship.

The recom­men­da­tion sys­tem was pre­vi­ously tested with the help of 30 mem­bers of a poli­ti­cal dis­cus­sion fo­rum. The tests have shown that most user prefer­ences can be rea­son­ably well de­scribed by just two pa­ram­e­ters. The sys­tem chooses the pa­ram­e­ters (prin­ci­pal vec­tors) in­de­pen­dently based only on the nu­mer­i­cal data (com­ment rat­ings), but it was easy to see that one vec­tor cor­re­sponded to the “leftwing—rightwing” and an­other to the “well writ­ten – poorly writ­ten” axis.

About a month ago we started dis­cus­sions on the new web­site. This time, all our par­ti­ci­pants were LW mem­bers and the re­sults were very differ­ent. There was rel­a­tively lit­tle vari­a­tion along “well writ­ten – poorly writ­ten” axis. There was sig­nifi­cant vari­a­tion along what seemed to be the poli­ti­cal views axis, but it could no longer be perfectly de­scribed by the con­ven­tional “leftwing—rightwing” la­bels. For the mo­ment, we adopted “pop­u­lares” and “op­ti­mates” terms for the two camps (the former seems some­what cor­re­lated with “left-wing/​liberal” and the lat­ter with “right-wing/​liber­tar­ian”).

The re­sults have shown an in­ter­est­ing asym­me­try be­tween the camps. In the pre­vi­ous tests, both left and right lean­ing users up­voted users from their own camp much more fre­quently. How­ever, one group was sev­eral times more likely to up­vote their op­po­nents than the other. Among “pop­u­lares” and “op­ti­mates” the asym­me­try was a lot weaker (cur­rently 27%), but still no­tice­able.

In both cases our sam­ple sizes were small and may not be rep­re­sen­ta­tive of the LW com­mu­nity or the US pop­u­la­tion. Still, it would be in­ter­est­ing to find an ex­pla­na­tion for this asym­me­try. One pos­si­bil­ity is that, on av­er­age, one side pre­sents sig­nifi­cantly bet­ter ar­gu­ments. Another pos­si­bil­ity is that the other group is more open-minded.

Can any­one sug­gest a test that can ob­jec­tively de­cide which (if any) hy­poth­e­sis is cor­rect?