Experiment: Changing minds vs. preaching to the choir
1. Problem
In the market economy production is driven by monetary incentives – higher reward for an economic activity makes more people willing to engage in it. Internet forums follow the same principle but with a different currency—instead of money the main incentive of internet commenters is the reaction of their audience. A strong reaction expressed by a large number of replies or “likes” encourages commenters to increase their output. Its absence motivates them to quit posting or change their writing style.
On neutral topics, using audience reaction as an incentive works reasonably well: attention focuses on the most interesting or entertaining comments. However, on partisan issues, such incentives become counterproductive. Political forums and newspaper comment sections demonstrate the same patterns:
The easiest way to maximize “likes” for a given amount of effort is by posting an emotionally charged comment which appeals to audience’s biases (“preaching to the choir”).
The easiest way to maximize the number of replies is by posting a low quality comment that goes against audience’s biases (“trolling”).
Both effects are amplified when the website places comments with most replies or “likes” at the top of the page.
The data suggests that the easiest way to maximize the number of replies is to write posts that are disliked by most readers. For instance, articles with the karma of −1 on average generate twice as many comments (20.1±3.4) as articles with the karma of +1 (9.3±0.8).
2. Technical Solution
Enabling constructive discussion between people with different ideologies requires reversing the incentives – people need to be motivated to write posts that sound persuasive to the opposite side rather than to their own supporters.
We suggest addressing this problem that this problem by changing the voting system. In brief, instead of votes from all readers, comment ratings and position on the page should be based on votes from the opposite side only. For example, in the debate on minimum wage, for arguments against minimum wage only the upvotes of minimum wage supporters would be counted and vice versa.
The new voting system can simultaneously achieve several objectives:
· eliminate incentives for preaching to the choir
· give posters a more objective feedback on the impact of their contributions, helping them improve their writing style
· focus readers’ attention on comments most likely to change their minds instead of inciting comments that provoke an irrational defensive reaction.
3. Testing
If you are interested in measuring and improving your persuasive skills and would like to help others to do the same, you are invited to take part in the following experiment:
Step I. Submit Pro or Con arguments on any of the following topics (up to 3 arguments in total):
Should the government give all parents vouchers for private school tuition?
Should developed countries increase the number of immigrants they receive?
Should there be a government mandated minimum wage?
Step II. For each argument you have submitted, rate 15 arguments submitted by others.
Step III. Participants will be emailed the results of the experiment including:
- ratings their arguments receive from different reviewer groups (supporters, opponents and neutrals)
- the list of the most persuasive Pro & Con arguments on each topic (i.e. arguments that received the highest ratings from opposing and neutral groups)
- rating distribution in each group
Step IV (optional). If interested, sign up for the next round.
The experiment will help us test the effectiveness of the new voting system and develop the best format for its application.
You seem to be assuming commenters either agree or disagree with a post. But usually there are more than just two opposite or pro/con positions, and many more things to say on a subject than “yes” or “no”. Many posts aren’t arguing for or against something to begin with, they’re describing, reporting, or asking something.
Also, did you use the total number of comments on a post, or just the top-level comments? The former would count many extended discussions that often have little to do with the OP.
Is there a reason to think that the number of extended discussions that have little to do with the OP is higher for articles with negative karma? If not, counting the total number or just the top-level comments should not affect the conclusions.
Solving the problem for a simple binary case is a starting point in our tests.
If the number of extended discussions is uncorrelated with the post’s karma (except maybe for strongly downvoted posts), and the number of extended discussion comments dominates the number of total comments, then that is evidence that correlations between the number of total comments and the post’s karma are spurious.
But that simple case isn’t a representative or typical one...
If the number of extended discussions is uncorrelated with the post’s karma, then they would simply add a random noise component to the graph. I think it’s pretty obvious from the graph that the signal to noise ratio is quite high.
Evidence?
My own impressions. I’ve read LW regularly since it existed and I believe few posts describe a topic where there are mostly two opposite opinions or options. I haven’t done a quantitative analysis.
There are also LW (and allied) posts that argue such situations are abnormal, and usually come about due to motivated reasoning (including politics) or fallacies and biases. And I believe LWers mostly accept this and follow this approach. For instance, the posts related to color politics argue this point.
Karl Popper wrote at length over 60 years ago about the benefits of seeking out and attending to critics and the dangers of listening to those who agree. See ‘Conjectures and Refutations,’ ‘In Search of a Better World,’ or, well, most of his work.
Yep. If I want to make Less Wrong actually think, I start by saying something just slightly outrageous with a hint of annoying.
If you really want to maximize your audience, you do both—preach to one choir while infuriating the other(s).
At the moment Omnilibrium seems to give users only the aggregate of the votes and doesn’t give users per post feedback. I think that means that such a feedback circle doesn’t really exist at present.
The research on this website’s comments is interesting, but I think we can safely assume that on the average, on other websites this effect is much stronger. Here I’ve seen several times that people appreciate posts even if they don’t agree with them, if the arguments are well-presented. I think this happens much less often on other sites when dealing with ideological topics.
would it be possible to graph the length of post (word count)? if possible on each of the existing axies?
That would require a non-trivial amount of work. Is there a particular reason you are interested in this?
I am interested in the worthyness of a post as measured by the number of comments it gets, as well as the karma. Trying to squeeze some juice of metric out of the data we have available. For some reason I figured that it wouldn’t be a lot of work, people have made a word-counted VS karma before (but for comments). If it’s too much effort then don’t worry about it.
How do I do step II? I can’t seem to find the relevant debates. I found one debate with the same title as the minimum wage one I argued about, but I don’t see my argument appearing there.