I really hate the focus on karma. My preference would be just to use +1/-1 on posts, and not store totals at all (or to cap at 100 or so).
That said, it’s an interesting puzzle how to encourage the behaviors you want (and how to even describe the behaviors you want, in measurable terms). I think I want neutrally-OK comments (and posts) to be positive karma—more content is better, and I want to encourage participation. A zero-karma comment is an indication to me that I’ve wasted people’s time with no benefit at all.
Some problems you haven’t considered (or at least haven’t talked about) are:
1) how to distinguish between lack of votes and mixed up- and down-votes. If 100 people have viewed a comment and nobody has voted, it’s a boring but not harmful post. If it has 40 upvotes and 40 downvotes (or even +30, − 40), it’s a high-value comment that people care about.
2) relatedly, how to capture the value of simple existence of content that’s not good (or bad) enough to get votes. If 50 people have read a comment, and nobody has voted (or the votes balance out), it should still get some credit.
3) strategic voting. I’m far more likely to downvote a post or comment if it seems mediocre but has high karma than if it seems mediocre and has low karma. Same for upvoting—I don’t bother with things already upvoted by others.
4) Time. New posts/comments are going to have low karma because nobody’s had a chance to vote. New comments on old posts can be in this state for a long long time. For posters, it’s impossible to distinguish between an old-timer who’s made a whole lot of small-value comments and posts vs someone who’s made a few very-high value posts. This is even worse for karma from the old site, where the same content was worth 10x as a post than as a comment.
One idea might be for people to set level rather than direction for their votes—each user enters their desired level for a post/comment to reach, and their directional vote is always in the direction of that level. If I set a level of 20 when a comment is at 5, it’s an upvote. It remains an upvote unless the value goes above 20, when it becomes a downvote.
Also, make karma decay (perhaps not to 0, but maybe to 1000) so people active years ago don’t have unfixably-high karma today.
3) strategic voting. I’m far more likely to downvote a post or comment if it seems mediocre but has high karma than if it seems mediocre and has low karma. Same for upvoting—I don’t bother with things already upvoted by others.
I’ve noticed many people saying this, and I don’t see the value in voting that way. Your vote should carry information about your preferences about posts, not your preferences about the displayed information about the collective preferences about posts. That’s the best way to capture the collective preferences, isn’t it? As an example of the flaws of the latter perspective, changing the view order of a post can change its final karma, even though averages don’t have an order.
Edit: strategic voting in general seems similar to Defect: it gives the individual greater effectiveness at conveying their preferences, but makes the collective preference estimates less accurate. (It’s a negative-sum choice.)
What Dagon said. Your advice makes sense if the main signal people received is “this received one −5 vote, two −4 votes, one −1 vote, three +1 votes, and five +2 votes”, but not if people are just receiving a “net upvotes” summary number. By default, the aggregate effect of everyone trying to “vote according to what’s really in their heart” and disregard current vote totals is that either (a) lots of content gets absurdly, unwarrantedly high/low karma totals because people’s opinions are correlated, or (b) lots of content gets no upvotes or downvotes at all because people are trying to correct for the possibility that things will be over-voted (even though they can see with their own eyes whether a vote total is currently too high or too low).
Perhaps this is a reason to replace the “net upvotes” system with one that lists the number of votes (at different levels).
I briefly had it display totals of each vote type, and immediately found myself having a bad experience whenever I saw downvotes. Although I can still figure it out roughly from the list of total-number-of-votes and total-score, it felt fine instead of upsetting.
Not sure if that generalizes but it’s the reason we have the current configuration.
I suspect your reaction is a common one, but not universal and I have no way to guess whether it’s the majority. I STRONGLY prefer to see my downvotes, and at times have sought to make posts that were controversial (got both up- and down-votes) rather than popular.
lots of content gets absurdly, unwarrantedly high/low karma totals because people’s opinions are correlated
How is this absurd and unwarranted? The numerical value doesn’t have any inherent meaning (as it would if any posters who received at least 1000 karma were given moderating powers, ferex). Is it that it produces an unusual vote distribution? A possible solution would be to adjust total votes downward by a factor that increases with total votes, if this is a problem, but I disagree that any solution is needed.
“People’s opinions are correlated” is just another way of stating “many people agree about X”, and that sounds like something that the voting system should be able to record (perhaps the only thing—isn’t the voting system just a way of recording public opinion on a post?).
Votes determine comment order, which is a counterexample to my claim of “the numbers don’t really matter,” but that’s scale-invariant so my point holds. But perhaps low-value posts are inflated more than high-value posts? But if LW voters tend to upvote low-value posts, then I think there’s a larger issue that can’t be solved by people occasionally throwing a wrench into inflating post-votes (how do we know that the strategic downvotes will correlate with low-value posts when upvotes can’t do the same?).
lots of content gets no upvotes or downvotes at all because people are trying to correct for the possibility that things will be over-voted (even though they can see with their own eyes whether a vote total is currently too high or too low).
All the votes, past and future, are combined into one total. If someone aims for the final vote to be X, and their expected final vote is X, then why should they vote? That’s just rational strategic voting. (However, possible details that reverse the optimal decision: everybody acts the same way and nobody votes. But wouldn’t people realize that they all think that way? Alternatively, snowballing votes means that your vote could tip the final vote to either zero or larger than your wanted value, but only if people look at the total vote when deciding how to vote, which wouldn’t happen in this hypothetical scenario where people vote from their hearts.)
Your vote should carry information about your preferences about posts, not your preferences about the displayed information about the collective preferences about posts.
Nope. My vote is not an individual message to the poster. It’s simply a change to the total that the poster sees. I submit my vote as an direction, but it’s received as a sum.
Since I know that the signal received is a sum, I prefer to influence the sum rather than just picking a direction which may lead to a very different sum than I want.
I agree that people have preferences about vote sums, and that each individual’s preferences can be better realized through strategic voting. The crux is that I think that strategic voting worsens our ability to estimate the collective opinion of LW on a post. Strategic voters act to absorb votes past a certain point that they choose, which means that the added presence of non-strategic voters may not have any effect on the total vote. (Also note the order dependence, which probably indicates something wrong.) Perhaps we could accept these costs in exchange for some gain, but I don’t see what collective gain there is from strategic voting.
Perhaps we could accept these costs in exchange for some gain, but I don’t see what collective gain there is from strategic voting.
Vote total from strategic voting can better reflect a post or comment’s quality as opposed to its quality*readership (number of people who read a post/comment), which is what you would get if people did non-strategic voting. With the latter, it’s hard to tell whether a post’s vote total is high because people think it’s very high quality, or if it’s just moderately high quality but read (and hence voted on) by a lot of people.
I’ve changed my mind—I think strategic voting might send more information than karma-blind voting. It counteracts visibility spirals as you describe. There might also be another effect: consider a community of identical, deterministic, karma-blind voters. Disregarding visibility spirals, everything gets sorted into five categories (corresponding to the number of ways for a single user to vote). In reality, deterministic and karma-blind voters aren’t identical, so karma still varies smoothly. But is “people are different” the only way information should be sent? Doesn’t a group of identical voters hold more than a quint of useful information? This is why I have a vague suspicion that strategic voters can send more information—they send more information in a degenerate case.
The thing is that any system needs to be resilient against people strategic voting (with people having different goals and opinions about how to vote and why). Given that at least a nontrivial chunk of people vote this way, if we don’t want them to, it seems like the preferred solution is to change the voting system so that it no longer incentivizes that in the first place.
I agree with this statement, but I don’t think you have as much control as you like about people’s perception of votes, and what the incentive actually is for diverse individuals.
My incentives in voting are:
encourage more posts by new and not-yet-established posters. Getting someone into the low hundreds so they’re comfortable is a priority here.
encourage BETTER posts from a given poster, rather than encouraging more posts from some and fewer than others. How to do this is idiosyncratic based on what else the poster has posted and what reactions and comments have been given in the past. Usually comments and PMs are more effective than votes.
mark truly negative-value posts (spam and incoherent, as opposed to just imperfect) to get it noticed and removed by mods.
I think people perceive different types of vote systems fairly differently. I’m actually talking about bigger changes than you’re probably thinking of.
If I wanted to incentivize voters to “vote their true beliefs without regard for the rest of social consensus” (what Wakalix seems to be describing, and which I think is orthogonal to the set of things you’re talking about – you’re pointing at what posting behavior the votes should incentivize, and I think Wakalix is talking more about voting behavior), I would:
a) hide the karma before they vote
b) use something closer to a 1-5 star rating system (not precisely that, but closer). I think this would dramatically change the relationship with voting, and would much more naturally output the the voting behavior that Wakalix describes.
Strong-upvoted, mostly because I had a positive system-1 reaction to the line “I really hate the focus on karma.” I was going to straightforwardly agree with it, but then thought about it some more and came up with the following:
@ mostly the mod team: I’m not necessarily sure I grok why this site seems to care so much about karma, and I’m curious about it. I’m getting the impression that maybe it’s more important than I thought though, as a tool for guiding site culture. Like, every time I see a post talking about changing the karma system, my first thought is “Whoah, isn’t this way overthinking it? Why is this such an important issue?” Then I remind myself “oh, yeah, maybe it’s for shaping site culture, which is important I guess, so maybe this is important.” But then the next time I have the same system 1 reaction of “Why bother caring about karma so much?” Now I’m new here, and wasn’t around for the death of old-LW (I came in around the time LW2 started), so maybe this is just due to the fact that “LW dying” isn’t a particularly salient possibility for me, so I’m not worried so much about the nitty-gritty details of how to shape incentive gradients on the site so that doesn’t happen.
I would also say my intuitive reaction is that low-positive karma seems the right place for neutral comments. I’m not sure I like the “levels” idea, just because I don’t know how to determine what level I want a comment to be at on a scale that goes from -infinity to +infinity.
Appreciate spelling out your reasons and thoughts here.
I think, ideally, karma would fade into the background and not be something people overly worry about. BUT we still need a way to determine what posts get what level of visibility.
I think you mostly answer the question the way I would have – we think maintaining (and improving!) the site culture is really important, and this should ideally be as seamless a part of the site as possible.
I basically agree about low-positive karma being correct for what you see on a given comment. The question is about how the karma for that comment should translate into your longterm ability to influence the site. The easiest way to hand out higher-level privileges is automatically based on total karma. There are other ways one could think about this.
[Edit: I’d add that I see us as fortunate enough to have a dedicated userbase that cares about getting online discussion right, and this provides a rare an valuable opportunity to experiment.
Looking at most of the internet, you can see how technology shapes discussion. The sort of conversation that twitter incentivizes is different than what facebook incentivizes is different from what reddit incentivies – but all of those also share certain features by optimizing along a sort of “lowest common denominator” axis. Simple karma systems encourage posts with mass appeal, which isn’t necessarily the same as high-quality discussion.
LessWrong has the potential to deliberately engineer a platform and culture that is robustly focused on high quality discourse. Our approaches to this manifest as a lot of discussion of karma, but one of the underlying pieces of that is an approach to experimentation]
I really hate the focus on karma. My preference would be just to use +1/-1 on posts, and not store totals at all (or to cap at 100 or so).
That said, it’s an interesting puzzle how to encourage the behaviors you want (and how to even describe the behaviors you want, in measurable terms). I think I want neutrally-OK comments (and posts) to be positive karma—more content is better, and I want to encourage participation. A zero-karma comment is an indication to me that I’ve wasted people’s time with no benefit at all.
Some problems you haven’t considered (or at least haven’t talked about) are:
1) how to distinguish between lack of votes and mixed up- and down-votes. If 100 people have viewed a comment and nobody has voted, it’s a boring but not harmful post. If it has 40 upvotes and 40 downvotes (or even +30, − 40), it’s a high-value comment that people care about.
2) relatedly, how to capture the value of simple existence of content that’s not good (or bad) enough to get votes. If 50 people have read a comment, and nobody has voted (or the votes balance out), it should still get some credit.
3) strategic voting. I’m far more likely to downvote a post or comment if it seems mediocre but has high karma than if it seems mediocre and has low karma. Same for upvoting—I don’t bother with things already upvoted by others.
4) Time. New posts/comments are going to have low karma because nobody’s had a chance to vote. New comments on old posts can be in this state for a long long time. For posters, it’s impossible to distinguish between an old-timer who’s made a whole lot of small-value comments and posts vs someone who’s made a few very-high value posts. This is even worse for karma from the old site, where the same content was worth 10x as a post than as a comment.
One idea might be for people to set level rather than direction for their votes—each user enters their desired level for a post/comment to reach, and their directional vote is always in the direction of that level. If I set a level of 20 when a comment is at 5, it’s an upvote. It remains an upvote unless the value goes above 20, when it becomes a downvote.
Also, make karma decay (perhaps not to 0, but maybe to 1000) so people active years ago don’t have unfixably-high karma today.
I’ve noticed many people saying this, and I don’t see the value in voting that way. Your vote should carry information about your preferences about posts, not your preferences about the displayed information about the collective preferences about posts. That’s the best way to capture the collective preferences, isn’t it? As an example of the flaws of the latter perspective, changing the view order of a post can change its final karma, even though averages don’t have an order.
Edit: strategic voting in general seems similar to Defect: it gives the individual greater effectiveness at conveying their preferences, but makes the collective preference estimates less accurate. (It’s a negative-sum choice.)
What Dagon said. Your advice makes sense if the main signal people received is “this received one −5 vote, two −4 votes, one −1 vote, three +1 votes, and five +2 votes”, but not if people are just receiving a “net upvotes” summary number. By default, the aggregate effect of everyone trying to “vote according to what’s really in their heart” and disregard current vote totals is that either (a) lots of content gets absurdly, unwarrantedly high/low karma totals because people’s opinions are correlated, or (b) lots of content gets no upvotes or downvotes at all because people are trying to correct for the possibility that things will be over-voted (even though they can see with their own eyes whether a vote total is currently too high or too low).
Perhaps this is a reason to replace the “net upvotes” system with one that lists the number of votes (at different levels).
I briefly had it display totals of each vote type, and immediately found myself having a bad experience whenever I saw downvotes. Although I can still figure it out roughly from the list of total-number-of-votes and total-score, it felt fine instead of upsetting.
Not sure if that generalizes but it’s the reason we have the current configuration.
I suspect your reaction is a common one, but not universal and I have no way to guess whether it’s the majority. I STRONGLY prefer to see my downvotes, and at times have sought to make posts that were controversial (got both up- and down-votes) rather than popular.
How is this absurd and unwarranted? The numerical value doesn’t have any inherent meaning (as it would if any posters who received at least 1000 karma were given moderating powers, ferex). Is it that it produces an unusual vote distribution? A possible solution would be to adjust total votes downward by a factor that increases with total votes, if this is a problem, but I disagree that any solution is needed.
“People’s opinions are correlated” is just another way of stating “many people agree about X”, and that sounds like something that the voting system should be able to record (perhaps the only thing—isn’t the voting system just a way of recording public opinion on a post?).
Votes determine comment order, which is a counterexample to my claim of “the numbers don’t really matter,” but that’s scale-invariant so my point holds. But perhaps low-value posts are inflated more than high-value posts? But if LW voters tend to upvote low-value posts, then I think there’s a larger issue that can’t be solved by people occasionally throwing a wrench into inflating post-votes (how do we know that the strategic downvotes will correlate with low-value posts when upvotes can’t do the same?).
All the votes, past and future, are combined into one total. If someone aims for the final vote to be X, and their expected final vote is X, then why should they vote? That’s just rational strategic voting. (However, possible details that reverse the optimal decision: everybody acts the same way and nobody votes. But wouldn’t people realize that they all think that way? Alternatively, snowballing votes means that your vote could tip the final vote to either zero or larger than your wanted value, but only if people look at the total vote when deciding how to vote, which wouldn’t happen in this hypothetical scenario where people vote from their hearts.)
I second this. This is a good expression of the view that underlies a lot of what I’ve said on this topic.
Nope. My vote is not an individual message to the poster. It’s simply a change to the total that the poster sees. I submit my vote as an direction, but it’s received as a sum.
Since I know that the signal received is a sum, I prefer to influence the sum rather than just picking a direction which may lead to a very different sum than I want.
I agree that people have preferences about vote sums, and that each individual’s preferences can be better realized through strategic voting. The crux is that I think that strategic voting worsens our ability to estimate the collective opinion of LW on a post. Strategic voters act to absorb votes past a certain point that they choose, which means that the added presence of non-strategic voters may not have any effect on the total vote. (Also note the order dependence, which probably indicates something wrong.) Perhaps we could accept these costs in exchange for some gain, but I don’t see what collective gain there is from strategic voting.
Vote total from strategic voting can better reflect a post or comment’s quality as opposed to its quality*readership (number of people who read a post/comment), which is what you would get if people did non-strategic voting. With the latter, it’s hard to tell whether a post’s vote total is high because people think it’s very high quality, or if it’s just moderately high quality but read (and hence voted on) by a lot of people.
I’ve changed my mind—I think strategic voting might send more information than karma-blind voting. It counteracts visibility spirals as you describe. There might also be another effect: consider a community of identical, deterministic, karma-blind voters. Disregarding visibility spirals, everything gets sorted into five categories (corresponding to the number of ways for a single user to vote). In reality, deterministic and karma-blind voters aren’t identical, so karma still varies smoothly. But is “people are different” the only way information should be sent? Doesn’t a group of identical voters hold more than a quint of useful information? This is why I have a vague suspicion that strategic voters can send more information—they send more information in a degenerate case.
The thing is that any system needs to be resilient against people strategic voting (with people having different goals and opinions about how to vote and why). Given that at least a nontrivial chunk of people vote this way, if we don’t want them to, it seems like the preferred solution is to change the voting system so that it no longer incentivizes that in the first place.
I agree with this statement, but I don’t think you have as much control as you like about people’s perception of votes, and what the incentive actually is for diverse individuals.
My incentives in voting are:
encourage more posts by new and not-yet-established posters. Getting someone into the low hundreds so they’re comfortable is a priority here.
encourage BETTER posts from a given poster, rather than encouraging more posts from some and fewer than others. How to do this is idiosyncratic based on what else the poster has posted and what reactions and comments have been given in the past. Usually comments and PMs are more effective than votes.
mark truly negative-value posts (spam and incoherent, as opposed to just imperfect) to get it noticed and removed by mods.
I think people perceive different types of vote systems fairly differently. I’m actually talking about bigger changes than you’re probably thinking of.
If I wanted to incentivize voters to “vote their true beliefs without regard for the rest of social consensus” (what Wakalix seems to be describing, and which I think is orthogonal to the set of things you’re talking about – you’re pointing at what posting behavior the votes should incentivize, and I think Wakalix is talking more about voting behavior), I would:
a) hide the karma before they vote
b) use something closer to a 1-5 star rating system (not precisely that, but closer). I think this would dramatically change the relationship with voting, and would much more naturally output the the voting behavior that Wakalix describes.
Strong-upvoted, mostly because I had a positive system-1 reaction to the line “I really hate the focus on karma.” I was going to straightforwardly agree with it, but then thought about it some more and came up with the following:
@ mostly the mod team: I’m not necessarily sure I grok why this site seems to care so much about karma, and I’m curious about it. I’m getting the impression that maybe it’s more important than I thought though, as a tool for guiding site culture. Like, every time I see a post talking about changing the karma system, my first thought is “Whoah, isn’t this way overthinking it? Why is this such an important issue?” Then I remind myself “oh, yeah, maybe it’s for shaping site culture, which is important I guess, so maybe this is important.” But then the next time I have the same system 1 reaction of “Why bother caring about karma so much?” Now I’m new here, and wasn’t around for the death of old-LW (I came in around the time LW2 started), so maybe this is just due to the fact that “LW dying” isn’t a particularly salient possibility for me, so I’m not worried so much about the nitty-gritty details of how to shape incentive gradients on the site so that doesn’t happen.
I would also say my intuitive reaction is that low-positive karma seems the right place for neutral comments. I’m not sure I like the “levels” idea, just because I don’t know how to determine what level I want a comment to be at on a scale that goes from -infinity to +infinity.
Appreciate spelling out your reasons and thoughts here.
I think, ideally, karma would fade into the background and not be something people overly worry about. BUT we still need a way to determine what posts get what level of visibility.
I think you mostly answer the question the way I would have – we think maintaining (and improving!) the site culture is really important, and this should ideally be as seamless a part of the site as possible.
I basically agree about low-positive karma being correct for what you see on a given comment. The question is about how the karma for that comment should translate into your longterm ability to influence the site. The easiest way to hand out higher-level privileges is automatically based on total karma. There are other ways one could think about this.
[Edit: I’d add that I see us as fortunate enough to have a dedicated userbase that cares about getting online discussion right, and this provides a rare an valuable opportunity to experiment.
Looking at most of the internet, you can see how technology shapes discussion. The sort of conversation that twitter incentivizes is different than what facebook incentivizes is different from what reddit incentivies – but all of those also share certain features by optimizing along a sort of “lowest common denominator” axis. Simple karma systems encourage posts with mass appeal, which isn’t necessarily the same as high-quality discussion.
LessWrong has the potential to deliberately engineer a platform and culture that is robustly focused on high quality discourse. Our approaches to this manifest as a lot of discussion of karma, but one of the underlying pieces of that is an approach to experimentation]