You should rate highly people whose judgment you would trust when it differed from yours. We can use machine learning to find people who generate similar ratings to you, if the need arises.
I thought about the Slashdot thing, but I don’t think it makes the best use of people’s time. I’d like people reading only the innermost circle to be able to basically ignore the existence of the other circles. I don’t even want a prompt that says “7 hidden comments”.
You should rate highly people whose judgment you would trust when it differed from yours. We can use machine learning to find people who generate similar ratings to you, if the need arises.
It would be much harder to decide whose judgment I would trust, if I couldn’t see how they rated in the past. I’d have to do it only based on their general reputation and their past posts/comments, but what if some people write good comments but don’t rate the way I would prefer (for example they often downvote others who disagree with them)? The system would also essentially ignore ratings from lurkers, which seems wasteful.
If we use ML to find people who generate similar ratings, that seems to generate bad incentives. When your user rating is low, you’re incentivized to vote the same way as others, so that ML would pick you to recommend to people, then when your rating is high, you’d switch to voting based on your own opinions, which might be totally untrustworthy, but people who already rated you highly wouldn’t be able to tell that they should no longer trust you.
I thought about the Slashdot thing, but I don’t think it makes the best use of people’s time.
Aside from the issue of weird incentives I talked about earlier, I would personally prefer to have the option of viewing highly rated comments independent of parent ratings, since I’ve found those to be valuable to me in other systems (e.g., Slashdot and the current LW). Do you have an argument why that shouldn’t be allowed as a user setting?
It’s hard to be attack resistant and make good use of ratings from lurkers.
The issues you mention with ML are also issues with deciding who to trust based on how they vote, aren’t they?
It’s hard to make a strong argument for “shouldn’t be allowed as a user setting”. There’s an argument for documenting the API so people can write their own clients and do whatever they like. But you have to design the site around the defaults. Because of attention conservation, I think this should be the default, and that people should know that it’s the default when they comment.
The issues you mention with ML are also issues with deciding who to trust based on how they vote, aren’t they?
If everyone can see everyone else’s votes, then when someone who was previous highly rated starts voting in an untrustworthy manner, that would be detectable and the person can at least be down-rated by others who are paying attention. On the other hand, if we had a pure ML system (without any manual trust delegation) then when someone starts deviating from their previous voting patterns the ML algorithm can try to detect that and start discounting their votes. The problem I pointed out seems especially bad in a system where people can’t see others’ votes and depend on ML recommendations to pick who to rate highly, because then neither the humans nor ML can respond to someone changing their pattern of votes after getting a high rating.
You should rate highly people whose judgment you would trust when it differed from yours. We can use machine learning to find people who generate similar ratings to you, if the need arises.
I thought about the Slashdot thing, but I don’t think it makes the best use of people’s time. I’d like people reading only the innermost circle to be able to basically ignore the existence of the other circles. I don’t even want a prompt that says “7 hidden comments”.
It would be much harder to decide whose judgment I would trust, if I couldn’t see how they rated in the past. I’d have to do it only based on their general reputation and their past posts/comments, but what if some people write good comments but don’t rate the way I would prefer (for example they often downvote others who disagree with them)? The system would also essentially ignore ratings from lurkers, which seems wasteful.
If we use ML to find people who generate similar ratings, that seems to generate bad incentives. When your user rating is low, you’re incentivized to vote the same way as others, so that ML would pick you to recommend to people, then when your rating is high, you’d switch to voting based on your own opinions, which might be totally untrustworthy, but people who already rated you highly wouldn’t be able to tell that they should no longer trust you.
Aside from the issue of weird incentives I talked about earlier, I would personally prefer to have the option of viewing highly rated comments independent of parent ratings, since I’ve found those to be valuable to me in other systems (e.g., Slashdot and the current LW). Do you have an argument why that shouldn’t be allowed as a user setting?
It’s hard to be attack resistant and make good use of ratings from lurkers.
The issues you mention with ML are also issues with deciding who to trust based on how they vote, aren’t they?
It’s hard to make a strong argument for “shouldn’t be allowed as a user setting”. There’s an argument for documenting the API so people can write their own clients and do whatever they like. But you have to design the site around the defaults. Because of attention conservation, I think this should be the default, and that people should know that it’s the default when they comment.
If everyone can see everyone else’s votes, then when someone who was previous highly rated starts voting in an untrustworthy manner, that would be detectable and the person can at least be down-rated by others who are paying attention. On the other hand, if we had a pure ML system (without any manual trust delegation) then when someone starts deviating from their previous voting patterns the ML algorithm can try to detect that and start discounting their votes. The problem I pointed out seems especially bad in a system where people can’t see others’ votes and depend on ML recommendations to pick who to rate highly, because then neither the humans nor ML can respond to someone changing their pattern of votes after getting a high rating.