Here’s what I consider the relative merits of each Tag Relevance system. (Note: it’d be helpful if you read the OP and form your own opinion before reading this)
...
...
Karma-Style
Advantages:
It re-uses an existing system, which makes it easier to program, and easier to learn.
It’s very easy to understand how the sorting works – it very transparently sorts in descending relevance order.
Disadvantages:
It’s hard to move a recently-tagged post to the bottom of the list without removing it entirely, or upvoting every other post instead.
I personally find that it messes with my intuitions about how karma works. I find myself wanting to upvote posts I like in the usual way… but I already did that. While re-using an existing system is mechanically simpler, I personally find it requires more conceptual orientation.
My votes are often not about which posts are objectively relevant, but about which posts I think are more relevant than the current consensus suggests. This a) contributes to voting feeling unintuitive to me, and b) since my votes are very “current-context dependent”, I worry that later I’ll have to go and re-vote on things to make sure my vote is still doing what I want.
It’s fairly overwhelming for the tag page to list both “relevance” and “karma”, but if you only show one of those, it’s confusing in a different way (if you leave off karma, people may mistakenly think relevance is karma. If you leave off relevance, people will be confused about how the page is sorted)
A single user can’t set up an entire tag page in a reasonable fashion – if you’re adding a new post, you basically only have two choices, “smallVote” and “bigVote”, so if you want to add anything that’s only tangentially relevant, you have to have it mixed in with all the other smallVotes until another user comes along to flesh things out.
When only a couple people have voted on a Tag Page’s posts, I think the results are worse than just sorting by karma.
I expect many of these problems to mostly effect less-visited tags. More visited tags will probably end up with lots of users contributing, resulting in a “wisdom of crowds”. But I expect at least half of Tag Pages to be sparsely attended, and for the relevance votes to be much noisier than just sorting by karma.
Multiple Choice Voting Style
Advantages:
A single user can vote honestly based on their true beliefs about how relevant a tag is, adding a bunch of tags at once and accurately sorting them.
I personally find it more intuitive (I don’t have any conflicting intuitions ported over from karma)
The Tag Page becomes simpler – you don’t need to show relevance at all, you just use it to establish major categories.
It lets karma remain relevant, without being very confusing. (A few people have suggested “sort by relevance + karma”, but the problem is that makes relevance much more confusing, removing it’s clarity advantage)
Disadvantages:
It requires building a whole new system. This is nontrivial. (It’s probably only around a week of work, but there are lots of other things we could build in a week, and if most people find Karma Relevance pretty intuitive it may not be worth it)
Users have to learn a new system. I’m guessing that system wouldn’t be that hard to learn, but I think “median vote” is a somewhat more confusing concept than “sort by a number” (especially when you factor in vote weight)
Here’s what I consider the relative merits of each Tag Relevance system. (Note: it’d be helpful if you read the OP and form your own opinion before reading this)
...
...
Karma-Style
Advantages:
It re-uses an existing system, which makes it easier to program, and easier to learn.
It’s very easy to understand how the sorting works – it very transparently sorts in descending relevance order.
Disadvantages:
It’s hard to move a recently-tagged post to the bottom of the list without removing it entirely, or upvoting every other post instead.
I personally find that it messes with my intuitions about how karma works. I find myself wanting to upvote posts I like in the usual way… but I already did that. While re-using an existing system is mechanically simpler, I personally find it requires more conceptual orientation.
My votes are often not about which posts are objectively relevant, but about which posts I think are more relevant than the current consensus suggests. This a) contributes to voting feeling unintuitive to me, and b) since my votes are very “current-context dependent”, I worry that later I’ll have to go and re-vote on things to make sure my vote is still doing what I want.
It’s fairly overwhelming for the tag page to list both “relevance” and “karma”, but if you only show one of those, it’s confusing in a different way (if you leave off karma, people may mistakenly think relevance is karma. If you leave off relevance, people will be confused about how the page is sorted)
A single user can’t set up an entire tag page in a reasonable fashion – if you’re adding a new post, you basically only have two choices, “smallVote” and “bigVote”, so if you want to add anything that’s only tangentially relevant, you have to have it mixed in with all the other smallVotes until another user comes along to flesh things out.
When only a couple people have voted on a Tag Page’s posts, I think the results are worse than just sorting by karma.
I expect many of these problems to mostly effect less-visited tags. More visited tags will probably end up with lots of users contributing, resulting in a “wisdom of crowds”. But I expect at least half of Tag Pages to be sparsely attended, and for the relevance votes to be much noisier than just sorting by karma.
Multiple Choice Voting Style
Advantages:
A single user can vote honestly based on their true beliefs about how relevant a tag is, adding a bunch of tags at once and accurately sorting them.
I personally find it more intuitive (I don’t have any conflicting intuitions ported over from karma)
The Tag Page becomes simpler – you don’t need to show relevance at all, you just use it to establish major categories.
It lets karma remain relevant, without being very confusing. (A few people have suggested “sort by relevance + karma”, but the problem is that makes relevance much more confusing, removing it’s clarity advantage)
Disadvantages:
It requires building a whole new system. This is nontrivial. (It’s probably only around a week of work, but there are lots of other things we could build in a week, and if most people find Karma Relevance pretty intuitive it may not be worth it)
Users have to learn a new system. I’m guessing that system wouldn’t be that hard to learn, but I think “median vote” is a somewhat more confusing concept than “sort by a number” (especially when you factor in vote weight)