Tag Relevance Systems (Feedback Requested)
We recently launched our Tagging Beta (See here for the full anouncement). We’ve spent the past few months thinking a lot about our longterm vision for tags on the site, but still have some uncertainty about exactly what shape tagging should take.
One major question that the team hadn’t yet come to agreement on is how exactly to vote on Tag Relevance.
When you go to a tag page (such as the Rationality Tag), somehow we have to decide which posts to list first. An obvious answer is “just sort by karma”, but this isn’t quite right: High karma posts aren’t necessarily the most relevant posts.
For example, if “Meditations on Moloch” had originally been a LessWrong post, one might have tagged it “economics”. (It does touch on economic concepts). And it might have gotten lots of upvotes because it was an evocative, powerfully written piece. But this wouldn’t make it appropriate to be listed first on the Economics Tag Page.
So, we want some other system to determine how relevant a post is. And we want that system to be scalable, where lots of users could potentially go tagging posts and determining their relevance. But this leaves us needing a dispute mechanism – what happens if two people disagree?
We shipped the Tagging Beta with one such system, but are considering an alternative. I’d love your feedback.
System 1: Karma-style Relevance
We shipped with a system that simply re-used our existing Karma System. For each Tag-Post relationship, you can vote on how relevant the post is to the tag, using your normal array of upvotes/downvotes/weakvotes/strongvotes.
Here, relevance is a simple number. Higher relevance posts are sorted higher on the tag page. Posts are considered “tagged” if they have a positive amount of relevance. If you think a post was tagged incorrectly, you can downvote it to zero.
This system has some advantages and disadvantages. I’m going to describe my thoughts on those in the comments, so that you have a chance to form your own opinion if you want.
System 2: Multiple Choice Voting
There were some aspects of the Karma-Relevance system that I personally found dissatisfying. So I created an alternate system and pitched it to the team, and we collectively decided to present them both and give you guys the chance to give some feedback.
This system uses multiple-choice voting, where you sort the tag into one of five buckets. Within those buckets, posts are sorted by karma.
Here’s a video showing how that would work. (I recommended watching on fullscreen mode to more easily read the text)
Instead of adding up votes to get a total relevance, this uses the median-vote to determine which bucket to put the post in. Then, on the tag page, all Top Relevance posts will be listed first (in karma order), followed by all High Relevance posts, then Medium, then Low.
Votes would still get vote-weight based on your karma, so established users can more easily counter an overzealous newbie who went on a tagging spree.
What do you think?
In the comments, I’ll go into more details about how each system works, and I think their relative merits are. But an important question is “how intuitive is each system?”.
So for now, I’m just going to end the post asking for you to try out the current karma-relevance voting system (you can try upvoting this post for the Site Meta tag to get an idea of how that works), and to watch the video so you have an idea of what the new system would look like.
If you’re up for it, it’d be quite helpful if you could write up your “first impressions” before reading the more detailed comments.
Here’s what I consider the relative merits of each Tag Relevance system. (Note: it’d be helpful if you read the OP and form your own opinion before reading this)
...
...
Karma-Style
Advantages:
It re-uses an existing system, which makes it easier to program, and easier to learn.
It’s very easy to understand how the sorting works – it very transparently sorts in descending relevance order.
Disadvantages:
It’s hard to move a recently-tagged post to the bottom of the list without removing it entirely, or upvoting every other post instead.
I personally find that it messes with my intuitions about how karma works. I find myself wanting to upvote posts I like in the usual way… but I already did that. While re-using an existing system is mechanically simpler, I personally find it requires more conceptual orientation.
My votes are often not about which posts are objectively relevant, but about which posts I think are more relevant than the current consensus suggests. This a) contributes to voting feeling unintuitive to me, and b) since my votes are very “current-context dependent”, I worry that later I’ll have to go and re-vote on things to make sure my vote is still doing what I want.
It’s fairly overwhelming for the tag page to list both “relevance” and “karma”, but if you only show one of those, it’s confusing in a different way (if you leave off karma, people may mistakenly think relevance is karma. If you leave off relevance, people will be confused about how the page is sorted)
A single user can’t set up an entire tag page in a reasonable fashion – if you’re adding a new post, you basically only have two choices, “smallVote” and “bigVote”, so if you want to add anything that’s only tangentially relevant, you have to have it mixed in with all the other smallVotes until another user comes along to flesh things out.
When only a couple people have voted on a Tag Page’s posts, I think the results are worse than just sorting by karma.
I expect many of these problems to mostly effect less-visited tags. More visited tags will probably end up with lots of users contributing, resulting in a “wisdom of crowds”. But I expect at least half of Tag Pages to be sparsely attended, and for the relevance votes to be much noisier than just sorting by karma.
Multiple Choice Voting Style
Advantages:
A single user can vote honestly based on their true beliefs about how relevant a tag is, adding a bunch of tags at once and accurately sorting them.
I personally find it more intuitive (I don’t have any conflicting intuitions ported over from karma)
The Tag Page becomes simpler – you don’t need to show relevance at all, you just use it to establish major categories.
It lets karma remain relevant, without being very confusing. (A few people have suggested “sort by relevance + karma”, but the problem is that makes relevance much more confusing, removing it’s clarity advantage)
Disadvantages:
It requires building a whole new system. This is nontrivial. (It’s probably only around a week of work, but there are lots of other things we could build in a week, and if most people find Karma Relevance pretty intuitive it may not be worth it)
Users have to learn a new system. I’m guessing that system wouldn’t be that hard to learn, but I think “median vote” is a somewhat more confusing concept than “sort by a number” (especially when you factor in vote weight)
I found the current karma-style relevance to be surprisingly intuitive. My impression of tags is that the main problem is figuring out what is allowed to be a tag or not—I’m not sure how this was done, but the tags seem to be good.
I think “top” posts can be some combination of high karma, high tag relevance, and curated.
There is the additional problem of “Meditations on Moloch” was read by a lot of people, so it will get high karma and high tag relevance, and maybe be curated because it is high quality. This suggests that the multiple choice system is better able to bucket posts.
I think I want any tagging system to be able to separate between posts that use the concept and posts that try and explain the concept. There will not be that many of the later, but they should be prioritized on the tag’s page (there are possibly only 1 or 2, in which case a form of curation but for specific tags might solve the issue).
I like the current karma style relevance system and think it’s worthy to be given a good trial period. If after a year we find that not enough people vote on tags, we can introduce other features, but I think the tag functionality as it is functional enough to warrant a good trial period.
Meta on format: Why didn’t you use the question feature for this post?
Meta on content: yay, I really like those kind of improvement!
Object: epistemic status: thinking outloud right after having read the post, before reading the comments (as requested), and not having thought about this much before; not confident
it seems like system 1 asks “which categories belong to the post the most?” whereas system 2 asks “which posts belong to the category the most?”, sort of. I think most of the time, categorization is useful for system 2′s purpose (ie. searching for articles in a specific category), and when you want to know the categories of a specific post before reading it, system 2 should still be able to play system 1′s role.
also, system 2 shows the average instead of the sum, which might better represent the categorization. on the other hand, knowing the number of votes that the categorizations received is useful to know how large the sample was.
both systems seems to allow for 5 options, so are equally fine-grained (although maybe system 2 for more as it seems you can select two options from the video?)
voting system is more intuitive (flows better?) because we already use it
I wonder if the post writer should have more votes or something
Voting seems to fit if you’re okay with things not fitting together. If you want things to fit together, a set of “Economic” sequences/posts with an intentional order might work better.