Tag Relevance Systems (Feedback Requested)

We recently launched our Tagging Beta (See here for the full anouncement). We’ve spent the past few months thinking a lot about our longterm vision for tags on the site, but still have some uncertainty about exactly what shape tagging should take.

One major question that the team hadn’t yet come to agreement on is how exactly to vote on Tag Relevance.

When you go to a tag page (such as the Rationality Tag), somehow we have to decide which posts to list first. An obvious answer is “just sort by karma”, but this isn’t quite right: High karma posts aren’t necessarily the most relevant posts.

For example, if “Meditations on Moloch” had originally been a LessWrong post, one might have tagged it “economics”. (It does touch on economic concepts). And it might have gotten lots of upvotes because it was an evocative, powerfully written piece. But this wouldn’t make it appropriate to be listed first on the Economics Tag Page.

So, we want some other system to determine how relevant a post is. And we want that system to be scalable, where lots of users could potentially go tagging posts and determining their relevance. But this leaves us needing a dispute mechanism – what happens if two people disagree?

We shipped the Tagging Beta with one such system, but are considering an alternative. I’d love your feedback.

System 1: Karma-style Relevance

We shipped with a system that simply re-used our existing Karma System. For each Tag-Post relationship, you can vote on how relevant the post is to the tag, using your normal array of upvotes/​downvotes/​weakvotes/​strongvotes.

Here, relevance is a simple number. Higher relevance posts are sorted higher on the tag page. Posts are considered “tagged” if they have a positive amount of relevance. If you think a post was tagged incorrectly, you can downvote it to zero.

This system has some advantages and disadvantages. I’m going to describe my thoughts on those in the comments, so that you have a chance to form your own opinion if you want.

System 2: Multiple Choice Voting

There were some aspects of the Karma-Relevance system that I personally found dissatisfying. So I created an alternate system and pitched it to the team, and we collectively decided to present them both and give you guys the chance to give some feedback.

This system uses multiple-choice voting, where you sort the tag into one of five buckets. Within those buckets, posts are sorted by karma.

Here’s a video showing how that would work. (I recommended watching on fullscreen mode to more easily read the text)

Instead of adding up votes to get a total relevance, this uses the median-vote to determine which bucket to put the post in. Then, on the tag page, all Top Relevance posts will be listed first (in karma order), followed by all High Relevance posts, then Medium, then Low.

Votes would still get vote-weight based on your karma, so established users can more easily counter an overzealous newbie who went on a tagging spree.

What do you think?

In the comments, I’ll go into more details about how each system works, and I think their relative merits are. But an important question is “how intuitive is each system?”.

So for now, I’m just going to end the post asking for you to try out the current karma-relevance voting system (you can try upvoting this post for the Site Meta tag to get an idea of how that works), and to watch the video so you have an idea of what the new system would look like.

If you’re up for it, it’d be quite helpful if you could write up your “first impressions” before reading the more detailed comments.