Forum Karma: view stats and find highly-rated comments for any LW user

[Max H]

Link post

I’ve been posting and commenting pretty frequently over the last few months, and I was curious about some stats. What started as a few GraphQL queries and some Python scripting turned into an interactive web app:

Enter a username, and it will give you some stats and a graph, broken down by post and comment karma. You can use the slider to adjust the date range, and the stats are automatically recalculated for the selected time period.

Another feature is the list of “Gems”—comments with at least a few votes that have the highest net karma score, i.e. comments that received strong upvotes from high-karma users, and few downvotes. I found that this often gives a better sense of a user’s best comments than just looking at the highest-scoring ones.

It can be useful for finding your own historical top comments, or to get a sense for the best contributions of other users, which are often buried in their comment history and hard to access directly.^[1]

Interesting example users:

• https://​​forumkarma.com/​​users/​​johnswentworth

• https://​​forumkarma.com/​​users/​​eliezer_yudkowsky

• https://​​forumkarma.com/​​users/​​gpt2

• Me!

It works for the EA forum too, though you have to specify a full profile URL to disambiguate:

• https://​​forumkarma.com/​​users/​​https://​​forum.effectivealtruism.org/​​users/​​lizka

• https://​​forumkarma.com/​​users/​​https://​​forum.effectivealtruism.org/​​users/​​holdenkarnofsky

Limitations and caveats:

• The app works by querying the LW GraphQL API. It makes a couple of GraphQL queries to fetch data for up to 5000 comments and 5000 posts all at once. I only fetch the minimal fields needed to calculate the stats, not the actual contents of the posts or comments. (For top comments links and preview, the comment data is fetched individually as needed, on hover). I think this shouldn’t place too much stress on the LW backend, but I can take it down if the developers or mods find it to be problematic.

• Data and stats for users with more than 5000 comments will be incomplete. I tried for a while to find a GraphQL query that supports fetching old comments for users with lots of comments, but couldn’t find one (offset doesn’t work; the API returns an error for too-big offsets.) If any LW devs (or anyone else) finds a GraphQL query that allows fetching old comments for users with >5000 comments, I can fix this. GreaterWrong somehow accomplishes this to implement sort by top or old, but they appear to be doing it on the backend, so I couldn’t reverse engineer it.

• The comment previewer is pretty basic, since I’m just dumping the raw HTML from LW into a tooltip with minimal formatting. I might improve this at some point.

• Stats for old users will be somewhat inaccurate—I assumed for the purposes of removing the user’s self-vote (which doesn’t count towards their karma) that everyone always strong upvotes their own posts and weak upvotes their own comments, and that strong /​ weak upvotes are worth the amounts described here. This will be slightly inaccurate for contributions from before strong voting was introduced, and more inaccurate for contributions from before self-voting was introduced. I was mainly interested in stats for more recent content, but if anyone has a handy schedule of when all the voting rules were implemented /​ changed, I can probably make it more accurate pretty easily.

• Karma is counted as if it were all received at the time the post or comment was first posted. In my experience, the vast majority of votes on something happen relatively soon after it is posted—people tend not to vote on old stuff too much. So this introduces only a small inaccuracy.

Aside: this was the first nontrivial greenfield project that I’ve developed since GPT-4 was released. I used some technologies I was less familiar with (GraphQL, Chart.js, Tailwind CSS), and had GPT-4 write as much of the code as possible. I’ve been using GitHub Copilot for day-to-day programming for a while, but this was another level. The code quality wasn’t always great, and I’m not sure how much time it really saved me in the end, but it definitely helped me get up to speed with unfamiliar libraries faster, and it was pretty wild to just ask GPT to re-arrange or format the elements on the page and have it produce something that looked pretty good on the first try.

Feel free to report bugs or suggest features in the comments here. (No promises that I’ll keep working on it any time soon though.)

1. ^

   GreaterWrong supports sorting user comments by top scoring or old, which can also be useful for this purpose, but there’s no option to filter by a date range.

[jefftk]

Neat! Looking over my “gems” brings up a few comments that I should revisit to consider turning into top level posts.

[Dagon]

Neat, agreed! I am a little confused by “gems”—is that basically selecting for things with the higher proportion of strong upvotes (and no/​few downvotes)? Are there other ways to have karma per vote be higher than 1?

[Max H]

The gems list is the head of the list when comments are sorted by karma per vote (baseScore /​ voteCount), filtered for comments that received at least 3 votes. The only way to have this be higher than 1 is to receive a strong upvote, or a weak upvote from someone with >1000 karma. It can also be much less than 1, if the comment also got downvotes. I explicitly decided not to add a “controversial” sort, i.e. comments that received a lot of votes with karma (or karma per vote) close to 0.

[Max H]

Disclaimer I forgot to put in the OP: The stats are intended to be for informational and curiosity-satisfaction purposes; I don’t want to encourage people to Goodhart on karma /​ upvotes more than they already do.

One of the things that the Gems vs. Top comments feature shows is that the highest-karma comments are not always the user’s best or highest-effort contributions. Many very-highly upvoted comments tend to be top-level comments on popular /​ general-interest posts, while higher-effort contributions are buried deep in a nested comment thread. In my experience, comments at even one level of nesting tend to receive far fewer votes than top-level comments, regardless of their quality.

[Vladimir_Nesov]

Effort is cost, not value.

[Elizabeth]

This is true, but I think Max is correct that karma breaks down at the edges as a measure of value. My highest karma comment was just “I like that you included this bit” on a popular post. Some of my others are great and detailed, others you could at least argue were insightful even if they were short and low-effort, but that comment was an applause light I happened to say before anyone else did. I’m glad someone said it and seems fine that it was me, but jesus christ that comment did not deserve 106 karma.

I think the applause light problem has gotten a lot better since agree/​disagree voting was implemented, but it karma still has some applause light function. It doesn’t even try to track counterfactuality (“would someone else have said this if I didn’t?”) so there’s no way it could be a perfect measure of value.

[Vladimir_Nesov]

My comment is a specific aside on using effort interchangeably with value, “higher-effort contributions” as an expression of approval, thereby implicitly valorizing effort. This is not good for a forum, a norm that asks for participation to be painful.

Effort is some sort of proxy for value, when there is a process that converts higher effort into higher value, but goodharting it is predictably a problem, so it’s prudent to avoid putting it in the role of a proxy. When it’s more hedonic to only participate in a lower-effort way, but still manage to select higher-value things to say, that can be vastly better than aiming for effort and ultimately leaving because effort is less comfortable.

[Raemon]

I think I’ve noticed a few places lately where you make what’s intended to be a local correction, without engaging with the broader point being made. (I think you’ve stated explicitly this is intentional although I can’t remember offhand)

I vaguely recall a few cases where people interpreted you as making a point about the overall validity of the broader OP, with a resulting confused back-and-forth.

In this case, I’m guessing you’d agree with “One of the things that the Gems vs. Top comments feature shows is that the highest-karma comments are not always the user’s best or highest-effort contributions”, if the “or highest effort” was removed? Assuming so, I think it might be worth erring a bit more on spelling out how your local point relates to the broader point being made? (and hopefully save everyone a round trip of comments).

i.e something like “I agree highest karma comments aren’t necessarily the best, but, I want to flag that effort is cost, not value”. (Or, if you do still disagree with the broader point, or don’t have an opinion on it, maybe state that instead. But, like, make the intended scope more clear. The brevity of the initial comment seems outweighed by the subsequent back-and-forth IMO)

(edited a few times for hopefully more clarity)

[Vladimir_Nesov]

I think it might be worth erring a bit more on spelling out how your local point relates to the broader point being made?

It often doesn’t. I think allowing some misunderstanding is efficient, and nuance that makes things clearer can still be loss of efficiency.

Nuance that frames decoupling in disclaimers feeds the norm of contextualizing, making decoupling higher-effort. Avoiding the disclaimers directly makes the asides less cumbersome, and works against the expectation that asides are something they are not.

Trivial inconveniences are key to what actually happens. Norms that make some things more inconvenient are key to what actually happens all the time. Making discussion of local validity more effortful hurts sanity of the conversation in the long term.

[Max H]

I actually agree with @Vladimir_Nesov’s reply to my initial comment (and wasn’t planning to reply at all until I read this, haha). I have now used reactions to indicate precisely how I feel about it.

Re-reading my own comment, I definitely didn’t intend to “implicitly valorize effort”, so if people were reading it that way, the correction /​ clarification was helpful.

I was more trying to say: comments and threads with the highest ROI for the reader are often not the ones with the most karma, so readers may benefit from making some effort to seek out high-value content in less-obvious places (e.g. nested comment threads, replies to less popular posts). But I’m not trying to valorize that effort either, just pointing it out as something that exists and which can have a distorting effect on karma sometimes.

[Zach Stein-Perlman]

Nice.

I have an EA Forum linkpost that was backdated to 1972, which makes my karma-over-time graph pretty funny!

[jefftk]

Is it normal for link posts to have their dates adjusted to the content, or was this a joke?

[Zach Stein-Perlman]

It’s not normal but during the forum’s “Decade Review” old linkposts were encouraged and were all backdated (because posts had to be from before 2021 to qualify).

[MondSemmel]

I enjoyed browsing a few of the top comments in the linked profiles, so thanks for making this <3.

[philh]

Neat! I found it interesting that ⁸⁄₁₀ of my top comments by karma are from pre-LW 2.0. At least some of that is because the rationality quotes threads were good for karma farming, but apparently there were also just way more votes being cast.

This will be slightly inaccurate for contributions from before strong voting was introduced, and more inaccurate for contributions from before self-voting was introduced.

Not important, but I guess there’ll also be some inaccuracies to do with vote strength changing. (Out of interest, do you calculate vote strength based on current karma, or their fuzzily-back-computed karma at the time they made the comment/​post?)

[Max H]

I do take the way the user’s vote strength can change over time into account, yes. The algorithm looks like:

  contributions.forEach((c) => {
    if (c.type === "comment") {
      const karma = c.baseScore - userSmallVotePower(cumulativeKarma);
      cumulativeKarma += karma;
    } else {
      const karma = c.baseScore - userBigVotePower(cumulativeKarma);
      cumulativeKarma += karma;
    }
  });

Where userSmallVotePower and userBigVotePower are taken directly from the LW codebase.

I could make these functions take a date param though, and then return 1 if the postedAt date is before strong voting was introduced, and 0 if the date is before self-voting was introduced. I think these rules have changed a bit over time though, and I couldn’t find a comprehensive timeline of when all the changes were introduced.

There are still two remaining ways this can be inaccurate:

• The calculation assumes that all karma is received at the time the comment or post is first submitted, which might result in the calculating users’ self-votes as being worth more than they actually were at the time.

• Users can go back and adjust their self-votes after they get enough karma to change how much their votes are worth. (You can unvote and re-vote on your own old stuff. This doesn’t change your own karma, but it can change the score on an individual post or comment by a small amount.)