Analyzing FF.net reviews of ‘Harry Potter and the Methods of Rationality’
The unprecedented gap in Methods of Rationality updates prompts musing about whether readership is increasing enough & what statistics one would use; I write code to download FF.net reviews, clean it, parse it, load into R, summarize the data & depict it graphically, run linear regression on a subset & all reviews, note the poor fit, develop a quadratic fit instead, and use it to predict future review quantities.
Then, I run a similar analysis on a competing fanfiction to find out when they will have equal total review-counts. A try at logarithmic fits fails; fitting a linear model to the previous 100 days of _MoR_ and the competitor works much better, and they predict a convergence in <5 years.
Master version: http://www.gwern.net/hpmor#analysis
Some alleged chapter reviews may be for the work as a whole rather than the particular chapter. Such reviews are likely to accumulate on a particular chapter when that chapter is the last one and there hasn’t been an update for a while. That could explain why the faster pace in the 40-70 region lead to less reviews per chapter. Even for chapter-specific reviews I’d guess people are more likely to write a review when they haven’t got a new chapter to immediately go on to.
There’s clearly a ‘dead-end’ effect, yes—you can see it clearly in the first graph, with the thick black bar each chapter receives, proportional to its time as the ‘latest’ chapter. EDIT: you can also see this effect vividly in the Unexpected Circumstances per-chapter graphs.
But I don’t see why this would distort an entire region compared to another region: every reader in the 40-70 region is going to dead-end in the 40-70 region, almost by definition, unless they start reading at the very tail end such that an update happens before they finish that region (and would leave a dead-end review, one chapter before).
I really like this analysis a lot. For whatever it adds, Google Trends shows it peaking in July 2011, but mostly holding steady. There might be a small decline in the last six months though.
HPMOR’s been on the rise as a search term according to Google Trends. Though I would assume those are mostly not new readers if they’re using shorthand.
I can’t remember how often Eliezer mentioned how important getting reviews was for him, but I wonder if that correlates with more reviews.
It can’t be that often. I just went through the Notes on HPMoR.com (which seem to be incomplete) and C-f for ‘review’, and nowhere since February 2012 does Eliezer ask for reviews (he remonstrates demanding reviews, mentions total review count, talks about reviewing his plans, etc.)
Emphasis mine. That is the only instance of anything like that in the notes I saved, though; the only other thing that turned up that I think might have skewed it was the bit where he talked a few times about how HPMOR was the X most reviewed HP fic/fic on ff.net/etc.
I don’t know if that worked but if that was the only instance of asking for reviews, I don’t think it did much of anything. Here’s the 7-days-filtered reviews for ch16-23:
So ch18 had the same number as the previous ch17, then with ch19 it dropped, ch20 was higher than 17⁄18, but then ch21-22 dropped even further. If there’s any effect from asking for reviews, I’m not seeing it in the filtered reviews.
Heh, but I guess that stopped working...
I started reading too late to catch most notes of this sort by EY (and I often skip Author’s Notes anyway), but from personal real-time observation of other fanfics it seems to be a tremendous help for authors to beg for reviews, in any and all senses of “begging”. Asking for stuff is good, and holding updates hostage for the price of reviews is even better (assuming there actually are any readers). Giving public thanks to reviewers also works.
That was very interesting, thank you.
By the way, I see that you have fitted an quadratic polynomial to the “number of chapters as a function of time” dependency. Can you eloborate on the reasons you used quadratic polynomial as opposed to sqrt(x) or log(x), the predicted date of the next update and the degree of trust such model deserves?
EDIT: It appears that I had a brainfart and mixed up plots. Nevermind then.
PS: Also, I suspect that taking total length of reviews into account can reveal something new about distinction between “hard fans” and “populists”.
Possibly, and maybe also try employing some ‘sentiment analysis’ to estimate review favorability, yeah.
But I probably won’t bother with this because I’ve answered the questions I was interested in, and doing those extra analyses would be much more work—not only do I not know how to use any of those tools, I wouldn’t even be able to reuse any of my existing code, eg. the download code for downloading entire reviews would be more complicated (I would have to do real HTML parsing instead of just grabbing single lines with user/chapter/date in a stereotypical format).
/r/HPMOR Reddit submission: http://www.reddit.com/r/HPMOR/comments/12l6g6/ffnet_reviews_of_hpmor_statistics/
As far as I understand (and I could be wrong), the primary purpose of HPMOR is to serve as a recruiting tool for Less Wrong and SIAI. Therefore, we don’t necessarily want to measure raw readership numbers, but what the advertisers call the “conversion rate”: the number of HPMOR readers who join Less Wrong. This rate will be different from raw readership, because once a reader has converted, we no longer care whether he reads HPMOR or not.
In practical terms, the trick with HPMOR is to spend just the right amount of time on updates, in order to release the updates frequently enough to attract new readers (who might otherwise be turned off by what they perceive as a “dead” fanfic) -- yet not so frequently that their marginal utility becomes negative.
That’s an interesting angle. I think we actually could measure conversions to some extent.
The survey asks after MoR, and also how long a user has been around, so one could filter for any survey set mentioning MoR and subtract their longevity from the date to figure out what was their ‘latest’ chapter before joining LW.
A second way which would probably work better is to leverage the usernames in the review data set and compare to a list of LessWrong users & account registration date, to do much the same thing: take every username which appears on both lists, and the LW account registration date, and credit them to the chapter appearing just before their LW account registration. (If their LW account precedes their first MoR review, it’d probably be best to eliminate them from consideration.)
Also, I’d check for their mentioning MoR in introduction posts. If I hadn’t taken the survey, that’s how you’d pick me up, for instance.
I was curious about the unique number of reviewers, and (from your CSV) it seems that 6838 different FF.net accounts placed reviews. That’s fewer than I expected given the 9.6k favs and 8.6k follows.
Of course, that guy named “Guest” was pretty prolific.
Remember the 1% rule: 1% of your viewers will interact or participate in a meaningful way (and I’ve seen videos on YouTube with more than 100x views to total votes). So actually, given how much easier it is to favorite or follow than to write a review (however slight), a 6.8 : 9.6 ratio is pretty impressive.
The odd thing there for me is that favorites outnumber follows. I would have expected follows to be a superset of favorites: I might find a story worth following, but not so good I will publicly put it on my favorites list & endorse it.
And let us give due credit to anon. People mock him for a lot of stuff, but he has good taste in fanfiction.
FFDN, especially historically, has a terrible interface, and I’m amazed more than tens of people managed to follow or favorite the story.
I tried to explain once the procedure for following a story on FFDN to an interested friend, and my friend responded with an incredulous “You’re making this up” and did not end up taking any action.
But unless the interface makes following much easier than favoriting, doesn’t explain the disparity.
Isn’t it like… checking a radio box at the bottom of the page?
I see:
Yes, that’s a very recent development. And those actually work most of the time, if the login system behaves.
I’ve updated the analysis with a new direct attempt at measuring reviewer ‘mortality’, which seems to be higher in the later chapters: http://www.gwern.net/hpmor#survival-analysis (Reddit)