Doing Science! Open Thread Experiment Results
Early in the month I announced that I was doing an experiment: I was going to start two Open Threads in January (one on the 1st, and the other on the 15th) and compare the number of comments on these threads to those of other months. My hypothesis was that having two Open Threads would raise the overall number of comments.
The reason for this experiment was recent discussions regarding how useful threads such as these were quickly buried. Well, the experiment is over now, and here are the results:
I did a search for Open Threads, and entered all the monthly ones I could find into an Excel spreadsheet. I made them into a graph, and I discovered an anomaly. There was an 8-month timespan from February 2010-September 2010, in which the comment counts were extremely high (up to 2112). Many of these threads had 2, 3, or 4 parts, because they were getting filled up.
I wasn’t around LW back then, and I don’t feel like reading through them all, so I don’t know why this time period was so active. My current hypothesis (with P=.75) is that anomalous time period was before the Discussion section was created. I’m sure I could look it up to see if I’m right, but I bet one of the long-term LWers already knows if this is true or not, so I’ll crowd-source the info. (Comment below if you know that I am correct or incorrect in my hypothesis.)
Now for the data:
The January 1-15, 2012 thread had: 122 comments
The January 16-31, 2012 thread had: 236 comments
For a grand total of: 358 comments in Jan 2012
The average Open Thread had: 448.6 comments
The median Open Thread had: 204 comments
The average OT of the past 14 mo’s: 126.5 comments
So overall, the January thread had LESS than the average monthly thread, but more than the median.
IF however we look at the past 14 months (which was the end of the anomaly), then the January 2012 Open Thread had almost THREE TIMES the average.
My original hypothesis had probabilities assigned to various increases in comment rate, but I was way off because I didn’t at all think it would shrink (if we include the anomaly) or that it would be 300% bigger (if we don’t)
Here’s a handy-dandy chart, because everything is better with pictures in!
This is correct. The discussion section opened in late September, 2010.
I don’t interpret it as an anomaly. It’s easier to read it as noisy growth from March 09 until Sept 10. Agree that the drop-off is the introduction of the LW Discussion section.
I agree with you. It looks like a dirty exponenetial growth curve.
But when I was typing the post it was really bulky to keep saying “time period with extremely high comment count”, or “high comment volume time period”, so I decided to call it an anomaly, just so I could go back and replace all those 7-word phrases with a single word. It made it MUCH easier to read, at the trade-off of labeling a not-very-anomalous thing as an anomaly.
If you have a better work-around, I’d love to hear it (but I’m not going back to “time period with extremely high comment count”, because it makes it too hard to read)
Is the anomaly caused by LW starting? Did it suddenly drop when we could post Discussion posts?
Yeah, that data was pretty inconclusive. Personally, I don’t seem the point of open threads: Discussion allows you to talk about most things anyway, and it’s way easier to check when someone makes a new Discussion post than a new top-level comment on an open thread.
Kudos for actually checking, though, and for publishing your inconclusive result.
When I read the thread title, I was excited because I thought it was an open thread purely for reporting your own experimental results while doing science on your own, sort of like LW doing the Seth Roberts. I was mildly disappointed when I read the post.
(Good work, btw)
Thanks!
Why not make the Experimental Results Open Thread yourself? If you want it, then I’m sure other people do too!
Well, I’d probably need to seed the thread with my own experimental results, but I don’t have anything interesting to report. My interest was more voyeuristic. If I do happen to self experiment in some way, I’ll keep the thread in mind.
Does anyone has time to check the count of top-level comments in Open Threads?
It could have more or less noise than aggregate post count
Are you going to keep posting as OpenThreadGuy?
I am not OpenThreadGuy. I assume OTG is a sock puppet for someone who wants to be responsible for Open Thread posting. I’m all for that! If it’s like Feb 3rd and I realize there’s no Open Thread yet though, I’ll post it!
Oh, I guess I just assumed it was one of your other accounts since you proposed the second Monthly thread and OpenThreadGuy made one. And yeah, I like the idea too.