The delay between a likely critical comment and a likely critical post is probably quite substantial as well, which I have to imagine also makes a large impact on visibility.
What? Of course you would want to control for effort and delay. If you leave a comment on a post a week after it was published because the comment took a while to write, then it will of course also get less visibility.
Of course it will… which means that if that’s the only option, then the criticism will get much less attention in the context of the original post.
Sure, someone could quickly write a comment-tier post, but there is a strong cultural norm (reinforced by the existence of shortform, which strongly suggests that posts should take more effort) against doing such a thing — and going against these sorts of norms impacts how well it is received too. Furthermore, you also need to apply more skill to ensuring a commensurate degree of visibility (e.g. by picking a compelling title). I would expect the typical reaction to be anemic, relative to the same post as a comment.
If your goal is to have your criticism read by the people who saw the original thing, I don’t see how this isn’t a significant effect that you would want to consider.
I don’t understand. Like, writing a high-quality thing takes time. Yes, comments are encouraged to be lower-effort, but on the margin I want people to write more critique posts, not more comments, and this clearly is still very important noise in the thing we are trying to measure. Of course you need to do at least some controlling for effort and delay, though I agree how much is tricky (like, there are extra costs, but clearly posting immediately also just gives you a random boost that is anti-correlated with quality).
I would expect the typical reaction to be anemic, relative to the same post as a comment.
I disagree? Making a top-level post a week later would likely create vastly more visibility than leaving a high-effort comment with a shorter delay, because posts get promoted on their own, but comments only get promoted in the context of the post. Posts get a lot more visibility than comments on this site (for good reason, they are usually more valuable to read, and stand more on their own).
And critique posts are extremely popular on the site. A good chunk of the highest karma posts on the site are critique posts. This part honestly seems clearly wrong to me. I would take bets at very high odds that if Wei Dai left a slightly polished version of his comment on Richard’s post as a top-level post, then it would get much more visibility and karma than it had as a comment.
I don’t see how this isn’t a significant effect that you would want to consider.
I am not saying to not consider it! You are the person saying you shouldn’t control for this at all, which seems absurd to me. Obviously we both agree that “recency of publication” is a confounder you want to control for somehow, since it gives you a substantial boost orthogonal to what we are trying to measure here. We can start arguing about the right methodology for controlling for that, but it’s obviously a big effect.
I’m saying it’s not orthogonal to what I’m interested in measuring here, which is the impact of this policy on the visibility of critique. I’ll grant that’s not exactly what Eric seems to be considering in the comment above; sorry for the miscommunication.
Those critique posts seem to mainly be by popular authors, and quite long. I believe they additionally are mainly responses to posts that were themselves unusually popular, which is important because it means the author can assume a sort of common knowledge that a critique of a mid-level post would otherwise have to establish. I agree that Wei Dai could have achieved that, but that you are significantly underestimating the costs to him of doing so. And I’m also concerned about the case where the author does not have as strong of a reputation, of course.
I’ll grant that’s not exactly what Eric seems to be considering in the comment above; sorry for the miscommunication.
Cool, yeah, I was just saying you will have to adjust somehow for timeliness and effort from the baseline that Eric was proposing.
that you are significantly underestimating the costs to him of doing so
I haven’t made any statements about the cost. I think the cost is non-trivial, but it feels like the kind of thing that seems well in-line with other nudges and affordance-shaping we do on the site. In as much as there is a really crucial misunderstanding that people are forming, where they are using Wei’s absence of a response as evidence that a thing is robust or valid or good, then making a post allows you to address that.
Yes, the cost is somewhat higher than with a high-effort comment, but it just seems like an adequate solution to me, the same way that comments seem like an adequate solution to me, and I don’t need a publicly visible “I disendorse this” button at the bottom of each post that provides a minimum-effort way of expressing your opinion on something.
The delay between a likely critical comment and a likely critical post is probably quite substantial as well, which I have to imagine also makes a large impact on visibility.
That effect is definitely large! If we include that one it would easily be 10x, but I was assuming some metric where we control for this.
Why the hell would you control for that? Isn’t the whole issue at stake the allocation of actual attention?
What? Of course you would want to control for effort and delay. If you leave a comment on a post a week after it was published because the comment took a while to write, then it will of course also get less visibility.
Of course it will… which means that if that’s the only option, then the criticism will get much less attention in the context of the original post.
Sure, someone could quickly write a comment-tier post, but there is a strong cultural norm (reinforced by the existence of shortform, which strongly suggests that posts should take more effort) against doing such a thing — and going against these sorts of norms impacts how well it is received too. Furthermore, you also need to apply more skill to ensuring a commensurate degree of visibility (e.g. by picking a compelling title). I would expect the typical reaction to be anemic, relative to the same post as a comment.
If your goal is to have your criticism read by the people who saw the original thing, I don’t see how this isn’t a significant effect that you would want to consider.
I don’t understand. Like, writing a high-quality thing takes time. Yes, comments are encouraged to be lower-effort, but on the margin I want people to write more critique posts, not more comments, and this clearly is still very important noise in the thing we are trying to measure. Of course you need to do at least some controlling for effort and delay, though I agree how much is tricky (like, there are extra costs, but clearly posting immediately also just gives you a random boost that is anti-correlated with quality).
I disagree? Making a top-level post a week later would likely create vastly more visibility than leaving a high-effort comment with a shorter delay, because posts get promoted on their own, but comments only get promoted in the context of the post. Posts get a lot more visibility than comments on this site (for good reason, they are usually more valuable to read, and stand more on their own).
And critique posts are extremely popular on the site. A good chunk of the highest karma posts on the site are critique posts. This part honestly seems clearly wrong to me. I would take bets at very high odds that if Wei Dai left a slightly polished version of his comment on Richard’s post as a top-level post, then it would get much more visibility and karma than it had as a comment.
I am not saying to not consider it! You are the person saying you shouldn’t control for this at all, which seems absurd to me. Obviously we both agree that “recency of publication” is a confounder you want to control for somehow, since it gives you a substantial boost orthogonal to what we are trying to measure here. We can start arguing about the right methodology for controlling for that, but it’s obviously a big effect.
I’m saying it’s not orthogonal to what I’m interested in measuring here, which is the impact of this policy on the visibility of critique. I’ll grant that’s not exactly what Eric seems to be considering in the comment above; sorry for the miscommunication.
Those critique posts seem to mainly be by popular authors, and quite long. I believe they additionally are mainly responses to posts that were themselves unusually popular, which is important because it means the author can assume a sort of common knowledge that a critique of a mid-level post would otherwise have to establish. I agree that Wei Dai could have achieved that, but that you are significantly underestimating the costs to him of doing so. And I’m also concerned about the case where the author does not have as strong of a reputation, of course.
Cool, yeah, I was just saying you will have to adjust somehow for timeliness and effort from the baseline that Eric was proposing.
I haven’t made any statements about the cost. I think the cost is non-trivial, but it feels like the kind of thing that seems well in-line with other nudges and affordance-shaping we do on the site. In as much as there is a really crucial misunderstanding that people are forming, where they are using Wei’s absence of a response as evidence that a thing is robust or valid or good, then making a post allows you to address that.
Yes, the cost is somewhat higher than with a high-effort comment, but it just seems like an adequate solution to me, the same way that comments seem like an adequate solution to me, and I don’t need a publicly visible “I disendorse this” button at the bottom of each post that provides a minimum-effort way of expressing your opinion on something.