I also buy the econ story here (and, per Ruby, I’m somewhat pleasantly surprised by the amount of reviewing activity given this).
General observation suggests that people won’t find writing reviews that intrinsically motivating (compare to just writing posts, which all the authors are doing ‘for free’ with scant chance of reward, also compare to academia—I don’t think many academics find peer review/refereeing one of the highlights of their job). With apologies for the classic classical econ joke, if reviewing was so valuable, how come people weren’t doing it already? [It also looks like ~25%? of reviews, especially the most extensive, are done by the author on their own work].
If we assume there’s little intrinsic motivation (I’m comfortably in the ‘you’d have to pay me’ camp), the money doesn’t offer that much incentive. Given Rudy’s numbers suppose each of the 82 reviews takes an average of 45 minutes or so (factoring in (re)reading time and similar). If the nomination money is ~roughly allocated by person time spent, the marginal expected return of me taking an hour to review is something like $40. Facially, this isn’t too bad an hourly rate, but the real value is significantly lower:
The ‘person-time lottery’ model should not be denominated by observed person-time so far, but one’s expectation how much will be spent in total once reviewing finishes, which will be higher (especially conditioned on posts like this).
It’s very unlikely the reward is going to allocated proportionately to time spent (/some crude proxy thereof like word count). Thus the EV would be discounted by whatever degree of risk aversion one has (I expect the modal ‘payout’ for a review to be $0).
Opaque allocation also incurs further EV-reducing uncertainty, but best guesses suggest there will be Pareto-principle/tournament dynamic game dynamics, so those with (e.g.) reasons to believe they’re less likely to impress the mod team’s evaluation of their ‘pruning’ have strong reasons to select themselves out.
I definitely don’t expect the money to be directly rewarding in a standard monetary sense. (In general I think prizes do a bad job of providing expected monetary value). My hope for the prize was more to be a strong signal of the magnitude of how much this mattered, and how much recognition reviews would get.
It’s entirely plausible that reviewing is sufficiently “not sufficiently motivating” that actually, the thing to do is pay people directly for it. It’s also possible that the prizes should be lopsided in favor of reviews. (This year the whole process was a bit of an experiment so we didn’t want to spend too much money on it, but it might be that just adding more funding to subsidize things is the answer)
But I had some reason to think “actually things are mostly fine, it’s just that the Review was a new thing and not well understood, and communicating more clearly about it might help.”
My current sense is:
There have been some critical reviews, so there is at least some motivation latent motivation to do so.
There are people on the site who seem to be generally interested in giving critical feedback, and I was kinda hoping that they’d be up for doing so as part of a broader project. (Some of them have but not as many as I’d hoped. To be fair, I think the job being asked for the 2018 Review is harder than what they normally do)
One source of motivation I’d expected to tap into (which I do think has happened a bit) is “geez, that might be going into the official Community Recognized Good Posts Book? Okay, before it wasn’t worth worrying about Someone Being Wrong On the Internet, but now the stakes are raised and it is worth it.”
I also buy the econ story here (and, per Ruby, I’m somewhat pleasantly surprised by the amount of reviewing activity given this).
General observation suggests that people won’t find writing reviews that intrinsically motivating (compare to just writing posts, which all the authors are doing ‘for free’ with scant chance of reward, also compare to academia—I don’t think many academics find peer review/refereeing one of the highlights of their job). With apologies for the classic classical econ joke, if reviewing was so valuable, how come people weren’t doing it already? [It also looks like ~25%? of reviews, especially the most extensive, are done by the author on their own work].
If we assume there’s little intrinsic motivation (I’m comfortably in the ‘you’d have to pay me’ camp), the money doesn’t offer that much incentive. Given Rudy’s numbers suppose each of the 82 reviews takes an average of 45 minutes or so (factoring in (re)reading time and similar). If the nomination money is ~roughly allocated by person time spent, the marginal expected return of me taking an hour to review is something like $40. Facially, this isn’t too bad an hourly rate, but the real value is significantly lower:
The ‘person-time lottery’ model should not be denominated by observed person-time so far, but one’s expectation how much will be spent in total once reviewing finishes, which will be higher (especially conditioned on posts like this).
It’s very unlikely the reward is going to allocated proportionately to time spent (/some crude proxy thereof like word count). Thus the EV would be discounted by whatever degree of risk aversion one has (I expect the modal ‘payout’ for a review to be $0).
Opaque allocation also incurs further EV-reducing uncertainty, but best guesses suggest there will be Pareto-principle/tournament dynamic game dynamics, so those with (e.g.) reasons to believe they’re less likely to impress the mod team’s evaluation of their ‘pruning’ have strong reasons to select themselves out.
Helpful thoughts, thanks!
I definitely don’t expect the money to be directly rewarding in a standard monetary sense. (In general I think prizes do a bad job of providing expected monetary value). My hope for the prize was more to be a strong signal of the magnitude of how much this mattered, and how much recognition reviews would get.
It’s entirely plausible that reviewing is sufficiently “not sufficiently motivating” that actually, the thing to do is pay people directly for it. It’s also possible that the prizes should be lopsided in favor of reviews. (This year the whole process was a bit of an experiment so we didn’t want to spend too much money on it, but it might be that just adding more funding to subsidize things is the answer)
But I had some reason to think “actually things are mostly fine, it’s just that the Review was a new thing and not well understood, and communicating more clearly about it might help.”
My current sense is:
There have been some critical reviews, so there is at least some motivation latent motivation to do so.
There are people on the site who seem to be generally interested in giving critical feedback, and I was kinda hoping that they’d be up for doing so as part of a broader project. (Some of them have but not as many as I’d hoped. To be fair, I think the job being asked for the 2018 Review is harder than what they normally do)
One source of motivation I’d expected to tap into (which I do think has happened a bit) is “geez, that might be going into the official Community Recognized Good Posts Book? Okay, before it wasn’t worth worrying about Someone Being Wrong On the Internet, but now the stakes are raised and it is worth it.”