The abstract is really unimpressive. The effect was not statistically significant. Was it even positive?* It is only by throwing out games lost on time that they find improved performance. Systematically throwing out games lost and finding a lot of games won is a standard error. Did it improve the moves they made, or did it cause them just to ruminate on lost games? They could have tested this, asking computers how good the moves were, but I don’t think that they did. Even if the moves were better, was it because of the drugs, or because of the additional time spent, a decision that might have been possible without drugs? The claim that drugs made the players consume more time was interesting, though.
* Update: yes, the effect was positive before throwing out the games lost on time. The effect size was 7%, half of that reported in the abstract after throwing out losses on time. More concretely, the improvement was turning 2.5% of games from losses into wins and 2% of games from draws into wins.
Addendum: Stimulants mess with your sense of time. Time management is an important part of playing chess, but surely players don’t do it by looking at the clock, but by heuristics about how much to think, heuristics that could be messed up by their sense of time. So I think that the time management would be improved by practice with the drugs. Most chess players probably have practice at varying levels of caffeine, so the study is a more fair comparison of that drug than the others.
Added later: Indeed, caffeine lead to fewer losses on time than modafinil and methylphenidate, although more than placebo, even though raw win/loss/draw numbers were the same for the three drugs. (Oddly, although the score excluding games lost on time puts modafinil and methylphenidate together, a more general metric of performance controlling for time groups together methylphenidate with caffeine.)
I’m worried that I found the study far more convincing than I should have. If I recall, it was something like “this would be awesome if it replicates. Regression toward the mean suggests the effect size will shrink, but still.” This thought didn’t stop me from still updating substantially, though.
I remember being vaguely annoyed at them just throwing out the timeout losses, but didn’t discard the whole thing after reading that. Perhaps I should have.
I know about confirmation bias and p-hacking and half a dozen other such things, but none of that stopped me from overupdating on evidence I wanted to believe. So, thanks for your comment.
An interesting concept—un-updating. Should happen when you updated on evidence that turned out to be wrong/mistaken, so you need to update back and I suspect that some biases will be involved here :-/
The abstract is really unimpressive. The effect was not statistically significant. Was it even positive?* It is only by throwing out games lost on time that they find improved performance. Systematically throwing out games lost and finding a lot of games won is a standard error. Did it improve the moves they made, or did it cause them just to ruminate on lost games? They could have tested this, asking computers how good the moves were, but I don’t think that they did. Even if the moves were better, was it because of the drugs, or because of the additional time spent, a decision that might have been possible without drugs? The claim that drugs made the players consume more time was interesting, though.
* Update: yes, the effect was positive before throwing out the games lost on time. The effect size was 7%, half of that reported in the abstract after throwing out losses on time. More concretely, the improvement was turning 2.5% of games from losses into wins and 2% of games from draws into wins.
Addendum: Stimulants mess with your sense of time. Time management is an important part of playing chess, but surely players don’t do it by looking at the clock, but by heuristics about how much to think, heuristics that could be messed up by their sense of time. So I think that the time management would be improved by practice with the drugs. Most chess players probably have practice at varying levels of caffeine, so the study is a more fair comparison of that drug than the others.
Added later: Indeed, caffeine lead to fewer losses on time than modafinil and methylphenidate, although more than placebo, even though raw win/loss/draw numbers were the same for the three drugs. (Oddly, although the score excluding games lost on time puts modafinil and methylphenidate together, a more general metric of performance controlling for time groups together methylphenidate with caffeine.)
I’m worried that I found the study far more convincing than I should have. If I recall, it was something like “this would be awesome if it replicates. Regression toward the mean suggests the effect size will shrink, but still.” This thought didn’t stop me from still updating substantially, though.
I remember being vaguely annoyed at them just throwing out the timeout losses, but didn’t discard the whole thing after reading that. Perhaps I should have.
I know about confirmation bias and p-hacking and half a dozen other such things, but none of that stopped me from overupdating on evidence I wanted to believe. So, thanks for your comment.
An interesting concept—un-updating. Should happen when you updated on evidence that turned out to be wrong/mistaken, so you need to update back and I suspect that some biases will be involved here :-/