The abstract is really unimpressive. The effect was not statistically significant. Was it even positive?* It is only by throwing out games lost on time that they find improved performance. Systematically throwing out games lost and finding a lot of games won is a standard error. Did it improve the moves they made, or did it cause them just to ruminate on lost games? They could have tested this, asking computers how good the moves were, but I don’t think that they did. Even if the moves were better, was it because of the drugs, or because of the additional time spent, a decision that might have been possible without drugs? The claim that drugs made the players consume more time was interesting, though.
* Update: yes, the effect was positive before throwing out the games lost on time. The effect size was 7%, half of that reported in the abstract after throwing out losses on time. More concretely, the improvement was turning 2.5% of games from losses into wins and 2% of games from draws into wins.
Addendum: Stimulants mess with your sense of time. Time management is an important part of playing chess, but surely players don’t do it by looking at the clock, but by heuristics about how much to think, heuristics that could be messed up by their sense of time. So I think that the time management would be improved by practice with the drugs. Most chess players probably have practice at varying levels of caffeine, so the study is a more fair comparison of that drug than the others.
Added later: Indeed, caffeine lead to fewer losses on time than modafinil and methylphenidate, although more than placebo, even though raw win/loss/draw numbers were the same for the three drugs. (Oddly, although the score excluding games lost on time puts modafinil and methylphenidate together, a more general metric of performance controlling for time groups together methylphenidate with caffeine.)
I’m worried that I found the study far more convincing than I should have. If I recall, it was something like “this would be awesome if it replicates. Regression toward the mean suggests the effect size will shrink, but still.” This thought didn’t stop me from still updating substantially, though.
I remember being vaguely annoyed at them just throwing out the timeout losses, but didn’t discard the whole thing after reading that. Perhaps I should have.
I know about confirmation bias and p-hacking and half a dozen other such things, but none of that stopped me from overupdating on evidence I wanted to believe. So, thanks for your comment.
An interesting concept—un-updating. Should happen when you updated on evidence that turned out to be wrong/mistaken, so you need to update back and I suspect that some biases will be involved here :-/
Modafinil improved performances by 15 percent, methylphenidate by 13 percent, and caffeine by around 9 percent.”
“Dr. Lieb said that the researchers went into the study with the expectation that the stimulants would not show much benefit. “We primarily thought that it is not possible to enhance high cognitive tasks and were astonished to find such results,” he said.”
Actually, as a tournament player I feel I can help explain the slowness:
The article suggests that this isn’t due to increased computational speed or focus, but I think that’s wrong. Playing slowly doesn’t imply thinking slowly. In a chess game, you have a certain amount of time overall, and often when the position is very complicated players will spend half an hour delving into variations and sub-variations. If it’s hard to concentrate, they may just rely on low-calc alternatives, and play faster.
There might be great value in setting up a way for an individual to evaluate his personal performance improvement from different drugs so you know how to optimize your performance on days when it really matters such as when you take an important test.
The abstract is really unimpressive. The effect was not statistically significant. Was it even positive?* It is only by throwing out games lost on time that they find improved performance. Systematically throwing out games lost and finding a lot of games won is a standard error. Did it improve the moves they made, or did it cause them just to ruminate on lost games? They could have tested this, asking computers how good the moves were, but I don’t think that they did. Even if the moves were better, was it because of the drugs, or because of the additional time spent, a decision that might have been possible without drugs? The claim that drugs made the players consume more time was interesting, though.
* Update: yes, the effect was positive before throwing out the games lost on time. The effect size was 7%, half of that reported in the abstract after throwing out losses on time. More concretely, the improvement was turning 2.5% of games from losses into wins and 2% of games from draws into wins.
Addendum: Stimulants mess with your sense of time. Time management is an important part of playing chess, but surely players don’t do it by looking at the clock, but by heuristics about how much to think, heuristics that could be messed up by their sense of time. So I think that the time management would be improved by practice with the drugs. Most chess players probably have practice at varying levels of caffeine, so the study is a more fair comparison of that drug than the others.
Added later: Indeed, caffeine lead to fewer losses on time than modafinil and methylphenidate, although more than placebo, even though raw win/loss/draw numbers were the same for the three drugs. (Oddly, although the score excluding games lost on time puts modafinil and methylphenidate together, a more general metric of performance controlling for time groups together methylphenidate with caffeine.)
I’m worried that I found the study far more convincing than I should have. If I recall, it was something like “this would be awesome if it replicates. Regression toward the mean suggests the effect size will shrink, but still.” This thought didn’t stop me from still updating substantially, though.
I remember being vaguely annoyed at them just throwing out the timeout losses, but didn’t discard the whole thing after reading that. Perhaps I should have.
I know about confirmation bias and p-hacking and half a dozen other such things, but none of that stopped me from overupdating on evidence I wanted to believe. So, thanks for your comment.
An interesting concept—un-updating. Should happen when you updated on evidence that turned out to be wrong/mistaken, so you need to update back and I suspect that some biases will be involved here :-/
Modafinil improved performances by 15 percent, methylphenidate by 13 percent, and caffeine by around 9 percent.”
“Dr. Lieb said that the researchers went into the study with the expectation that the stimulants would not show much benefit. “We primarily thought that it is not possible to enhance high cognitive tasks and were astonished to find such results,” he said.”
I’m not surprised. But I also don’t see much utility from this study; most people already believe that coffee helps them focus.
Don’t you think there’s some value of doing a more controlled study of it?
Something about “makes play better but slower” feels especially persuasive to me.
Actually, as a tournament player I feel I can help explain the slowness:
The article suggests that this isn’t due to increased computational speed or focus, but I think that’s wrong. Playing slowly doesn’t imply thinking slowly. In a chess game, you have a certain amount of time overall, and often when the position is very complicated players will spend half an hour delving into variations and sub-variations. If it’s hard to concentrate, they may just rely on low-calc alternatives, and play faster.
Agreed.
There might be great value in setting up a way for an individual to evaluate his personal performance improvement from different drugs so you know how to optimize your performance on days when it really matters such as when you take an important test.
...paging gwern
:-)
What the hell is an improvement in Chess by 15%? How many Elo points are we talking about?
400*log10(1.15), I’d guess?