I want to emphasize that I agree that we ought to prefer study data to N=1 motivated self-experiment, and I appreciate you bringing this meta-analysis to the table.
Now, let’s take a close look at Pilcher and Huffcutt.
Guzey’s self-administered study would be coded by P&H as “partial sleep deprivation” (< 5 hours sleep in a 24 hour period), the group showing the largest negative effect sizes in P&H. Note, however, that P&H are aggregating results from just 6 studies (citation numbers 39-41,44,48,49).
The experimental groups of these 6 studies included:
Anesthesia residents after 24 hours of in-house call.
Medical interns at work.
Periodic testing of graduate students at a psychological research center over 8 months, while the students lived an otherwise normal life.
Soldiers on an 8-day field artillery trial manually handling a large quantity of artillery shells (weighing 45 kg) and charges (13 kg).
Surgical resident’s who’ve been up working all night.
Internal medicine residents after a night on call.
In all but the trial on the graduate students, the subjects were dealing with both sleep deprivation and work-related fatigue. Guzey’s specifically interested in sleep deprivation per se, rather than the effects of work-related fatigue. Note that in the (n=8) study of the graduate students, “It was concluded that 6-8 months of gradual sleep restriction, down to 4.5-5.5 hrs per night, does not result in behavioral effects measurable by the instruments used… At the end of an additional 12-month follow-up period, total sleep time was still 1-2.5 hrs below baseline, but measures of well-being had returned to baseline levels.”
How could we get such clear and obvious results from the fatigue studies on doctors, and see nothing in the graduate students? I think work-related fatigue in the hospital is the culprit.
In addition, the inclusion of the soldier study is inexplicable. It wasn’t a study of sleep loss, but of the impact of sustained manual labor on muscular strength and endurance. The control and experimental group had no statistically significant differences in sleep deprivation, though both were sleep-deprived. I can only interpret the meta-study as using the effect of sustained manual labor under (uniform) conditions of sleep deprivation as a proxy for the effects of sleep deprivation itself.
Here’s an outline of the study itself. Group A sleeps 4 hours per night and loads artillery shells, group B sleeps 4 hours per night and doesn’t load any artillery shells. When Group A experiences a greater loss in muscular strength and endurance than Group B after 8 days of this regimen, the sleep researchers interpret this as the result of the sleep deprivation (which was identical to Group B’s), rather than the manual labor (the actual experimental intervention). So that makes no sense. Rather than inexplicable, perhaps I should have said inexcusable.
Considering the undoubtedly important confounding from the intense work environments of the subjects and that one of the six studies in this wing of the meta-analysis doesn’t belong there at all, I think this meta-analysis cannot shed as much light on Guzey’s research question as we might hope. In fact, the most useful part is the study of the graduate students, as it’s likely the least confounded by the effects of a high-stress and taxing work environment. It’s this one that finds effects that match Guzey’s claim.
All this is to highlight the importance of the specific research question and methods. Fatigue from working in a stressful or physically demanding environment is not the same thing as the effects of sleep deprivation. If we are interested in the latter, we should not permit the former to confound our study. This isn’t meant as an attack on you at all, but I think that the impulse to use the results of a meta-study of “sleep deprivation” that is actually a meta-study of work-related fatigue in a high-stress environment with 1⁄6 erroneous inclusions is indicative of the problems Guzey’s highlighting in his overall interpretation and critique of sleep studies.
As a meta point, the fact that a journal in the field is willing to publish such a meta-study that draws misleading conclusions is evidence for Guzey’s claim that the field is cargo-culting and not trustworthy.
I want to emphasize that I agree that we ought to prefer study data to N=1 motivated self-experiment, and I appreciate you bringing this meta-analysis to the table.
Now, let’s take a close look at Pilcher and Huffcutt.
Guzey’s self-administered study would be coded by P&H as “partial sleep deprivation” (< 5 hours sleep in a 24 hour period), the group showing the largest negative effect sizes in P&H. Note, however, that P&H are aggregating results from just 6 studies (citation numbers 39-41,44,48,49).
The experimental groups of these 6 studies included:
Anesthesia residents after 24 hours of in-house call.
Medical interns at work.
Periodic testing of graduate students at a psychological research center over 8 months, while the students lived an otherwise normal life.
Soldiers on an 8-day field artillery trial manually handling a large quantity of artillery shells (weighing 45 kg) and charges (13 kg).
Surgical resident’s who’ve been up working all night.
Internal medicine residents after a night on call.
In all but the trial on the graduate students, the subjects were dealing with both sleep deprivation and work-related fatigue. Guzey’s specifically interested in sleep deprivation per se, rather than the effects of work-related fatigue. Note that in the (n=8) study of the graduate students, “It was concluded that 6-8 months of gradual sleep restriction, down to 4.5-5.5 hrs per night, does not result in behavioral effects measurable by the instruments used… At the end of an additional 12-month follow-up period, total sleep time was still 1-2.5 hrs below baseline, but measures of well-being had returned to baseline levels.”
How could we get such clear and obvious results from the fatigue studies on doctors, and see nothing in the graduate students? I think work-related fatigue in the hospital is the culprit.
In addition, the inclusion of the soldier study is inexplicable. It wasn’t a study of sleep loss, but of the impact of sustained manual labor on muscular strength and endurance. The control and experimental group had no statistically significant differences in sleep deprivation, though both were sleep-deprived. I can only interpret the meta-study as using the effect of sustained manual labor under (uniform) conditions of sleep deprivation as a proxy for the effects of sleep deprivation itself.
Here’s an outline of the study itself. Group A sleeps 4 hours per night and loads artillery shells, group B sleeps 4 hours per night and doesn’t load any artillery shells. When Group A experiences a greater loss in muscular strength and endurance than Group B after 8 days of this regimen, the sleep researchers interpret this as the result of the sleep deprivation (which was identical to Group B’s), rather than the manual labor (the actual experimental intervention). So that makes no sense. Rather than inexplicable, perhaps I should have said inexcusable.
Considering the undoubtedly important confounding from the intense work environments of the subjects and that one of the six studies in this wing of the meta-analysis doesn’t belong there at all, I think this meta-analysis cannot shed as much light on Guzey’s research question as we might hope. In fact, the most useful part is the study of the graduate students, as it’s likely the least confounded by the effects of a high-stress and taxing work environment. It’s this one that finds effects that match Guzey’s claim.
All this is to highlight the importance of the specific research question and methods. Fatigue from working in a stressful or physically demanding environment is not the same thing as the effects of sleep deprivation. If we are interested in the latter, we should not permit the former to confound our study. This isn’t meant as an attack on you at all, but I think that the impulse to use the results of a meta-study of “sleep deprivation” that is actually a meta-study of work-related fatigue in a high-stress environment with 1⁄6 erroneous inclusions is indicative of the problems Guzey’s highlighting in his overall interpretation and critique of sleep studies.
As a meta point, the fact that a journal in the field is willing to publish such a meta-study that draws misleading conclusions is evidence for Guzey’s claim that the field is cargo-culting and not trustworthy.