Speculation on OP’s education is irrelevant. You reject lots of studies by PhDs that did study the field. If she misunderstood something, address the specific error.
Deep skepticism of the sleep literature is fine, even if you rely on some sleep research yourself, but it’s insufficient to respond to the objection of hypocrisy of relying on the sleep literature with “well, I’m really careful about which studies I use”. You need to explain why the studies you use somehow avoid the methodological problems that cause you to reject other studies. If you don’t, it seems like you are just cherry-picking supporting studies because they support you.
It is SO strange to me that you rely so heavily on your personal experience, which is almost the least reliable scientific method available. Who’s to say that your experience generalizes? You’re a very unusual person. I’m always incredibly skeptical of people who have the position “everyone else’s introspection is unreliable, but somehow I’m above all of that” or “you are all brainwashed by consensus, but I can figure out what’s true and what’s BS because I’m smart, educated, and careful”.
Your claim here relies on an apparent failure to seriously consider the gap between ordinary language and scientific claims, which is a gap I’m confident that you are aware of in other contexts: “I think that the very fact that acute sleep deprivation sometimes increases energy and mood should make almost everyone almost completely re-evaluate everything they know or think about sleep because this should just not happen under the “sleep is restorative and is necessary for good functioning” paradigm.” Here’s the thing. A common error in science articles is using a non-standard, specific, operationalized definition for a concept in the actual study and then using an ordinary language meaning for the concept in the conclusions, leading to overgeneralization. IF science were formal logic, then yes, one counter-example should cause us to reject a general proposition. BUT, science articles that say “sleep is restorative” aren’t claiming “sleep is restorative in every case for every person on every night”, they are making a claim about average population effects. THUS, the fact that acute sleep deprivation sometimes increases energy and mood should not make us think that we must reject everything we know about sleep. This also responds to your no-of-1 proof by contradiction point. An analogy: cold weather makes us feel cold. The fact that people suffering from hypothermia will, just before death, suddenly feel super-hot and take off their clothes doesn’t mean that we should reject everything we know about weather and subjective temperature. It means that a general probabilistic claim doesn’t always apply. That’s why I’m not persuaded by your personal experience. You live a super atypical lifestyle. Your experience makes me think that reduced sleep times is something worth studying further, but doesn’t make me reject existing findings.
Some is addressed above, but I am just floored that someone who is skeptical of much of published science due to methodological problems is so willing to be persuaded by “trusted personal experience” and “trusted anecdote”. I agree that we should be much more skeptical of popularized scientific findings due to valid methodological criticisms, but the solution is not to embrace lousy methodology. If we embraced trusted personal experience and anecdotes as reliable methods for truth findings, we lose the ability to reject massively harmful alternative medicine and new age movements that are propped up by them. The whole reason we set up scientific methods such as systematic data-gathering and double-blind trials is to prevent self-deception.
Again, re: the default claim “sufficient sleep is good and necessary for proper functioning for normal people on the scale of a few days to a week”, this is not a formal logic claim about all cases, it is a claim about trends and norms. And I am not convinced that the default claim is limited to the scale of a few days to a week, but is instead about sleep in general.
I’m pretty surprised at Guzey’s tone in responding; even this last response starting with an apology makes arguments that suggest argument motivated by some sort of psychological trigger rather than rational consideration.
I don’t know if this generalizes, but my experience with tone is that it’s mostly unintentional. There’ve been many instances where I’ve written something that seemed perfectly appropriate to me at the time, only to be horrified at how sound when I read it a month later (and the result pattern-matches to guzey’s comment). It also does not require a psychological trigger, it just happens by default when arguing with someone in text form (and it happens more easily when it’s about something status-related like who made better arguments). Took a lot of deliberate effort to change the default to sounding respectful.
I agree that it’s bad enough to be worth mentioning, but I’d be quite surprised if it’s the result of a strategic effort rather than of an unconscious signaling-related instinct.
I’ve found that I’ve lessened this experience of reading something I’ve wrote and being horrified at its tone by going back and reading my comments at various sites. At least once a year I find myself going to my profile page at LW or some other site and just spending a couple of hours reading what I’ve wrote in the past. I think this has helped me be more aware of what my tone is conveying.
Deep skepticism of the sleep literature is fine, even if you rely on some sleep research yourself, but it’s insufficient to respond to the objection of hypocrisy of relying on the sleep literature with “well, I’m really careful about which studies I use”. You need to explain why the studies you use somehow avoid the methodological problems that cause you to reject other studies. If you don’t, it seems like you are just cherry-picking supporting studies because they support you.
I have a tentative guess on why he’s doing that, based on Scott Alexander’s post about trapped priors.
I’ll give an example of the basic problem outlined in the post myself, to spare you from having to read all of it before understanding my comment. Suppose that a physicist spends two hours trying to convince you that the Earth is flat. Would you see that as strong evidence that the Earth is flat? Personally, I’d see that as extremely weak evidence. Instead of updating much that the Earth is flat, the conversation would instead make me seriously consider the following more-plausible-to-me hypotheses:
The physicist really enjoys pulling very elaborate pranks on people.
It’s April fools or something similar and I for some reason just haven’t realized it yet.
The physicist currently has some sort of untreated psychosis.
Any physics department the physicist has studied or worked on is extremely terrible and ought not to be trusted in the future.
I am hallucinating, or otherwise have perceptions of the world that don’t track reality. Perhaps I’m in a dream, perhaps I have extremely-early-onset Alzheimer’s, perhaps I’m on a potent perception-altering drug.
Similarly, if I saw a lot of studies claiming to show something absurd like that prohibited-by-the-laws-of-physics “psychic” phenomena are real, I won’t need to read their methodology to conclude that there’s something wrong with them. And if a group of people claims to have such psychic powers, I won’t think twice before dismissing their personal experience as unreliable. And, at the same time, I’ll accept, without batting an eye, studies and anecdotes claiming that such powers are not possible.
So dismissing arguments from experts, studies, and personal anecdotes as horribly flawed and no more than weak evidence — even before trying to assess their quality — is perfectly reasonable and Bayesian if they’re claiming something that you think is absurd. But clearly, if you have that attitude towards a belief of yours that does not reflect the territory, that will be problematic.
Guzey’s attitude towards sleep research and anecdotes seems compatible with him having a prior that it’s outright absurd that sleep restriction could be harmful. And, like, it’s not an epistemological sin to be born with that prior. But it does mean that what he sees as very weak evidence against his theses won’t necessarily be very weak evidence for people with different priors, if he can’t explain why he thinks that sleep restriction being harmful is absurd.
(I’m not claiming that that is what is going on here. I just thought that the idea of trapped priors was probably relevant.)
Again, re: the default claim “sufficient sleep is good and necessary for proper functioning for normal people on the scale of a few days to a week”, this is not a formal logic claim about all cases, it is a claim about trends and norms. And I am not convinced that the default claim is limited to the scale of a few days to a week, but is instead about sleep in general.
My main objection to Guzey’s response would be a bit different than yours here. I’d point out that I haven’t argued anywhere that “sufficient sleep” is required for “proper functioning,” and that the meta-analyses I quoted don’t purport to show that either. Rather, they report the magnitude and direction of the effect of experimental sleep restriction on e.g. cognitive ability across a variety of different studies, and the magnitude of the effect they show, although substantial, is a far cry from being enough to outright incapacitate you from most everyday tasks. Guzey seems to be responding to a claim like “people do worse on math exams when they’re tipsy compared to when they’re sober” with “but have you talked to tipsy people? They act almost entirely normally!”
I feel sort of weird about the ‘trapped prior’ point, because I think it’s more reasonably pointed at academic fields than individual people? Like, it is not that surprising for an academic field to have ‘core beliefs’ that everyone who disagrees with is ‘not in the field’, given the forces that people in the field can exert on each other. One fun example of this is PhilGoetz’s post Too Good To Be True, wherein he points to a claim that out of 60 studies studying vaccines and autism, none of them find any link, and then observes that by standard frequentist analysis, ~3 of them should have been significant at the p=0.05 level, and getting none of them significant at that level is pretty unlikely without suppression. And it’s obvious why there would be suppression; no one wants to give ammunition to the enemy.
Separately, it feels like it doesn’t really distinguish ‘justified priors’ from ‘unjustified priors’. If you tell me that a psychology experiment found evidence of psychic effects, I will basically just not believe it. But hopefully if you transported my mind to a universe where psychics were real, I would believe the corresponding studies in those universes—because I had seen things like dowsing being used by oil companies in that universe. This is, from the perspective of a new study, ‘my prior’, but that prior is built out of all of the evidence that I’ve seen before. It’s kind of fair to call my position on psychics a “trapped prior” but it feels more fair to call it “a mountain of evidence”.
If a physicist were to spend two hours trying to explain to me how they knew that the earth was flat, I’d expect to come away from that conversation with a better understanding of the physical world or the social construction of physics knowledge, which would better help me navigate my life, even if I ended up wronger on the bottom-line answer—because that’s how epistemically persuasive explanations work, they have to show an ability to win bets either more often or with less computational cost than alternative hypotheses.
I’m pretty surprised at Guzey’s tone in responding; even this last response starting with an apology makes arguments that suggest argument motivated by some sort of psychological trigger rather than rational consideration.
Maybe better to assume good intentions? (And even if someone is biased or motivated by “impure” motives, we try to turn this into a high quality discussion if we can?)
Fair point, although I wasn’t assuming any bad intentions, more like a hard-to-explain emotional intensity that seemed out of character for someone whose writing I am familiar with. But perhaps expressing my genuine surprise was not constructive—thanks. I removed this intro from my post.
I think that if you ask anyone who knows me in-person they will tell you that I’m an unusually emotionally intensive person. My writing is also usually very emotionally intense but it tend to go through getting feedback from like 20 people who tell me to remove all of the excessive language and to tone it down before publication, so it ends up sounding normal. Comment do not go through this kind of process.
I found this reply unpersuasive.
By numerical point:
Speculation on OP’s education is irrelevant. You reject lots of studies by PhDs that did study the field. If she misunderstood something, address the specific error.
Deep skepticism of the sleep literature is fine, even if you rely on some sleep research yourself, but it’s insufficient to respond to the objection of hypocrisy of relying on the sleep literature with “well, I’m really careful about which studies I use”. You need to explain why the studies you use somehow avoid the methodological problems that cause you to reject other studies. If you don’t, it seems like you are just cherry-picking supporting studies because they support you.
It is SO strange to me that you rely so heavily on your personal experience, which is almost the least reliable scientific method available. Who’s to say that your experience generalizes? You’re a very unusual person. I’m always incredibly skeptical of people who have the position “everyone else’s introspection is unreliable, but somehow I’m above all of that” or “you are all brainwashed by consensus, but I can figure out what’s true and what’s BS because I’m smart, educated, and careful”.
Your claim here relies on an apparent failure to seriously consider the gap between ordinary language and scientific claims, which is a gap I’m confident that you are aware of in other contexts: “I think that the very fact that acute sleep deprivation sometimes increases energy and mood should make almost everyone almost completely re-evaluate everything they know or think about sleep because this should just not happen under the “sleep is restorative and is necessary for good functioning” paradigm.”
Here’s the thing. A common error in science articles is using a non-standard, specific, operationalized definition for a concept in the actual study and then using an ordinary language meaning for the concept in the conclusions, leading to overgeneralization. IF science were formal logic, then yes, one counter-example should cause us to reject a general proposition. BUT, science articles that say “sleep is restorative” aren’t claiming “sleep is restorative in every case for every person on every night”, they are making a claim about average population effects. THUS, the fact that acute sleep deprivation sometimes increases energy and mood should not make us think that we must reject everything we know about sleep. This also responds to your no-of-1 proof by contradiction point.
An analogy: cold weather makes us feel cold. The fact that people suffering from hypothermia will, just before death, suddenly feel super-hot and take off their clothes doesn’t mean that we should reject everything we know about weather and subjective temperature. It means that a general probabilistic claim doesn’t always apply. That’s why I’m not persuaded by your personal experience. You live a super atypical lifestyle. Your experience makes me think that reduced sleep times is something worth studying further, but doesn’t make me reject existing findings.
Some is addressed above, but I am just floored that someone who is skeptical of much of published science due to methodological problems is so willing to be persuaded by “trusted personal experience” and “trusted anecdote”. I agree that we should be much more skeptical of popularized scientific findings due to valid methodological criticisms, but the solution is not to embrace lousy methodology. If we embraced trusted personal experience and anecdotes as reliable methods for truth findings, we lose the ability to reject massively harmful alternative medicine and new age movements that are propped up by them. The whole reason we set up scientific methods such as systematic data-gathering and double-blind trials is to prevent self-deception.
Again, re: the default claim “sufficient sleep is good and necessary for proper functioning for normal people on the scale of a few days to a week”, this is not a formal logic claim about all cases, it is a claim about trends and norms. And I am not convinced that the default claim is limited to the scale of a few days to a week, but is instead about sleep in general.
I have no comment on this!
I don’t know if this generalizes, but my experience with tone is that it’s mostly unintentional. There’ve been many instances where I’ve written something that seemed perfectly appropriate to me at the time, only to be horrified at how sound when I read it a month later (and the result pattern-matches to guzey’s comment). It also does not require a psychological trigger, it just happens by default when arguing with someone in text form (and it happens more easily when it’s about something status-related like who made better arguments). Took a lot of deliberate effort to change the default to sounding respectful.
I agree that it’s bad enough to be worth mentioning, but I’d be quite surprised if it’s the result of a strategic effort rather than of an unconscious signaling-related instinct.
I agree with you.
I’ve found that I’ve lessened this experience of reading something I’ve wrote and being horrified at its tone by going back and reading my comments at various sites. At least once a year I find myself going to my profile page at LW or some other site and just spending a couple of hours reading what I’ve wrote in the past. I think this has helped me be more aware of what my tone is conveying.
I have a tentative guess on why he’s doing that, based on Scott Alexander’s post about trapped priors.
I’ll give an example of the basic problem outlined in the post myself, to spare you from having to read all of it before understanding my comment. Suppose that a physicist spends two hours trying to convince you that the Earth is flat. Would you see that as strong evidence that the Earth is flat? Personally, I’d see that as extremely weak evidence. Instead of updating much that the Earth is flat, the conversation would instead make me seriously consider the following more-plausible-to-me hypotheses:
The physicist really enjoys pulling very elaborate pranks on people.
It’s April fools or something similar and I for some reason just haven’t realized it yet.
The physicist currently has some sort of untreated psychosis.
Any physics department the physicist has studied or worked on is extremely terrible and ought not to be trusted in the future.
I am hallucinating, or otherwise have perceptions of the world that don’t track reality. Perhaps I’m in a dream, perhaps I have extremely-early-onset Alzheimer’s, perhaps I’m on a potent perception-altering drug.
Similarly, if I saw a lot of studies claiming to show something absurd like that prohibited-by-the-laws-of-physics “psychic” phenomena are real, I won’t need to read their methodology to conclude that there’s something wrong with them. And if a group of people claims to have such psychic powers, I won’t think twice before dismissing their personal experience as unreliable. And, at the same time, I’ll accept, without batting an eye, studies and anecdotes claiming that such powers are not possible.
So dismissing arguments from experts, studies, and personal anecdotes as horribly flawed and no more than weak evidence — even before trying to assess their quality — is perfectly reasonable and Bayesian if they’re claiming something that you think is absurd. But clearly, if you have that attitude towards a belief of yours that does not reflect the territory, that will be problematic.
Guzey’s attitude towards sleep research and anecdotes seems compatible with him having a prior that it’s outright absurd that sleep restriction could be harmful. And, like, it’s not an epistemological sin to be born with that prior. But it does mean that what he sees as very weak evidence against his theses won’t necessarily be very weak evidence for people with different priors, if he can’t explain why he thinks that sleep restriction being harmful is absurd.
(I’m not claiming that that is what is going on here. I just thought that the idea of trapped priors was probably relevant.)
My main objection to Guzey’s response would be a bit different than yours here. I’d point out that I haven’t argued anywhere that “sufficient sleep” is required for “proper functioning,” and that the meta-analyses I quoted don’t purport to show that either. Rather, they report the magnitude and direction of the effect of experimental sleep restriction on e.g. cognitive ability across a variety of different studies, and the magnitude of the effect they show, although substantial, is a far cry from being enough to outright incapacitate you from most everyday tasks. Guzey seems to be responding to a claim like “people do worse on math exams when they’re tipsy compared to when they’re sober” with “but have you talked to tipsy people? They act almost entirely normally!”
I feel sort of weird about the ‘trapped prior’ point, because I think it’s more reasonably pointed at academic fields than individual people? Like, it is not that surprising for an academic field to have ‘core beliefs’ that everyone who disagrees with is ‘not in the field’, given the forces that people in the field can exert on each other. One fun example of this is PhilGoetz’s post Too Good To Be True, wherein he points to a claim that out of 60 studies studying vaccines and autism, none of them find any link, and then observes that by standard frequentist analysis, ~3 of them should have been significant at the p=0.05 level, and getting none of them significant at that level is pretty unlikely without suppression. And it’s obvious why there would be suppression; no one wants to give ammunition to the enemy.
Separately, it feels like it doesn’t really distinguish ‘justified priors’ from ‘unjustified priors’. If you tell me that a psychology experiment found evidence of psychic effects, I will basically just not believe it. But hopefully if you transported my mind to a universe where psychics were real, I would believe the corresponding studies in those universes—because I had seen things like dowsing being used by oil companies in that universe. This is, from the perspective of a new study, ‘my prior’, but that prior is built out of all of the evidence that I’ve seen before. It’s kind of fair to call my position on psychics a “trapped prior” but it feels more fair to call it “a mountain of evidence”.
If a physicist were to spend two hours trying to explain to me how they knew that the earth was flat, I’d expect to come away from that conversation with a better understanding of the physical world or the social construction of physics knowledge, which would better help me navigate my life, even if I ended up wronger on the bottom-line answer—because that’s how epistemically persuasive explanations work, they have to show an ability to win bets either more often or with less computational cost than alternative hypotheses.
Maybe better to assume good intentions? (And even if someone is biased or motivated by “impure” motives, we try to turn this into a high quality discussion if we can?)
Fair point, although I wasn’t assuming any bad intentions, more like a hard-to-explain emotional intensity that seemed out of character for someone whose writing I am familiar with. But perhaps expressing my genuine surprise was not constructive—thanks. I removed this intro from my post.
I think that if you ask anyone who knows me in-person they will tell you that I’m an unusually emotionally intensive person. My writing is also usually very emotionally intense but it tend to go through getting feedback from like 20 people who tell me to remove all of the excessive language and to tone it down before publication, so it ends up sounding normal. Comment do not go through this kind of process.