I don’t understand. Why is “they used the wrong statistical formula” worth 47 upvotes on the main article? Because people here are interested in supplementation? Because it’s a fun math problem?
In the other comments, people are discussing which algorithm would be more appropriate, and debating the nuances of each particular method. Not willing to take the time to understand the math, it comes across as, “This could be right, or wrong, depending on such-and-such, and boy isn’t that stupid...”
I run into this problem every time I read anything on health or medicine (it seems limited to these topics). Someone says it’s good for you, someone says it’s bad for you, both sides attack the other’s (complex, expert) methods, and the non-expert is left even more confused than when they first started looking into the matter. And it doesn’t help that personal outcomes can be drastically different regardless of the normal result.
To me, this topic is still confusing, with a slight update toward “take more vitamins.” Without taking classes in statistics and/or medicine, how can I become less wrong on problems like this? Who can I trust, and why?
That is not entirely true. Personally, I think it is a fine article, but I didn’t upvote it because I felt that 60 upvotes should be enough for it. Is there a site-wide consensus about the interpretation of karma scores? If not, is there even a thread where LWers debate about the best semantics?
I don’t understand. Why is “they used the wrong statistical formula” worth 47 upvotes on the main article? Because people here are interested in supplementation? Because it’s a fun math problem?
Both. It’s instrumental in that vitamin supplementation is a concern many here have. It’s also useful as an example of how studies can have flaws, and how these flaws can be found with surprisingly little analysis. Dissections of bad studies helps us avoid similar flaws in our own conclusions. And there are indeed researchers on LessWrong, as well as motivated laymen that can follow the math, and even run their own mathematical regressions. This truly is valuable.
Not willing to take the time to understand the math, … Without taking classes in statistics and/or medicine, how can I become less wrong on problems like this?
You can’t learn to be less wrong about mathematical questions without learning more math. (By definition.)
You can’t learn to be less wrong about mathematical questions without learning more math. (By definition.)
Depends. I could become less wrong about mathematical questions by learning to listen to people who are less wrong about math. (More generally: I may be able to improve my chance of answering a question correctly even if I can’t directly answer it myself.)
The “problem like this” I was referring to was “health advice and information is often faulty,” not “linear regression analysis of mortality effects from supplementation is faulty.”
I’d like to get better at correcting for the former while avoiding the (potentially enormous) amount of learning and effort involved in getting better at all necessary forms of the latter.
From what I can tell, you’re saying “there is no way; the two are inextricably linked.” In which case, I guess I’ll just wait until they get better at it.
All of those points are true, but there’s one I’d like to flag as true but potentially misleading:
Linear regression assumes a linear relationship
Linear regression does assume this in that it tries to find the optimal linear combination of predictors to represent a dependent variable. However, there’s nothing stopping a researcher from feeding in e.g. x and x squared as predictors, and thereby finding the best quadratic relationship between x and some dependent variable.
The way traditional rationalists without special training relate to scientific findings is usually by uncritically accepting them as authoritative. One can become less wrong by learning that scientists are not close to perfect. They make mistakes and sometimes deceive themselves and others. Probably the single most common way this happens is through statistical malpractice. This post, in excellent detail and language non-experts can comprehend, explains one such case and identifies a general type of statistical screw-up: using the wrong tool.
Who can I trust, and why?
Trust no one. Learn a little math.
You don’t need to be able to solve the problems on your own, just enough to understand the arguments. I’m not a math guy either but only some of the statistical stuff is totally out of my grasp. Did you understand the math in this post?
The way traditional rationalists without special training relate to scientific findings is usually by accepting them as authoritative. One can become less wrong by learning that scientists are not close to perfect.
“Trust experts except when you don’t”?
Trust no one. Learn a little math.
“Don’t trust experts; become one yourself”? Wouldn’t that put me in the category of people-not-to-be-trusted? Isn’t that what Phil is pointing out, that most people don’t understand statistics? Why would I expect myself to be better at judging these kinds of problems than experts who spend their lives on it? Should I not expect myself to be just as bad at it, and potentially much worse (know enough to be dangerous)?
Did you understand the math in this post?
Yes. But it seems fundamental enough that experts should have caught it, therefore I am skeptical.
Scientists can be wrong. Certain kinds of science are more likely to involve screw-ups. Learn to identify these kinds of findings and learn to identify sources of screw-ups so you don’t fall for them.
“Don’t trust experts; become one yourself”?
If two experts disagree about something and you want to evaluate the disagreement one way is to understand their arguments. Sometimes you can look into both sides and discover that one of them isn’t really the expert you thought they were. You can evaluate the arguments or evaluate the expertise. I can’t think of anything else.
Why would I expect myself to be better at judging these kinds of problems than experts who spend their lives on it? Should I not expect myself to be just as bad at it, and potentially much worse (know enough to be dangerous)?
I assume you’re not planning on trying to publish statistical analyses so I doubt you’re dangerous.
You can probably learn more about statistics than at least some of the shoddy scientists out there. If you find yourself disagreeing about stats with a prominent statistician then, yeah, you’re probably wrong.
You aren’t learning how to run different kinds of statistical analyses. You’re learning about statistical errors scientists make. It’s a different set of knowledge which means you can know less about statistics in certain ways but still be able to point out where scientists go wrong.
I run into this problem every time I read anything on health or medicine (it seems limited to these topics).
This is an interesting point in itself. Why health and medicine?
Maybe causal inference is straight up more difficult in health and medicine: effects are smaller and more ambiguous than in hard sciences, and have many hard-to-manipulate causes that blur the signal.
There are borderline results in fields like physics, obviously, but they’re usually more esoteric and tend to have relatively clear cut theory behind them (which is why I’d guess you’re not too worried about, say, last year’s ambiguous results from the Cryogenic Dark Matter Search), so they don’t provoke so much back-and-forthing.
This leads me to a prediction: you’d have as much difficulty reading up on results in psychology and sociology as you do in health and medicine. As for what to do about it? Uh...not sure. I’m still chewing over this thread.
This is an interesting point in itself. Why health and medicine?
Maybe causal inference is straight up more difficult in health and medicine: effects are smaller and more ambiguous than in hard sciences, and have many hard-to-manipulate causes that blur the signal.
My self-serving explanation is that health/ medicine/ biology select for people who enjoy (or better tolerate) rote memorization (Levels 0-1 in my hierarchy) rather than “how it works”-type understanding (Levels 2-3). This gets the group of intelligent people with worse ability to know the broader meaning of what they’re doing, a skill that tends to curtail questionable statistical practices.
Yes, I know it sounds insulting, but what really turned me off from taking more biology in high school and college, and from med school, is that it’s so much more memorization-oriented rather than generative-model oriented. This suspicion is confirmed when I hear about e.g. ecologists just now getting around to using the method of adjacency matrix eigenvectors (i.e., Google’s PageRank) to identify key organisms in ecosystems.
And an alternate alternate explanation: Poor priorities. Doctors want to hear all the clinical details, and are mentally worn out by the time they finish with those. There’s just no time or energy to do the math too.
When I used to work for NASA in theoretical air traffic management, I’d try to explain some abstract point about turbulent or chaotic traffic flow to operational FAA guys, and they would get bogged-down in details about what kind of planes we were talking about, what altitudes they were flying at, which airlines they belonged to, and on and on.
You won’t hear that phrase, but I mean the theoretical study of air traffic. For instance, I studied “free flight”, which is when airplanes can fly directly from their takeoff airport to their landing airport and manage their own collision-avoidance, and showed that in certain free-flight situations you can increase the throughput of the airspace by reducing the information that you give to pilots.
It’s an interesting general phenomenon: They try to optimize their route using all of their information. So the more information they have, the more unpredictable their behavior is. More information can actually cause more trouble than it solves in some cases, at least when it’s information about what other agents are doing.
I had a contract from NASA to look for chaotic behavior in en-route air traffic, but my conclusion was that it is unlikely and nothing need be done to avoid it at present.
You don’t get these problems with economics. In economics journals its standard practice to include your specification, as well as the whole regression output, including a full list of included terms and their significance tests.
When I was completing my Master’s degree I was a sessional assistant for an introductory quantitative methods course for economics and finance majors. The type of simple linear regression would be considered overly simplistic at that level (at least in the absence of some simple specification testing), and if the j curve is already accepted in medicine, to model linearly is unforgivable. It’s not like non-linear transformations are hard to do either, you can do them in Excel without too much trouble.
FWIW, I’m of the impression that economists get a better grounding in quantitative methods than other social scientists (and I would say that the profession is a bit too keen on mathematical approaches in some cases), so maybe you would have similar problems with psychology or sociology. But I don’t think economics has this problem.
There are borderline results in fields like physics, obviously, but they’re usually more esoteric and tend to have relatively clear cut theory behind them (which is why I’d guess you’re not too worried about, say, last year’s ambiguous results from the Cryogenic Dark Matter Search), so they don’t provoke so much back-and-forthing.
Also, maybe more importantly, less in the way of financial and ideological commitment.
(Incidentally, my impression is that theoretical debates are more intense in physics than medicine, though I don’t know much about such theoretical debates as might exist in medicine.)
In the other comments, people are discussing which algorithm would be more appropriate, and debating the nuances of each particular method. Not willing to take the time to understand the math, it comes across as, “This could be right, or wrong, depending on such-and-such, and boy isn’t that stupid...”
It shouldn’t. It should come across as “this is wrong, it can make people die, it is evil, don’t listen to it.”
I don’t understand. Why is “they used the wrong statistical formula” worth 47 upvotes on the main article? Because people here are interested in supplementation? Because it’s a fun math problem?
I am approximately cynical enough to suggest “the LessWrong community likes debunking” as a reason.
Is it worth 49 points? Almost certainly not. This indicates a flaw with the karma system, not a flaw in the post.
Another possible explanation: LW clientele likes topics related to survival & life extension. Without turning this into a health discussion group I find these broad strokes of how to think about medical information very valuable.
Why is “they used the wrong statistical formula” worth 47 upvotes on the main article? Because people here are interested in supplementation?
In my case, yes. I’d posit that a future-interested community like this is likely to have lots of supplement-interested individuals trying to squeeze in an extra few years.
Simply put, I think this article has bridged what was a gap between the interests of LessWrong and the interests of LessWrong readers.
I don’t understand. Why is “they used the wrong statistical formula” worth 47 upvotes on the main article? Because people here are interested in supplementation? Because it’s a fun math problem?
In the other comments, people are discussing which algorithm would be more appropriate, and debating the nuances of each particular method. Not willing to take the time to understand the math, it comes across as, “This could be right, or wrong, depending on such-and-such, and boy isn’t that stupid...”
I run into this problem every time I read anything on health or medicine (it seems limited to these topics). Someone says it’s good for you, someone says it’s bad for you, both sides attack the other’s (complex, expert) methods, and the non-expert is left even more confused than when they first started looking into the matter. And it doesn’t help that personal outcomes can be drastically different regardless of the normal result.
To me, this topic is still confusing, with a slight update toward “take more vitamins.” Without taking classes in statistics and/or medicine, how can I become less wrong on problems like this? Who can I trust, and why?
47 votes doesn’t mean “This is a great article”. It means 47 more people liked it than disliked it. Peanut butter gets more karma than caviar.
That is not entirely true. Personally, I think it is a fine article, but I didn’t upvote it because I felt that 60 upvotes should be enough for it. Is there a site-wide consensus about the interpretation of karma scores? If not, is there even a thread where LWers debate about the best semantics?
If you like something you read and would like to see more of it, vote it up.
And vice versa.
Both. It’s instrumental in that vitamin supplementation is a concern many here have. It’s also useful as an example of how studies can have flaws, and how these flaws can be found with surprisingly little analysis. Dissections of bad studies helps us avoid similar flaws in our own conclusions. And there are indeed researchers on LessWrong, as well as motivated laymen that can follow the math, and even run their own mathematical regressions. This truly is valuable.
You can’t learn to be less wrong about mathematical questions without learning more math. (By definition.)
Depends. I could become less wrong about mathematical questions by learning to listen to people who are less wrong about math. (More generally: I may be able to improve my chance of answering a question correctly even if I can’t directly answer it myself.)
The “problem like this” I was referring to was “health advice and information is often faulty,” not “linear regression analysis of mortality effects from supplementation is faulty.”
I’d like to get better at correcting for the former while avoiding the (potentially enormous) amount of learning and effort involved in getting better at all necessary forms of the latter.
From what I can tell, you’re saying “there is no way; the two are inextricably linked.” In which case, I guess I’ll just wait until they get better at it.
The general advice here is
Not all regression is the same; beware anyone who reports doing “a regression”
Linear regression assumes a linear relationship
Don’t trust a report that bases its authority on numbers if you can’t say what those numbers mean
A conclusion can be both true and misleading
A little unreflective folk-psychology (“vitamins” as being “more is better” instead of having a dose-response curve) can do a lot of damage
All of those points are true, but there’s one I’d like to flag as true but potentially misleading:
Linear regression does assume this in that it tries to find the optimal linear combination of predictors to represent a dependent variable. However, there’s nothing stopping a researcher from feeding in e.g. x and x squared as predictors, and thereby finding the best quadratic relationship between x and some dependent variable.
The way traditional rationalists without special training relate to scientific findings is usually by uncritically accepting them as authoritative. One can become less wrong by learning that scientists are not close to perfect. They make mistakes and sometimes deceive themselves and others. Probably the single most common way this happens is through statistical malpractice. This post, in excellent detail and language non-experts can comprehend, explains one such case and identifies a general type of statistical screw-up: using the wrong tool.
Trust no one. Learn a little math.
You don’t need to be able to solve the problems on your own, just enough to understand the arguments. I’m not a math guy either but only some of the statistical stuff is totally out of my grasp. Did you understand the math in this post?
“Trust experts except when you don’t”?
“Don’t trust experts; become one yourself”? Wouldn’t that put me in the category of people-not-to-be-trusted? Isn’t that what Phil is pointing out, that most people don’t understand statistics? Why would I expect myself to be better at judging these kinds of problems than experts who spend their lives on it? Should I not expect myself to be just as bad at it, and potentially much worse (know enough to be dangerous)?
Yes. But it seems fundamental enough that experts should have caught it, therefore I am skeptical.
Some questions (this is an obviously incomplete* list, of course) to ask when you are in this situation:
Is the source pointing out the error reliable?
Does the criticized work acknowledge or otherwise address the claim?
Does the criticized work contain other flaws? (Subcategory: is the criticized work sloppy or lazy in execution?)
In this particular case, the answer to the third question appears to be “yes”. This is probably good reason to raise your probability that this particular criticism is correct.
* Bear in mind, of course, Eliezer Yudkowsky’s warning: If you want to shoot your foot off, it is never the least bit difficult to do so.
Thank you. These steps for analysis are very useful to me, and I feel they answer my original questions.
Scientists can be wrong. Certain kinds of science are more likely to involve screw-ups. Learn to identify these kinds of findings and learn to identify sources of screw-ups so you don’t fall for them.
If two experts disagree about something and you want to evaluate the disagreement one way is to understand their arguments. Sometimes you can look into both sides and discover that one of them isn’t really the expert you thought they were. You can evaluate the arguments or evaluate the expertise. I can’t think of anything else.
I assume you’re not planning on trying to publish statistical analyses so I doubt you’re dangerous.
You can probably learn more about statistics than at least some of the shoddy scientists out there. If you find yourself disagreeing about stats with a prominent statistician then, yeah, you’re probably wrong.
You aren’t learning how to run different kinds of statistical analyses. You’re learning about statistical errors scientists make. It’s a different set of knowledge which means you can know less about statistics in certain ways but still be able to point out where scientists go wrong.
This is an interesting point in itself. Why health and medicine?
Maybe causal inference is straight up more difficult in health and medicine: effects are smaller and more ambiguous than in hard sciences, and have many hard-to-manipulate causes that blur the signal.
There are borderline results in fields like physics, obviously, but they’re usually more esoteric and tend to have relatively clear cut theory behind them (which is why I’d guess you’re not too worried about, say, last year’s ambiguous results from the Cryogenic Dark Matter Search), so they don’t provoke so much back-and-forthing.
This leads me to a prediction: you’d have as much difficulty reading up on results in psychology and sociology as you do in health and medicine. As for what to do about it? Uh...not sure. I’m still chewing over this thread.
My self-serving explanation is that health/ medicine/ biology select for people who enjoy (or better tolerate) rote memorization (Levels 0-1 in my hierarchy) rather than “how it works”-type understanding (Levels 2-3). This gets the group of intelligent people with worse ability to know the broader meaning of what they’re doing, a skill that tends to curtail questionable statistical practices.
Yes, I know it sounds insulting, but what really turned me off from taking more biology in high school and college, and from med school, is that it’s so much more memorization-oriented rather than generative-model oriented. This suspicion is confirmed when I hear about e.g. ecologists just now getting around to using the method of adjacency matrix eigenvectors (i.e., Google’s PageRank) to identify key organisms in ecosystems.
And an alternate alternate explanation: Poor priorities. Doctors want to hear all the clinical details, and are mentally worn out by the time they finish with those. There’s just no time or energy to do the math too.
When I used to work for NASA in theoretical air traffic management, I’d try to explain some abstract point about turbulent or chaotic traffic flow to operational FAA guys, and they would get bogged-down in details about what kind of planes we were talking about, what altitudes they were flying at, which airlines they belonged to, and on and on.
What is theoretical air traffic management?
I mean, in more detail than I can glean just from knowing what those words mean.
You won’t hear that phrase, but I mean the theoretical study of air traffic. For instance, I studied “free flight”, which is when airplanes can fly directly from their takeoff airport to their landing airport and manage their own collision-avoidance, and showed that in certain free-flight situations you can increase the throughput of the airspace by reducing the information that you give to pilots.
It’s an interesting general phenomenon: They try to optimize their route using all of their information. So the more information they have, the more unpredictable their behavior is. More information can actually cause more trouble than it solves in some cases, at least when it’s information about what other agents are doing.
I had a contract from NASA to look for chaotic behavior in en-route air traffic, but my conclusion was that it is unlikely and nothing need be done to avoid it at present.
Reminds me of Braess’s paradox; a route people don’t know about is similar to one that doesn’t exist.
Here’s an alternate insulting explanation, based on many (but not all) of the doctors I’ve met:
Doctors are bad at being unsure of themselves.
You’re right. I mentally boxed them off when I made my original statement. Thinking further, I might add economics to the list.
You don’t get these problems with economics. In economics journals its standard practice to include your specification, as well as the whole regression output, including a full list of included terms and their significance tests.
When I was completing my Master’s degree I was a sessional assistant for an introductory quantitative methods course for economics and finance majors. The type of simple linear regression would be considered overly simplistic at that level (at least in the absence of some simple specification testing), and if the j curve is already accepted in medicine, to model linearly is unforgivable. It’s not like non-linear transformations are hard to do either, you can do them in Excel without too much trouble.
FWIW, I’m of the impression that economists get a better grounding in quantitative methods than other social scientists (and I would say that the profession is a bit too keen on mathematical approaches in some cases), so maybe you would have similar problems with psychology or sociology. But I don’t think economics has this problem.
Also, maybe more importantly, less in the way of financial and ideological commitment.
(Incidentally, my impression is that theoretical debates are more intense in physics than medicine, though I don’t know much about such theoretical debates as might exist in medicine.)
It shouldn’t. It should come across as “this is wrong, it can make people die, it is evil, don’t listen to it.”
I am approximately cynical enough to suggest “the LessWrong community likes debunking” as a reason.
Is it worth 49 points? Almost certainly not. This indicates a flaw with the karma system, not a flaw in the post.
Absolutely. It’s worth the 59 points that it is at now. How science, and health related science in particular is used poorly is valuable information.
Another possible explanation: LW clientele likes topics related to survival & life extension. Without turning this into a health discussion group I find these broad strokes of how to think about medical information very valuable.
In my case, yes. I’d posit that a future-interested community like this is likely to have lots of supplement-interested individuals trying to squeeze in an extra few years.
Simply put, I think this article has bridged what was a gap between the interests of LessWrong and the interests of LessWrong readers.