I’m having a pretty intense reaction to reading certain articles and could use some support or a solution:
Here’s what I read and my reactions:
Feynman’s Cargo Cult Science (Which is about how a lot of scientific studies are done badly, often due to researchers not being allowed to do the research correctly.)
“During many of my 20 years at Stanford University, Albert Bandura and I tried to hold on to a science-based clinical training program. The bizarre situation we faced there is of more than personal and historical interest: I suspect that many of the same conflicts still exist and motivate the efforts described by Baker and colleagues. Bandura and I, and our students and other colleagues, were discovering the remarkable discrepancies between what the scientific work was revealing and the requirements imposed by the pressures for maintaining accreditation. The professional accreditation requirements insisted on continuing practices whose value was contradicted by the empirical findings. Those requirements not only flew in the face of the data but also made enormous demands on faculty and student time in the clinical program.”
What meaning is there in doing anything (being a doctor or a psychologist for instance… or any number of other professions) if we can’t even trust the research or the schooling? How can I make a difference in the world or do anything useful with no real knowledge? How do you find meaning, LessWrong?
Thank goodness I found this place. I am in love with the glimmers of sanity I see here. Before I found LessWrong I was just kind of… “WTF humanity is a mess.” Now it’s more like “WTF humanity is a mess but at least there’s a group of people trying not to be.” If anyone is up to describing this wonderful and horrible feeling in their own words, I could really use to feel related to about this.
Do you know of a website where one can look up a piece of research to see what flaws it has? Is one planned? I need this because it would take a very long time for me to read enough on each relevant topic to discover whether a piece of research I want to use is flawed or not. For instance, Feynman explained about how lots of studies have been done with mazes and rats, but people didn’t seem to realize that the rats were using methods to find the food that were unexpected and all sorts of stuff has to be controlled for ranging from the scent of food to the type of flooring in the maze. If you don’t know that all of these things need to be controlled for, you won’t know that the vast majority of studies done on putting rats into mazes are useless. It’s simply not realistic to expect ourselves to be able to single handedly give every single study we read a thorough enough review to detect all the flaws. I love research, but I now feel that it’s futile. Does anyone know a solution? I know that peer reviewed journals are supposed to address this type of problem, but I don’t see the online studies that I find being rated or marked as flawed in an obvious way.
What meaning is there in doing anything (being a doctor or a psychologist for instance… or any number of other professions) if we can’t even trust the research or the schooling? How can I make a difference in the world or do anything useful with no real knowledge?
That makes things sound worse than they are. I disagree that we have no real knowledge, and I’m also not sure about lumping doctors or psychologists together in this context. In medicine there are effects so huge that explaining them away as publication bias or spurious correlations is implausible (maybe because the relative risk is so huge, as with smoking causing lung cancer, or because the base rate is so low, as with asbestos causing malignant mesothelioma), so I count them as real knowledge. But I don’t know of similarly huge effects in psychology, so psychology might differ in that key respect.
(Here’s a speculative tangent that belongs in brackets. The foregoing might partly explain bad epistemic habits in research. Historically, lots of research went into things we basically fixed with magic bullets. So it didn’t much matter when people suppressed negative results or leaned heavily on observational studies; the true effect of the magic bullets was so huge that it held up despite the biases. This might’ve gotten researchers into the habit of not worrying about, or not finding out about, methodological biases. But now we’re searching for smaller effects those biases matter.)
Better still, most of the problems you refer to above are solvable. We could, for instance,
publish negative results
learn about warning signs that can indicate flaky study results
force researchers to publicly announce trials and their endpoint measures before their trials begin
force researchers to disclose funding sources and possible conflicts of interest
put more effort into searching the grey literature and foreign literature when reviewing studies
focus on randomized experiments (and use placebo controls where applicable in medical or psychological trials) over observational studies
impress the importance of evidence-based methods & treatments on practitioners and professional organizations
use statistical tests for detecting publication biases in reviews and meta-analyses
So, supposing I did accept the premise that the research base is so bad as to make doctors and psychologists useless, there’d still be an obvious alternative to giving up and walking away: I could become an epidemiologist or a medical statistician or a policy pundit, and encourage people to do the things I listed above.
Thank you for responding to this, Satt. I really did need some input here, and it’s very good to see another perspective and to have been shown a whole list of things that could be done.
I am in an unusually bad situation because the subject I’m most interested in is psychology. I noticed something was wrong with the psychology industry while I was still young enough to avoid getting into it. The three main problems are:
That you have to diagnose people immediately to collect insurance payments when in reality it takes a long time to know whether there’s even anything wrong with them at all, and being deemed “messed up” by a professional could be very hurtful to the patient.
I could tell that a lot of what was passing for therapy was BS and decided there must be something drastically wrong with the schooling. I didn’t know that that it was this bad, but I am glad I noticed something was drastically wrong early on.
I am primarily interested in gifted adults. Neither an abnormal psychology degree or developmental psychology degree would give me a solid understanding of gifted adults—those are focused on the average Joe and children with learning disabilities respectively. Gifted adults are neither very well served by the typical therapist (imagine taking a space ship to a car mechanic) or by schooling methods intended for children with learning disabilities. I didn’t realize that my main interest was in gifted adults until later, but I could tell that the psychology that I had been exposed to wasn’t what I was looking for. I have a space ship myself, and wanted psychology that taught me about space ships like mine.
So I went to college for web design instead. I studied psychology on my own. I love being a web developer, a lot, but I want to really make a difference in the world and I don’t feel that adding little buttons to websites is making that happen. Of course, web development can be used for making a difference, too, but if most of what I know about psychology is wrong (it quite possibly could be?) then how am I supposed to pursue my main interest? I was hoping to do self-improvement writing, and I can still do that at any time, and possibly gain an audience that way, but if the foundation of knowledge I am working from is bad, then it’s not useful to do so. What I want to get from writing about self-improvement is meaning, not money, so that would be unacceptable to me.
Something occurred to me: I’ve learned enough about the psychology of gifted adults now that I’d probably have a strong advantage when it comes to writing review articles or meta-analyses on gifted adults. I’m not credentialed, so could not give the articles any traditional “credibility” (that’s in quotes for a reason, now that I know all of this...). However, considering the circumstance (that getting an accredited psychology degree requires you to learn a bunch of mumbo-jumbo and that they don’t teach about gifted adults anyway), I’m thinking that getting a degree would not increase the quality of my articles substantially enough to justify spending tens of thousands of dollars and so many hours on it. Reading the key books on research practices would probably be the best action, though I do not know what they are.
If you (or other LWers) have thoughts on how to approach this sticky problem, I’m interested in hearing them.
What do you mean by “gifted adults”? Just “adults with very high IQ”? I think there’s a standard trick for that when you pen them all together and then you have a regular human society where the social effects of giftedness disappear. Or do gifted people have abnormal psychology in absolute terms, not just relative with alienation and boredom and so on?
There are lots and lots of definitions for “gifted”. State’s legal definitions range from vague things like “people with a talent” to numerical specifications. The gist: I’ve seen definitions that range from a rarity of 1 in 4 to 1 in 50. Truth be told, my real interest is highly gifted adults and geniuses, not just “gifted adults” in general.
From what I’ve read, “highly gifted” tends to be associated with IQs > 145.
The people in each IQ range have their own characteristics. People with IQs near 130 tend to be more popular. People with IQs around 160 or greater have difficulty fitting in and tend to limit social contact because they are too different. These are relative obviously. It has been observed that people with IQs over 145 frequently have enough intensity that it results in them coming across in an energetic way that is called a variety of things from electric to charismatic. This appears to be genetic. There are other things like how exceptionally gifted children have trouble answering “simple” questions and doing “simple” tasks like “draw a bird”—too many options come to mind, and they have to choose, then, between 100 kinds of birds.
This is just the tip of the iceburg when it comes to the differences that have been talked about. I am not sure that any one piece of research I’ve read is true, but there are probably over a hundred differences that have been either researched or observed by psychologists who work with gifted individuals. I have observed a lot of these differences for myself, and have seen patterns. I can also use what I know to make guesses about who is gifted and how gifted they are and I am usually close. I feel certain that there are a huge number of differences of both types, though what, specifically they are and how common they are to each IQ range would be hard to say.
Also, I don’t think it’s called “abnormal psychology” when there’s nothing wrong with them.
Wikipedia addresses this… I was just reading the wiki on the Paleo diet and saw a bunch of stuff about repeatability and study relevance like:
Loren Cordain, a proponent of a low-carbohydrate Paleolithic diet, responded to the U.S. News ranking, stating that their “conclusions are erroneous and misleading” and pointing out that “five studies, four since 2007, have experimentally tested contemporary versions of ancestral human diets and have found them to be superior to Mediterranean diets, diabetic diets and typical western diets in regards to weight loss, cardiovascular disease risk factors and risk factors for type 2 diabetes.”[27] The editors of U.S. News replied that their ranking included a review of all five studies which found that all of them were small and/or of short duration.
I realize Wikipedia isn’t credible for citing or anything but I feel heartened because:
I bet they often link to a credible meta-analysis, making it easier to find them (I’ve been told by Gwern that one way of coping with this is to read a meta-analysis because it gives you a number of advantages over reading individual pieces of research).
It serves as a method for finding out about some of the flaws you need to look for when reading studies on the topic.
It often lists a collection of relevant research, which can save time.
It might be a good starting point for creating your own thorough reviews of studies because a lot of things will already have been hashed out, so it’s just a matter of verifying that what’s there is correct, which should save time if you build on it.
Hm...
Wikipedia is not a perfect solution but I think this will help me cope.
.oO I wonder if there are features that could be added to Wikipedia that would encourage the entries to transform into credible meta-analyses...
A very good Wikipedia article will be equivalent to a review article, but such an article isn’t a meta-analysis: it doesn’t include only studies which can be boiled down to a few summary statistics like d. There’s also little way of being sure that the article is comprehensive and unbiased—one reason meta-analyses usually make a point of how they did a big search on Pubmed and looked through hundreds of results etc.
I don’t know what features could be added to deal with either problem. Any meta-analyses tucked into WP articles would be rightly considered Original Research.
I’m having a pretty intense reaction to reading certain articles and could use some support or a solution:
Here’s what I read and my reactions:
Feynman’s Cargo Cult Science (Which is about how a lot of scientific studies are done badly, often due to researchers not being allowed to do the research correctly.)
The PLOS Medicine article “Why Most Published Research Findings Are False”
An article about how psychologists aren’t usually using the treatments most supported by science which links to a document that contains a horrifying account:
http://www.psychologicalscience.org/journals/pspi/inpress/baker.pdf
I’m having a variety of reactions:
What meaning is there in doing anything (being a doctor or a psychologist for instance… or any number of other professions) if we can’t even trust the research or the schooling? How can I make a difference in the world or do anything useful with no real knowledge? How do you find meaning, LessWrong?
Thank goodness I found this place. I am in love with the glimmers of sanity I see here. Before I found LessWrong I was just kind of… “WTF humanity is a mess.” Now it’s more like “WTF humanity is a mess but at least there’s a group of people trying not to be.” If anyone is up to describing this wonderful and horrible feeling in their own words, I could really use to feel related to about this.
Do you know of a website where one can look up a piece of research to see what flaws it has? Is one planned? I need this because it would take a very long time for me to read enough on each relevant topic to discover whether a piece of research I want to use is flawed or not. For instance, Feynman explained about how lots of studies have been done with mazes and rats, but people didn’t seem to realize that the rats were using methods to find the food that were unexpected and all sorts of stuff has to be controlled for ranging from the scent of food to the type of flooring in the maze. If you don’t know that all of these things need to be controlled for, you won’t know that the vast majority of studies done on putting rats into mazes are useless. It’s simply not realistic to expect ourselves to be able to single handedly give every single study we read a thorough enough review to detect all the flaws. I love research, but I now feel that it’s futile. Does anyone know a solution? I know that peer reviewed journals are supposed to address this type of problem, but I don’t see the online studies that I find being rated or marked as flawed in an obvious way.
That makes things sound worse than they are. I disagree that we have no real knowledge, and I’m also not sure about lumping doctors or psychologists together in this context. In medicine there are effects so huge that explaining them away as publication bias or spurious correlations is implausible (maybe because the relative risk is so huge, as with smoking causing lung cancer, or because the base rate is so low, as with asbestos causing malignant mesothelioma), so I count them as real knowledge. But I don’t know of similarly huge effects in psychology, so psychology might differ in that key respect.
(Here’s a speculative tangent that belongs in brackets. The foregoing might partly explain bad epistemic habits in research. Historically, lots of research went into things we basically fixed with magic bullets. So it didn’t much matter when people suppressed negative results or leaned heavily on observational studies; the true effect of the magic bullets was so huge that it held up despite the biases. This might’ve gotten researchers into the habit of not worrying about, or not finding out about, methodological biases. But now we’re searching for smaller effects those biases matter.)
Better still, most of the problems you refer to above are solvable. We could, for instance,
publish negative results
learn about warning signs that can indicate flaky study results
force researchers to publicly announce trials and their endpoint measures before their trials begin
force researchers to disclose funding sources and possible conflicts of interest
put more effort into searching the grey literature and foreign literature when reviewing studies
focus on randomized experiments (and use placebo controls where applicable in medical or psychological trials) over observational studies
impress the importance of evidence-based methods & treatments on practitioners and professional organizations
use statistical tests for detecting publication biases in reviews and meta-analyses
So, supposing I did accept the premise that the research base is so bad as to make doctors and psychologists useless, there’d still be an obvious alternative to giving up and walking away: I could become an epidemiologist or a medical statistician or a policy pundit, and encourage people to do the things I listed above.
Thank you for responding to this, Satt. I really did need some input here, and it’s very good to see another perspective and to have been shown a whole list of things that could be done.
I am in an unusually bad situation because the subject I’m most interested in is psychology. I noticed something was wrong with the psychology industry while I was still young enough to avoid getting into it. The three main problems are:
That you have to diagnose people immediately to collect insurance payments when in reality it takes a long time to know whether there’s even anything wrong with them at all, and being deemed “messed up” by a professional could be very hurtful to the patient.
I could tell that a lot of what was passing for therapy was BS and decided there must be something drastically wrong with the schooling. I didn’t know that that it was this bad, but I am glad I noticed something was drastically wrong early on.
I am primarily interested in gifted adults. Neither an abnormal psychology degree or developmental psychology degree would give me a solid understanding of gifted adults—those are focused on the average Joe and children with learning disabilities respectively. Gifted adults are neither very well served by the typical therapist (imagine taking a space ship to a car mechanic) or by schooling methods intended for children with learning disabilities. I didn’t realize that my main interest was in gifted adults until later, but I could tell that the psychology that I had been exposed to wasn’t what I was looking for. I have a space ship myself, and wanted psychology that taught me about space ships like mine.
So I went to college for web design instead. I studied psychology on my own. I love being a web developer, a lot, but I want to really make a difference in the world and I don’t feel that adding little buttons to websites is making that happen. Of course, web development can be used for making a difference, too, but if most of what I know about psychology is wrong (it quite possibly could be?) then how am I supposed to pursue my main interest? I was hoping to do self-improvement writing, and I can still do that at any time, and possibly gain an audience that way, but if the foundation of knowledge I am working from is bad, then it’s not useful to do so. What I want to get from writing about self-improvement is meaning, not money, so that would be unacceptable to me.
Something occurred to me: I’ve learned enough about the psychology of gifted adults now that I’d probably have a strong advantage when it comes to writing review articles or meta-analyses on gifted adults. I’m not credentialed, so could not give the articles any traditional “credibility” (that’s in quotes for a reason, now that I know all of this...). However, considering the circumstance (that getting an accredited psychology degree requires you to learn a bunch of mumbo-jumbo and that they don’t teach about gifted adults anyway), I’m thinking that getting a degree would not increase the quality of my articles substantially enough to justify spending tens of thousands of dollars and so many hours on it. Reading the key books on research practices would probably be the best action, though I do not know what they are.
If you (or other LWers) have thoughts on how to approach this sticky problem, I’m interested in hearing them.
What do you mean by “gifted adults”? Just “adults with very high IQ”? I think there’s a standard trick for that when you pen them all together and then you have a regular human society where the social effects of giftedness disappear. Or do gifted people have abnormal psychology in absolute terms, not just relative with alienation and boredom and so on?
There are lots and lots of definitions for “gifted”. State’s legal definitions range from vague things like “people with a talent” to numerical specifications. The gist: I’ve seen definitions that range from a rarity of 1 in 4 to 1 in 50. Truth be told, my real interest is highly gifted adults and geniuses, not just “gifted adults” in general.
From what I’ve read, “highly gifted” tends to be associated with IQs > 145.
The people in each IQ range have their own characteristics. People with IQs near 130 tend to be more popular. People with IQs around 160 or greater have difficulty fitting in and tend to limit social contact because they are too different. These are relative obviously. It has been observed that people with IQs over 145 frequently have enough intensity that it results in them coming across in an energetic way that is called a variety of things from electric to charismatic. This appears to be genetic. There are other things like how exceptionally gifted children have trouble answering “simple” questions and doing “simple” tasks like “draw a bird”—too many options come to mind, and they have to choose, then, between 100 kinds of birds.
This is just the tip of the iceburg when it comes to the differences that have been talked about. I am not sure that any one piece of research I’ve read is true, but there are probably over a hundred differences that have been either researched or observed by psychologists who work with gifted individuals. I have observed a lot of these differences for myself, and have seen patterns. I can also use what I know to make guesses about who is gifted and how gifted they are and I am usually close. I feel certain that there are a huge number of differences of both types, though what, specifically they are and how common they are to each IQ range would be hard to say.
Also, I don’t think it’s called “abnormal psychology” when there’s nothing wrong with them.
http://www.bmj.com/content/331/7514/433
(“Most published research findings are false… including this one.”) (“I heard you like publication bias”)
Whoa neat. Yes, this brings to mind a certain internet meme… (:
suddenly thinks of a coping strategy
Wikipedia addresses this… I was just reading the wiki on the Paleo diet and saw a bunch of stuff about repeatability and study relevance like:
I realize Wikipedia isn’t credible for citing or anything but I feel heartened because:
I bet they often link to a credible meta-analysis, making it easier to find them (I’ve been told by Gwern that one way of coping with this is to read a meta-analysis because it gives you a number of advantages over reading individual pieces of research).
It serves as a method for finding out about some of the flaws you need to look for when reading studies on the topic.
It often lists a collection of relevant research, which can save time.
It might be a good starting point for creating your own thorough reviews of studies because a lot of things will already have been hashed out, so it’s just a matter of verifying that what’s there is correct, which should save time if you build on it.
Hm...
Wikipedia is not a perfect solution but I think this will help me cope.
.oO I wonder if there are features that could be added to Wikipedia that would encourage the entries to transform into credible meta-analyses...
A very good Wikipedia article will be equivalent to a review article, but such an article isn’t a meta-analysis: it doesn’t include only studies which can be boiled down to a few summary statistics like d. There’s also little way of being sure that the article is comprehensive and unbiased—one reason meta-analyses usually make a point of how they did a big search on Pubmed and looked through hundreds of results etc.
I don’t know what features could be added to deal with either problem. Any meta-analyses tucked into WP articles would be rightly considered Original Research.