I’m curious if anybody here frequents retraction watch enough to address this concern I have.
I find articles here very effective at announcing retractions and making testimonies from lead figures in investigations a frequent fallback, but rarely do you get to see the nuts and bolts of the investigations being discussed. For example, “How were the journals misleading?” or “What evidence was or was not analyzed, and how did the journal’s analysis deviate from correct protocol?” are questions I often ask myself as I read, followed by an urge to see the cited papers. And then upon investigating the articles and their retraction notices, I am given a reason that I can’t myself arbitrate. Maybe data was claimed to have been manipulated, or analyzed according to an incorrect framework.
Studies such as these I find alarming because I’m forced to trust the good intentions of a multi-billion dollar corporation in finding the truth. Often I find myself going on retraction watch, trusting the possibly non-existing good intentions of the organization’s leadership, as I read the headlines without time to read every detail of the article. I am given certain impressions from the pretentious writing of the articles, but none of the substance, when I choose to skim selections.
Perhaps I am warning against laziness. Perhaps I am concerned about the potential for corruption in even a crusade to fight misinformation that retraction watch seems to fight. Nonetheless, I’m curious if people here have had similar or differing experiences with these articles...
“Beall, who became an assistant professor, drew up a list of the known and suspected bad apples, known simply as Beall’s List. Since 2012, this list has been world’s main source of information on journals that publish conspiracy theories and incompetent research, making them appear real.”
Why are you consuming research at all? If you are a researcher considering building on someone else’s research, then you probably shouldn’t trust them and should replicate everything you really need. But you are also privy to a lot of gossip not on LW and so have a good grasp on base rates. If you are considering using a drug, then it has been approved by the FDA, which performs a very thorough check on the drug company. The FDA has access to all the raw data and performs all the analysis from scratch. The FDA has a lot of problems, but letting studies of new drugs get away with fraud is not one of them. But if you want to take a drug off-label, then you are stuck with research.
You say that you don’t trust the intentions of a multi-billion dollar corporations. Have you thought about what those intentions are? They don’t care about papers. Their main goal is to get the drug approved by the FDA. Their goal is for their early papers to be replicated by big, high quality, highly monitored studies. Whereas, the goal of multi-billion dollar universities is mainly to produce papers with too much focus on quantity and too little on replication.
I’m no researcher, and you’re right, if I did want to improve upon my study, I would, given the materials. However, I am not that affluent, I do not have such opportunities unless the research was based on coatings and adhesives (these materials I do have access to). The retraction I linked was merely presented on retraction watch as an example. An example for what? Let’s continue to...
Have you thought about what those intentions are?
My understanding is that as a public company your primary concern is bringing in enough value to the company to appease investors. A subset of that goal would be to get FDA approved.
I don’t trust the company because of the incentive system, and that is my gut reaction that stems from companies getting away with unscrupulous business practices in the past. Though now that I think about it, however, Pfizer would have nothing to gain from retracting papers they knew they couldn’t back up if someone asked them to. My guess is that either:
a) Min-Jean’s managers were planning on gambling with her research only to find out their moles in the FDA wouldn’t cooperate or,
b) there was no conspiracy, and Min-Jean was incentivized to fabricate her work on her own volition.
I do see your point, since a) is a more complicated theory in this case. But I distrust the situation. I smell a power play, at worst. But I can’t support that, unfortunately, from the articles alone. I can support power plays happening in big companies, but I can’t show those situations are related here. Not yet, anyway…
EDIT: With all that said, you seem to err on the side of trusting the FDA to do their job and trusting Pfizer to comply. Would you be able to back up that trust in this case alone?
I think waveman made my point clearer in that I don’t like the fact that I don’t know the details of the investigation. Down to the painfully detailed process of verifying image duplication. I’m not so sure a quick phone call to Pfizer or Min-Jean would help me either...
the FDA, which performs a very thorough check on the drug company
I think you have an overly sunny view of how effective the FDA is. (leaving aside the question of cost effectiveness and the opportunity cost of the delays and even outright prevention of useful drugs getting to market and their effect on the cost of drugs)
There are plenty of cases of the FDA being hoodwinked by drug companies. Regulatory capture is always a concern.
Statistical incompetence is very common. I still cannot believe that they let Vioxx on the market when the fourfold increase in heart attacks had a P value of about 10-11%. . This is the sort of stupidity that would (or should) get you as F in Statistics 101.
My experience over many decades is that over time the benefits of drugs often turn out to be way overstated and the dangers greatly underestimated.
I’m curious if anybody here frequents retraction watch enough to address this concern I have.
I find articles here very effective at announcing retractions and making testimonies from lead figures in investigations a frequent fallback, but rarely do you get to see the nuts and bolts of the investigations being discussed. For example, “How were the journals misleading?” or “What evidence was or was not analyzed, and how did the journal’s analysis deviate from correct protocol?” are questions I often ask myself as I read, followed by an urge to see the cited papers. And then upon investigating the articles and their retraction notices, I am given a reason that I can’t myself arbitrate. Maybe data was claimed to have been manipulated, or analyzed according to an incorrect framework.
Studies such as these I find alarming because I’m forced to trust the good intentions of a multi-billion dollar corporation in finding the truth. Often I find myself going on retraction watch, trusting the possibly non-existing good intentions of the organization’s leadership, as I read the headlines without time to read every detail of the article. I am given certain impressions from the pretentious writing of the articles, but none of the substance, when I choose to skim selections.
Perhaps I am warning against laziness. Perhaps I am concerned about the potential for corruption in even a crusade to fight misinformation that retraction watch seems to fight. Nonetheless, I’m curious if people here have had similar or differing experiences with these articles...
A blog list of bogus journals just went down too...
http://ottawacitizen.com/storyline/worlds-main-list-of-science-predators-vanishes-with-no-warning
“Beall, who became an assistant professor, drew up a list of the known and suspected bad apples, known simply as Beall’s List. Since 2012, this list has been world’s main source of information on journals that publish conspiracy theories and incompetent research, making them appear real.”
Crimes and trials are the same. Much goes on in closed rooms. You rightly feel that you are in the dark.
Often there is some material on pubpeer which can help understand what happened.
Gelman’s blog goes into messy details often enough.
No, you’re not. You are offered some results, you do NOT have to trust them.
Thanks for the tool.
Indeed, but I suppose the tool provided solves my problem of judging when data was misanalysed as I could just as easily do the analysis myself.
Why are you consuming research at all? If you are a researcher considering building on someone else’s research, then you probably shouldn’t trust them and should replicate everything you really need. But you are also privy to a lot of gossip not on LW and so have a good grasp on base rates. If you are considering using a drug, then it has been approved by the FDA, which performs a very thorough check on the drug company. The FDA has access to all the raw data and performs all the analysis from scratch. The FDA has a lot of problems, but letting studies of new drugs get away with fraud is not one of them. But if you want to take a drug off-label, then you are stuck with research.
You say that you don’t trust the intentions of a multi-billion dollar corporations. Have you thought about what those intentions are? They don’t care about papers. Their main goal is to get the drug approved by the FDA. Their goal is for their early papers to be replicated by big, high quality, highly monitored studies. Whereas, the goal of multi-billion dollar universities is mainly to produce papers with too much focus on quantity and too little on replication.
I’m no researcher, and you’re right, if I did want to improve upon my study, I would, given the materials. However, I am not that affluent, I do not have such opportunities unless the research was based on coatings and adhesives (these materials I do have access to). The retraction I linked was merely presented on retraction watch as an example. An example for what? Let’s continue to...
My understanding is that as a public company your primary concern is bringing in enough value to the company to appease investors. A subset of that goal would be to get FDA approved.
I don’t trust the company because of the incentive system, and that is my gut reaction that stems from companies getting away with unscrupulous business practices in the past. Though now that I think about it, however, Pfizer would have nothing to gain from retracting papers they knew they couldn’t back up if someone asked them to. My guess is that either:
a) Min-Jean’s managers were planning on gambling with her research only to find out their moles in the FDA wouldn’t cooperate or,
b) there was no conspiracy, and Min-Jean was incentivized to fabricate her work on her own volition.
I do see your point, since a) is a more complicated theory in this case. But I distrust the situation. I smell a power play, at worst. But I can’t support that, unfortunately, from the articles alone. I can support power plays happening in big companies, but I can’t show those situations are related here. Not yet, anyway…
EDIT: With all that said, you seem to err on the side of trusting the FDA to do their job and trusting Pfizer to comply. Would you be able to back up that trust in this case alone?
I think waveman made my point clearer in that I don’t like the fact that I don’t know the details of the investigation. Down to the painfully detailed process of verifying image duplication. I’m not so sure a quick phone call to Pfizer or Min-Jean would help me either...
I think you have an overly sunny view of how effective the FDA is. (leaving aside the question of cost effectiveness and the opportunity cost of the delays and even outright prevention of useful drugs getting to market and their effect on the cost of drugs)
There are plenty of cases of the FDA being hoodwinked by drug companies. Regulatory capture is always a concern.
Statistical incompetence is very common. I still cannot believe that they let Vioxx on the market when the fourfold increase in heart attacks had a P value of about 10-11%. . This is the sort of stupidity that would (or should) get you as F in Statistics 101.
My experience over many decades is that over time the benefits of drugs often turn out to be way overstated and the dangers greatly underestimated.
I stand by my narrow claims. Here is another narrow claim: you are wrong about what happened with Vioxx.
People can read about it for themselves.
https://en.wikipedia.org/wiki/Rofecoxib