Emotionally though… when you get a positive result in breast cancer screening even when you’re not at risk, you don’t just shrug and say “probably a false positive” even though it is. Instead, you irrationally do more screenings and possibly get a needless operation.
If you get a positive result, you run another test. If you keep getting positive results, you probably have breast cancer.
Similarly, if an experiment fails to replicate, you try again. If it replicates this time, then it’s probably fine. If it keeps failing to replicate, then there’s a problem.
At the very least, you need to try to replicate a random sample of studies, just to make sure there aren’t more false studies than you’ve been assuming.
Not an expert on cancer, but I don’t think it works that way .I think the cancer test accurately tests a variable wihch is a proxy for cancer risk. So a patient who doesn’t have cancer but tests positive will continue testing positive, because the variable that the cancer test measures as a proxy for cancer is elevated in that patient.
Experiments do work that way, but I’m not arguing against that. I’m only arguing that direct replication isn’t a better use of resources than just going on to a followup experiment with a different methodology (unless direct replication is really easy and you can just have some students do it or something).
Is there only one kind of test? Couldn’t they find another proxy?
I’m only arguing that direct replication isn’t a better use of resources than just going on to a followup experiment with a different methodology
If the followup is testing the same thing with a different methodology, then the metaphor works. If you run followup experiments just to find more detail, it would be like someone testing positive for cancer so then you run a test for what kind of cancer. You’re assuming they have cancer when you run the second test, so the results could be misleading.
If the followup is testing the same thing with a different methodology, then the metaphor works.
Generally an idea is considered well supported when multiple methodologies support it, yes. In the psychology lab I used to work in, at least, we never try to replicate, but we do try to show the same thing in multiple different ways. There are maybe 15 different experiments a year, but they’re generally all centered around proving or dis-proving a cluster of 2 or 3 broad, conceptually linked hypotheses.
Biology labs I’ve worked with do often do the whole “okay, the results are in and this is established now, let’s find additional detail’ thing, but that’s because they were usually looking at much simpler systems, like a single protein or something, so they could afford to take liberties and not be so paranoid about experimental methods.
If you get a positive result, you run another test. If you keep getting positive results, you probably have breast cancer.
Similarly, if an experiment fails to replicate, you try again. If it replicates this time, then it’s probably fine. If it keeps failing to replicate, then there’s a problem.
At the very least, you need to try to replicate a random sample of studies, just to make sure there aren’t more false studies than you’ve been assuming.
Not an expert on cancer, but I don’t think it works that way .I think the cancer test accurately tests a variable wihch is a proxy for cancer risk. So a patient who doesn’t have cancer but tests positive will continue testing positive, because the variable that the cancer test measures as a proxy for cancer is elevated in that patient.
Experiments do work that way, but I’m not arguing against that. I’m only arguing that direct replication isn’t a better use of resources than just going on to a followup experiment with a different methodology (unless direct replication is really easy and you can just have some students do it or something).
Is there only one kind of test? Couldn’t they find another proxy?
If the followup is testing the same thing with a different methodology, then the metaphor works. If you run followup experiments just to find more detail, it would be like someone testing positive for cancer so then you run a test for what kind of cancer. You’re assuming they have cancer when you run the second test, so the results could be misleading.
Generally an idea is considered well supported when multiple methodologies support it, yes. In the psychology lab I used to work in, at least, we never try to replicate, but we do try to show the same thing in multiple different ways. There are maybe 15 different experiments a year, but they’re generally all centered around proving or dis-proving a cluster of 2 or 3 broad, conceptually linked hypotheses.
Biology labs I’ve worked with do often do the whole “okay, the results are in and this is established now, let’s find additional detail’ thing, but that’s because they were usually looking at much simpler systems, like a single protein or something, so they could afford to take liberties and not be so paranoid about experimental methods.