Those sciences are based on observations. Controlled experimentation requires that you have some set of experimental units to which you randomly assign treatments. With geology, for instance, you are trying to figure out the structure of the Earth’s crust (mostly). There are no real treatments that you apply, instead you observe the “treatments” that have been applied by the earth to the earth. I.e. you can’t decide which area will have a volcano, or an earthquake: you can’t choose to change the direction of a plate or change the configuration of the plates: you can’t change the chemical composition of the rock under large scale, etc.
All one can do is carefully collect measurements, build models of them, and attempt to create a cohesive picture that explains the phenomena. Control implies that you can do more than just collect measurements.
Of the examples given, some of them certainly involve controled experiments in the classical sense. Evolutionary biology for example involves tests of genetic drift and speciation in the lab environment. For example, one matter that has been extensively tested in labs is different speciation mechanisms. The founder-effect mechanism is one that is particularly easy to test in a lab. For one major paper on the subject see this paper. A much older example is speciation by hybridization which has been tested in controlled lab environments for about a century now. The oldest I’m aware of in that regard is a 1912 paper by Digby (I haven’t read it, and I’d have to go look up the citation but it shouldn’t be hard to find ), and there have been many papers since then on the same topic.
Edit:Citation for Digby according to TOA is Digby, L. 1912. The cytology of Primula kewensis and of other related Primula hybrids. Ann. Bot. 26:357-388.
All the sciences mentioned above definitely do rely on controlled experimentation. But their central empirical questions are not amenable to being directly studied by controlled experimentation. We don’t have multiple earths or natural histories upon which we can draw inference about the origins of species.
There is a world of difference between saying “I have observed speciation under these laboratory conditions” and “speciation explains observed biodiversity”. These are distinct types of inferences. This of course does not mean that people who perform inference on natural history don’t use controlled experiments: indeed they should draw on as much knowledge as possible about the mechanisms of the world in order to construct plausible theories of the past: but they can’t run the world multiple times under different conditions to test their theories of the past in the way that we can test speciation.
I think that is a non-standard interpretation of the terminology:
“A controlled experiment generally compares the results obtained from an experimental sample against a control sample, which is practically identical to the experimental sample except for the one aspect whose effect is being tested (the independent variable).”
It never says the control sample has been influenced by the experimenter. It could instead be chosen by the experimenter—from the available spectrum of naturally-occurring phenomena.
I think it’s standard in the literature:
“The word experiment is used in a quite precise sense to mean an investigation where the system under study is under the control of the investigator. This means that the individuals or material investigated, the nature of the treatments or manipulations under study and the measurement procedures used are all settled, in their important features at least, by the investigator.”
The theory of the design of experiments
To be sure there are geological experiments where one, say, takes rock samples and subjects various samples to a variety of treatments, in order to simulate potential natural processes. But there is another chunk of the science which is meant to describe the Earth’s geological history and for a controlled experiment on that you would need to control the natural forces of the Earth and to have multiple Earths.
The reason why one needs to control an experiment (this is a point elaborated on at length in Cox and Reid) is in order to prevent bias. Take the hypothesis of continental drift. We have loads of “suspicious coincidences” that suggest continental drift (such as similar fossils on different landmasses, certain kinds of variations in the magnetic properties of the seafloor, the fact that the seafloor rocks are much younger than land rocks, earthquake patterns/fault-lines). Critically, however, we don’t have an example of an earth that doesn’t have continental drift. It is probably the case that some piece of “evidence” currently used to support the theory of continental drift will turn out to be a spurious correlation. Its very difficult to test for these because of the lack of control. The fact that we are almost certainly on a continental-drifting world biases us towards think that some geological phenomenon is caused by drift even when they not.
I think the usage in the cited book is bad and unorthodox. E.g. one can still study storms experimentally—though nobody can completely control a storm.
I think we are talking past each other. I agree that those are experiments in a broad and colloquial use of the term. They aren’t “controlled” experiments: which is a term that I was wanting to clarify (since I know a little bit about it). This means that they do not allow you to randomly assign treatments to experimental units which generally means that the risk of bias is greater (hence the statistical analysis must be done with care and the conclusions drawn should face greater scrutiny).
Pick up any textbook on statistical design or statistical analysis of experiments and the framework I gave will be what’s in there for “controlled experimentation”. There are other types of experiments. But these suffer from the problem that it can be difficult to sort out hidden causes. Suppose we want to know if the presence of A causes C (say eating meat causes heart disease). In an observational study we find units having trait A and those not (so find meat-eaters and vegetarians) and we then wait to observe response C. If we observe a response C in experimental units possessing trait A, its hard to know if A causes C or if there is some third trait B (present in some of the units) which causes both A and C.
In the case of a controlled experiment, A is now a treatment and not a trait of the units (so in this case you would randomly assign a carnivorous or vegetarian diet to people), thus we can randomly assign A to the units (and assume the randomization means that not every unit having hidden trait B will be given treatment A). In this case we might observe that A and C have no relation, whereas in the observational study we might. (For instance people who choose to be vegetarian may be more focused on health)
An example of how econometricians have dealt with “selection bias” or the fact that observation studies fail to have certain nice properties of controlled experiments is here
I believe they borrow results from fields that do have controlled experimentation, but they don’t have a great deal on their own—they concern the analysis of historical data, I believe. Unless “search this region of sky to see whether Planet X is there” counts as a controlled experiment.
I’m not an authority on any of these four fields, though—am I mistaken?
It’s a “controlled experiment” if there’s a sample which is used as a control. Whether that sample is under the control of the experimenter (or arises naturally) is a totally different issue:
Surely those have “controlled experimentation”.
Those sciences are based on observations. Controlled experimentation requires that you have some set of experimental units to which you randomly assign treatments. With geology, for instance, you are trying to figure out the structure of the Earth’s crust (mostly). There are no real treatments that you apply, instead you observe the “treatments” that have been applied by the earth to the earth. I.e. you can’t decide which area will have a volcano, or an earthquake: you can’t choose to change the direction of a plate or change the configuration of the plates: you can’t change the chemical composition of the rock under large scale, etc.
All one can do is carefully collect measurements, build models of them, and attempt to create a cohesive picture that explains the phenomena. Control implies that you can do more than just collect measurements.
Of the examples given, some of them certainly involve controled experiments in the classical sense. Evolutionary biology for example involves tests of genetic drift and speciation in the lab environment. For example, one matter that has been extensively tested in labs is different speciation mechanisms. The founder-effect mechanism is one that is particularly easy to test in a lab. For one major paper on the subject see this paper. A much older example is speciation by hybridization which has been tested in controlled lab environments for about a century now. The oldest I’m aware of in that regard is a 1912 paper by Digby (I haven’t read it, and I’d have to go look up the citation but it shouldn’t be hard to find ), and there have been many papers since then on the same topic.
Edit:Citation for Digby according to TOA is Digby, L. 1912. The cytology of Primula kewensis and of other related Primula hybrids. Ann. Bot. 26:357-388.
All the sciences mentioned above definitely do rely on controlled experimentation. But their central empirical questions are not amenable to being directly studied by controlled experimentation. We don’t have multiple earths or natural histories upon which we can draw inference about the origins of species.
There is a world of difference between saying “I have observed speciation under these laboratory conditions” and “speciation explains observed biodiversity”. These are distinct types of inferences. This of course does not mean that people who perform inference on natural history don’t use controlled experiments: indeed they should draw on as much knowledge as possible about the mechanisms of the world in order to construct plausible theories of the past: but they can’t run the world multiple times under different conditions to test their theories of the past in the way that we can test speciation.
I think that is a non-standard interpretation of the terminology:
“A controlled experiment generally compares the results obtained from an experimental sample against a control sample, which is practically identical to the experimental sample except for the one aspect whose effect is being tested (the independent variable).”
http://en.wikipedia.org/wiki/Experiment#Controlled_experiments
It never says the control sample has been influenced by the experimenter. It could instead be chosen by the experimenter—from the available spectrum of naturally-occurring phenomena.
I think it’s standard in the literature: “The word experiment is used in a quite precise sense to mean an investigation where the system under study is under the control of the investigator. This means that the individuals or material investigated, the nature of the treatments or manipulations under study and the measurement procedures used are all settled, in their important features at least, by the investigator.” The theory of the design of experiments
To be sure there are geological experiments where one, say, takes rock samples and subjects various samples to a variety of treatments, in order to simulate potential natural processes. But there is another chunk of the science which is meant to describe the Earth’s geological history and for a controlled experiment on that you would need to control the natural forces of the Earth and to have multiple Earths.
The reason why one needs to control an experiment (this is a point elaborated on at length in Cox and Reid) is in order to prevent bias. Take the hypothesis of continental drift. We have loads of “suspicious coincidences” that suggest continental drift (such as similar fossils on different landmasses, certain kinds of variations in the magnetic properties of the seafloor, the fact that the seafloor rocks are much younger than land rocks, earthquake patterns/fault-lines). Critically, however, we don’t have an example of an earth that doesn’t have continental drift. It is probably the case that some piece of “evidence” currently used to support the theory of continental drift will turn out to be a spurious correlation. Its very difficult to test for these because of the lack of control. The fact that we are almost certainly on a continental-drifting world biases us towards think that some geological phenomenon is caused by drift even when they not.
Natural experiments are experiments too. See:
http://en.wikipedia.org/wiki/Natural_experiment
http://en.wikipedia.org/wiki/Experiment
http://dictionary.reference.com/browse/experiment
I think the usage in the cited book is bad and unorthodox. E.g. one can still study storms experimentally—though nobody can completely control a storm.
I think we are talking past each other. I agree that those are experiments in a broad and colloquial use of the term. They aren’t “controlled” experiments: which is a term that I was wanting to clarify (since I know a little bit about it). This means that they do not allow you to randomly assign treatments to experimental units which generally means that the risk of bias is greater (hence the statistical analysis must be done with care and the conclusions drawn should face greater scrutiny).
Pick up any textbook on statistical design or statistical analysis of experiments and the framework I gave will be what’s in there for “controlled experimentation”. There are other types of experiments. But these suffer from the problem that it can be difficult to sort out hidden causes. Suppose we want to know if the presence of A causes C (say eating meat causes heart disease). In an observational study we find units having trait A and those not (so find meat-eaters and vegetarians) and we then wait to observe response C. If we observe a response C in experimental units possessing trait A, its hard to know if A causes C or if there is some third trait B (present in some of the units) which causes both A and C.
In the case of a controlled experiment, A is now a treatment and not a trait of the units (so in this case you would randomly assign a carnivorous or vegetarian diet to people), thus we can randomly assign A to the units (and assume the randomization means that not every unit having hidden trait B will be given treatment A). In this case we might observe that A and C have no relation, whereas in the observational study we might. (For instance people who choose to be vegetarian may be more focused on health)
An example of how econometricians have dealt with “selection bias” or the fact that observation studies fail to have certain nice properties of controlled experiments is here
I believe they borrow results from fields that do have controlled experimentation, but they don’t have a great deal on their own—they concern the analysis of historical data, I believe. Unless “search this region of sky to see whether Planet X is there” counts as a controlled experiment.
I’m not an authority on any of these four fields, though—am I mistaken?
It’s a “controlled experiment” if there’s a sample which is used as a control. Whether that sample is under the control of the experimenter (or arises naturally) is a totally different issue:
http://en.wikipedia.org/wiki/Natural_experiment
http://en.wikipedia.org/wiki/Experiment#Controlled_experiments
Fair enough—the author has changed the wording of the main article in any case.