When Observation Beats Experiment
Suppose we have a strain of lab rats which are colored purple, and we want to know why. We suspect that chemical X is responsible, so we run an experiment:
We genetically modify our purple rats to repress X production, and find that their purple coloration disappears.
We genetically modify ordinary rats to produce X, and find that their coats turn purple.
We conclude that chemical X is both necessary and sufficient to turn rats’ coats purple. Case closed!
… or maybe not.
Suppose that rats are purple-colored if-and-only-if they express Purple Pigment (PP) above some threshold level. Purple Pigment, in turn, is chemically produced from X and Y:
High levels of PP could result from high levels of X, or from high levels of Y. Either way, increasing X enough will always turn a rat purple, and decreasing X enough will always turn a rat not-purple. So our experiment doesn’t tell us whether our particular rats are purple due to high X or high Y—it could be either. In order to tell the difference, we need to go measure X and Y levels in our rats—not an experiment, but an observation.
(Warning: technical details not relevant to the main point were brushed under the rug there.)
Generalizing: experiments are really good for figuring out the structure of the underlying causal graph. How can we tell that Purple Pigment is produced from X and Y in the first place? Experiment: we try various levels of X and Y and see which rats are purple.
But if we want to know the state of the causal graph, in some real-world system, then observation beats experiment. To find out whether our particular rats are purple because of high X or high Y, we should measure their X and Y levels, without any experimental intervention. Of course, this only works if we’ve already done the experiments to figure out the structure of the system.
Curious about the motivating case for this. I was trying to think of an example where
A. I knew how the system worked.
and
B. I didn’t know the state of the system.
and
C. It wasn’t obvious that I should take a measurement instead of run an experiment.
But I couldn’t think of anything.
I was reading The Biology of Aging. Several times, the author says something to the effect of “experiment beats observation, therefore the gold standard in aging research is to show that adding X accelerates aging and removing X slows it.” For many values of X, experimental manipulation definitely makes organisms live longer/shorter, but it’s not clear that X actually changes significantly during actual aging.
So in this instance, we don’t fully understand how the system works either, but a measurement of X could still tell us whether it’s relevant (based on whether it changes).
Also, what you suggest is not enough to suspect that Y is needed in the first place. This is an assumption that has to be made based on something. In your model, it seems that in some rats even small “doses” of X in the experiment you suggest in the beginning will sometimes coincide with purple coloration. Doesn’t the experiment seem the more straightforward way?
As to the, uh, real systems, I agree that to many whys remain unnoticed, and observation should play a bigger role. That is a curse of a surveyor—you always wonder what you would have found if only you went just a hundred meters further...
Right, we need to use experiments to figure out that Y is needed in the first place. That’s the “figuring out the structure” part—figuring out what the relevant gears are and how they fit together.
Now, experiments also inherently involve some kind of observation. You make a change, then observe the effect of that change. In some cases, the observation built into the experiment may be enough to figure out the system’s state—that’s what happens in your small doses idea. But this is a very indirect (and likely error-prone) way of figuring out that Y is high in our rat strain.
Maybe it would be better to write something like X －＞ PP ＆ Y －＞ PP, because the way it is written it kind of reads to me like “if X is large but Y is not, than PP is not going to be large since it is produced in a reaction”.
Yeah, that gets into the technical details brushed under the rug. There’s two relevant types of equations governing the equilibrium value of PP:
The thermodynamic equilibrium equation, which fixes a product/ratio of the concentrations of the 3 species (linear in log-concentrations)
The stoichiometric constraints, which fix a couple linear combinations of the concentrations of the 3 species (linear in concentrations)
I’m effectively assuming that the thermodynamics favor X + Y over PP, so that the stoichiometric constraints can be approximated as “X and Y concentrations are each fixed”—there’s never enough PP produced to significantly decrease them. That way, we can ignore the stoichiometric limit (so long as X and Y are abundant), and just pay attention to the equilibrium equation. Then log-concentration of PP is a positive linear function of log-concentrations of X and Y, so everything is easy to think about. Increasing/decreasing either X or Y by enough can always shift the equilibrium PP above/below the threshold.
The problem with separate reactions (X → PP and Y → PP) is that, if Y is high, then increasing or decreasing X does nothing—PP will always be above threshold regardless of the X level. It’s an or-gate, rather than a linear function. Similarly, if PP were mainly determined by stoichiometric limits in my original reactions, we’d have an and-gate.
(I will need to think about it tomorrow, but are you effectively saying that the object of the experiment should be that thing Z which modifies what X+Y do?)