Great point! It’s a very similar problem, with a very similar solution. We have some complicated system with a large number of lines/variables which could influence the outcome (i.e. the bug), and the main problem is to figure out which lines/variables mediate the influence of everything else. The first step is to reproduce the bug—i.e. hunt down all the sources of “randomness”, until we can make the bug happen consistently. After that, the next step is to look for mediation—i.e. find lines/variables which are “in between” our original reproduction-inputs and the bug itself, and which are themselves sufficient to reproduce the problem.
An issue I find with debugging a complex program is that when you write tests (which put inputs into part of the program and then check whether the expected output is produced), your tests can themselves contain bugs, and often do (if they’re not trivially simple). That is, your experiments isolating a small set of variables can produce confusing results due to the experimental design, not just unpredictability in what they’re trying to test. Eg maybe your way of measuring the slope angle or sled weight is flawed. (Cf assumptions about the speed/straightness of light or a steady-state universe messing up your astronomical observations). As philosophers of science say, all observation is theory-laden.
Your description of the second type of science where you repeatedly control variables to isolate one reminds me a lot of debugging a complex program.
Great point! It’s a very similar problem, with a very similar solution. We have some complicated system with a large number of lines/variables which could influence the outcome (i.e. the bug), and the main problem is to figure out which lines/variables mediate the influence of everything else. The first step is to reproduce the bug—i.e. hunt down all the sources of “randomness”, until we can make the bug happen consistently. After that, the next step is to look for mediation—i.e. find lines/variables which are “in between” our original reproduction-inputs and the bug itself, and which are themselves sufficient to reproduce the problem.
An issue I find with debugging a complex program is that when you write tests (which put inputs into part of the program and then check whether the expected output is produced), your tests can themselves contain bugs, and often do (if they’re not trivially simple). That is, your experiments isolating a small set of variables can produce confusing results due to the experimental design, not just unpredictability in what they’re trying to test. Eg maybe your way of measuring the slope angle or sled weight is flawed. (Cf assumptions about the speed/straightness of light or a steady-state universe messing up your astronomical observations). As philosophers of science say, all observation is theory-laden.