when i was new to research, i wouldn’t feel motivated to run any experiment that wouldn’t make it into the paper. surely it’s much more efficient to only run the experiments that people want to see in the paper, right?
now that i’m more experienced, i mostly think of experiments as something i do to convince myself that a claim is correct. once i get to that point, actually getting the final figures for the paper is the easy part. the hard part is finding something unobvious but true. with this mental frame, it feels very reasonable to run 20 experiments for every experiment that makes it into the paper.
when i was new to research, i wouldn’t feel motivated to run any experiment that wouldn’t make it into the paper. surely it’s much more efficient to only run the experiments that people want to see in the paper, right?
now that i’m more experienced, i mostly think of experiments as something i do to convince myself that a claim is correct. once i get to that point, actually getting the final figures for the paper is the easy part. the hard part is finding something unobvious but true. with this mental frame, it feels very reasonable to run 20 experiments for every experiment that makes it into the paper.
What is often left out in papers is all of these experiments and the though chains people had about them.
This is also because of Jevon’s Paradox. As the cost of doing an experiment reduces with experience, the number of experiments run tends to rise.