Heh. This basically describes how I “get a feel” for noisy phenomena. If you have some confidence that the experiment you are running is not super chaotic, you just run it some times and the “probable true outcome” is… the mean of the observed outcomes. I think this can also be reframed as saying that the system of “performing experiments to test noisy phenomena” is ergodic i.e. if you keep performing experiments you will eventually collect a distribution behaviour that is the “average behaviour” of the system.
This heuristic breaks if the system is filled with extreme results/outliers and the outliers are rare. If the extreme results are common, then you should observe the system’s outcomes jumping around wildly in your few trials and assign a correspondingly high level of noise. But if the outliers are rare, you can fool yourself into assuming that you have a good grasp on the “average behaviour” of the system when in practice you’ve just been getting lucky a few times in a row.
Heh. This basically describes how I “get a feel” for noisy phenomena. If you have some confidence that the experiment you are running is not super chaotic, you just run it some times and the “probable true outcome” is… the mean of the observed outcomes. I think this can also be reframed as saying that the system of “performing experiments to test noisy phenomena” is ergodic i.e. if you keep performing experiments you will eventually collect a distribution behaviour that is the “average behaviour” of the system.
This heuristic breaks if the system is filled with extreme results/outliers and the outliers are rare. If the extreme results are common, then you should observe the system’s outcomes jumping around wildly in your few trials and assign a correspondingly high level of noise. But if the outliers are rare, you can fool yourself into assuming that you have a good grasp on the “average behaviour” of the system when in practice you’ve just been getting lucky a few times in a row.