“no statistically significant residual effects” is a fact about sample size, not reality
Sample size and population effect size both factor into the likelihood of obtaining a statistically significant result. So this is a fact about both the experiment and the population, not just the experiment alone.
I’d say sample size is more important if any experiment can get any statistical significance with the right sample size but not any sample size can get any statistical significance with the right experiment. But you’re right, I overstated my case; amended; thank you.
I’d say sample size is more important if any experiment can get any statistical significance with the right sample size but not any sample size can get any statistical significance with the right experiment.
Thanks for listening. I still think that this is a misleading statement.
If we are considering empirical experiments, then our approximation of samples as being iid and sampled with replacement may break down at relatively small sample sizes, invalidating fundamental assumptions of common statistical significance tests.
If we are considering a mathematical model of a random experiment, then when the null hypothesis is true, the probability of a Type I error remains fixed at the chosen level of alpha no matter the sample size.
Sample size and population effect size both factor into the likelihood of obtaining a statistically significant result. So this is a fact about both the experiment and the population, not just the experiment alone.
I’d say sample size is more important if any experiment can get any statistical significance with the right sample size but not any sample size can get any statistical significance with the right experiment. But you’re right, I overstated my case; amended; thank you.
Thanks for listening. I still think that this is a misleading statement.
If we are considering empirical experiments, then our approximation of samples as being iid and sampled with replacement may break down at relatively small sample sizes, invalidating fundamental assumptions of common statistical significance tests.
If we are considering a mathematical model of a random experiment, then when the null hypothesis is true, the probability of a Type I error remains fixed at the chosen level of alpha no matter the sample size.