Social psychology seems infamous for this sort of thing; social priming, stereotype threat, ego depletion, etc..
Evolutionary psychology can also be understood as having a sample size that is too small. IME they are doing less bad than social psychology in terms of number of participants, but evolutionary psychology should be interested in cross-cultural universals, and should therefore probably study people across diverse societies (as well as studying animals of other species, perhaps). However, often evolutionary psychology studies only investigate a single society.
If you are willing to generalize the question a bit, there’s also the issue of reliability. For instance, in polls, people may answer a question “differently depending on how they are asked” (really, it seems to me that often the different “ways of asking” are different but highly related questions—but the same point still holds). This introduces noise, and one way to reduce this noise is to ask multiple times in “different ways” and seeing what the overall response is. There’s also reliability statistics like Cronbach’s alpha which have been invented to see whether you’ve done a good enough job with asking in multiple ways. But in some contexts this is often not really done. (This is related to sample size in the sense that it is the sample size of the “transposed data”—where you turn variables into observations and observations into variables.)
Social psychology seems infamous for this sort of thing; social priming, stereotype threat, ego depletion, etc..
Evolutionary psychology can also be understood as having a sample size that is too small. IME they are doing less bad than social psychology in terms of number of participants, but evolutionary psychology should be interested in cross-cultural universals, and should therefore probably study people across diverse societies (as well as studying animals of other species, perhaps). However, often evolutionary psychology studies only investigate a single society.
If you are willing to generalize the question a bit, there’s also the issue of reliability. For instance, in polls, people may answer a question “differently depending on how they are asked” (really, it seems to me that often the different “ways of asking” are different but highly related questions—but the same point still holds). This introduces noise, and one way to reduce this noise is to ask multiple times in “different ways” and seeing what the overall response is. There’s also reliability statistics like Cronbach’s alpha which have been invented to see whether you’ve done a good enough job with asking in multiple ways. But in some contexts this is often not really done. (This is related to sample size in the sense that it is the sample size of the “transposed data”—where you turn variables into observations and observations into variables.)