It’s nice to see an experiment post*, I haven’t seen a lot of these. I think it’s really cool.
*Perhaps LW usually leans too much towards theory, or people are doing experiments but not writing them up.
Starting tomorrow, I’ll be conducting a (99%) blinded experiment to test whether drinking regular vs. decaf coffee has a detectable effect on my mood and alertness.
In an ideal world, I’d pre-register exactly what analyses I intend to do now
I tried to minimize the likelihood further by buying identical decaf and regular grounds but unfortunately couldn’t find a seller that sold the same beans in decaf and regular. In lieu of that, I settled for buying beans from the same region with the same flavor profile (I also don’t have very good taste sense) so as to limit the difference to a visual one.
If you’re worried about analysis you could try explaining your model/experiences in more detail, or collecting data about more variables.
Model example: you didn’t (in this post( consider the possibility in advance that there might exist both types of beans that work as good decaf as not, and beans that don’t, or that some kinds of beans are better than others. I have no reason to believe this is the case, but explicit assumptions might be useful, if only for later experiences. Depending on how you do this experiment, you could in theory find out that you like one kind of coffee/bean more than the other taste wise, even if they have the same effect on alertness.* This brings me to my second example:
*This would require finding out after the fact which coffee was had on which day.
Collecting more data (that could affect what you’re measuring) example: suppose there are other things that could affect your mood or alertness. Writing about these other factors could be useful. (Intuitively, if you got some surprising and really bad/really good news, and this was independent of which type of coffee you had that morning, but has a big impact on your mood, then that might be a good thing to note. Similarly, smaller things** could in theory have an impact on mood or alertness data, though the smaller the effect, the lower the risk of reversing the conclusion incorrectly.)
**Had breakfast, skipped breakfast, went for a walk, etc.
It’s nice to see an experiment post*, I haven’t seen a lot of these. I think it’s really cool.
*Perhaps LW usually leans too much towards theory, or people are doing experiments but not writing them up.
If you’re worried about analysis you could try explaining your model/experiences in more detail, or collecting data about more variables.
Model example: you didn’t (in this post( consider the possibility in advance that there might exist both types of beans that work as good decaf as not, and beans that don’t, or that some kinds of beans are better than others. I have no reason to believe this is the case, but explicit assumptions might be useful, if only for later experiences. Depending on how you do this experiment, you could in theory find out that you like one kind of coffee/bean more than the other taste wise, even if they have the same effect on alertness.* This brings me to my second example:
*This would require finding out after the fact which coffee was had on which day.
Collecting more data (that could affect what you’re measuring) example: suppose there are other things that could affect your mood or alertness. Writing about these other factors could be useful. (Intuitively, if you got some surprising and really bad/really good news, and this was independent of which type of coffee you had that morning, but has a big impact on your mood, then that might be a good thing to note. Similarly, smaller things** could in theory have an impact on mood or alertness data, though the smaller the effect, the lower the risk of reversing the conclusion incorrectly.)
**Had breakfast, skipped breakfast, went for a walk, etc.