Methodologically, each self-experiment is typically much more poorly run than the kinds of trials we try to discuss here (RCTs), so each self-experiment represents less than n=1 of data. The RCTs usually have at least a few dozen and ideally hundreds or thousands of subjects, either singly or pooled for meta-analysis. So a single such meta-analysis represents thousands of subjects times the fractional quality of a self-experiment, leading to the conclusion that one self-experiment is worth somewhere less than one-hundredths to one-thousandths of any comparable RCT.
The benefits of self experimentation in areas where there has already been actual bulk research to the level where meta analysis can be done is in the realm of personalizing—finding the effects on yourself in particular. Even then the degree to which self experiment can cause you to update the predicted benefits to yourself will depend on the degree to which variability between people is found (unfortunately not mentioned too much in most studies) and the degree to which self reports (or whatever metrics you use) tend to be reliable indicators of actual influence.
This means that if I am self-experimenting with modafinil I will update significantly on how useful the substance is to myself while I wouldn’t even bother self experimenting with respect to how much background noise polution influences my general wellbeing over an extended period. In the latter case p(background sound is detrimental | my self experiment indicates background sound is beneficial) > 0.7.
(Agreeing and elaborating.)
The benefits of self experimentation in areas where there has already been actual bulk research to the level where meta analysis can be done is in the realm of personalizing—finding the effects on yourself in particular. Even then the degree to which self experiment can cause you to update the predicted benefits to yourself will depend on the degree to which variability between people is found (unfortunately not mentioned too much in most studies) and the degree to which self reports (or whatever metrics you use) tend to be reliable indicators of actual influence.
This means that if I am self-experimenting with modafinil I will update significantly on how useful the substance is to myself while I wouldn’t even bother self experimenting with respect to how much background noise polution influences my general wellbeing over an extended period. In the latter case p(background sound is detrimental | my self experiment indicates background sound is beneficial) > 0.7.