Hi! I have a pretty good amount of experience with playing this game—I have a google spreadsheet collecting all sorts of data wrt food, exercise, habits etc. that I’ve been collecting for quite a while. I’ve had some solid successes (I’d say improved quality-of-life by 100x, but starting from an unrepresentatively low baseline), but also can share a few difficulties I’ve had with this approach; I’ll just note some general difficulties and then talk about how this might translate into a useful app sorta thing that one could use.
1. It’s hard to know what data to collect in advance of having the theory you want to test (this applies mainly to whole classes of hypothesis—tracking what you eat is helpful for food intolerance hypotheses, but not as much for things affecting your sleep). - I would recommend starting with a few classes of hypothesis and doing exploratory data analysis once you have data. e.g. I divide my spreadsheet into “food”, “sleep”, “habits tracking”, “activities”, and resultant variables “energy”, “mood”, and “misc. notes” (each estimated for each ~1/3 of the day). These might be different depending on your symptoms, but for non-specific symptoms, are a good place to start. Gut intuitions are, I find, more worth heeding in choosing hypothesis classes than specific hypotheses.
2. Multiple concurrent problems can mean that you might tend to discard hypotheses prematurely. As an example, I’m both Celiac (gluten-free) and soy-intolerant (though in a way that has no relation to soy-intolerance symptoms that I’ve seen online—take this as a datapoint). Getting rid of gluten helped a little, getting rid of soy helped a little, but each individually was barely above the signal-noise threshold; if I were unluckier, I would have missed both. It’s also worth noting that I’ve found “intervened on X, no effect on Y after 3 weeks of perfect compliance” is, in my experience, only moderately strong evidence to discard the hypothesis (rather than damningly strong like it feels) - Things like elimination-diets can help with this if you are willing to invest the effort. If you do it one at a time, it might be worth trying more than once, with a large time interval in between. - If you’re looking for small effects, be aware of what your inter-day variability is; at least for me, there’s a bias to assume that my status on day N is a direct result of days 1-(N-1). Really there’s just some fundamental amount of variability that you have to know to take as your noise threshold. You can measure this by ‘living the same day’ - sounds a little lame, but the value-of-information I’ve found to be high. - If you have cognitive symptoms (in my case, mood fluctuations), note that there might also be some interference in estimating-in-the-moment, which can be counteracted a little bit by taking recordings once at the time, and retroactively again the next ~day.
3. Conditioning on success: as DirectedEvolution says in another comment, the space of hypotheses of things you could do to improve becomes enormous if you allow for a large window of time-delay and conjunctions of hypotheses. I find a useful thinking technique for looking at complicated hypotheses is to ask “would I expect to succeed if the ground-truth were in a hypothesis-class of this level of complexity, and with no incremental improvements by implementing only parts of the plan?”. I don’t feel like I’ve ever had luck testing conjunctions, but timescales are trickier—a physicists estimate would be to take whatever is the variability timescale of symptom severity, and look in this range.
As an overall comment on the app idea: definitely a good idea and I’d love to use it! And super-duper would double-love to have a data-set over “mysterious chronic illnesses like mine”. I think there could ba a lot of value-added also in a different aspect of what you’re talking about accomplishing—specifically, having a well-curated list of “things people have found success by tracking in the past”, and “types of hypothesis which people have found success by testing in the past” might be more valuable than the ability to do a lot of statistics on your data (I’ve found that any hypothesis which is complex enough to need statistics fails my ‘conditioning-on-success’ test)
Hope there’s something useful in here; just something I think about a lot, so sorry if I went on too long ahah. I expect this advice is biased towards reflecting what actually worked for me—food-eliminations rather than other interventions, named-disease-diagnoses, etc. so feel free to correct as you see fit.
Hi! I have a pretty good amount of experience with playing this game—I have a google spreadsheet collecting all sorts of data wrt food, exercise, habits etc. that I’ve been collecting for quite a while. I’ve had some solid successes (I’d say improved quality-of-life by 100x, but starting from an unrepresentatively low baseline), but also can share a few difficulties I’ve had with this approach; I’ll just note some general difficulties and then talk about how this might translate into a useful app sorta thing that one could use.
1. It’s hard to know what data to collect in advance of having the theory you want to test (this applies mainly to whole classes of hypothesis—tracking what you eat is helpful for food intolerance hypotheses, but not as much for things affecting your sleep).
- I would recommend starting with a few classes of hypothesis and doing exploratory data analysis once you have data. e.g. I divide my spreadsheet into “food”, “sleep”, “habits tracking”, “activities”, and resultant variables “energy”, “mood”, and “misc. notes” (each estimated for each ~1/3 of the day). These might be different depending on your symptoms, but for non-specific symptoms, are a good place to start. Gut intuitions are, I find, more worth heeding in choosing hypothesis classes than specific hypotheses.
2. Multiple concurrent problems can mean that you might tend to discard hypotheses prematurely. As an example, I’m both Celiac (gluten-free) and soy-intolerant (though in a way that has no relation to soy-intolerance symptoms that I’ve seen online—take this as a datapoint). Getting rid of gluten helped a little, getting rid of soy helped a little, but each individually was barely above the signal-noise threshold; if I were unluckier, I would have missed both. It’s also worth noting that I’ve found “intervened on X, no effect on Y after 3 weeks of perfect compliance” is, in my experience, only moderately strong evidence to discard the hypothesis (rather than damningly strong like it feels)
- Things like elimination-diets can help with this if you are willing to invest the effort. If you do it one at a time, it might be worth trying more than once, with a large time interval in between.
- If you’re looking for small effects, be aware of what your inter-day variability is; at least for me, there’s a bias to assume that my status on day N is a direct result of days 1-(N-1). Really there’s just some fundamental amount of variability that you have to know to take as your noise threshold. You can measure this by ‘living the same day’ - sounds a little lame, but the value-of-information I’ve found to be high.
- If you have cognitive symptoms (in my case, mood fluctuations), note that there might also be some interference in estimating-in-the-moment, which can be counteracted a little bit by taking recordings once at the time, and retroactively again the next ~day.
3. Conditioning on success: as DirectedEvolution says in another comment, the space of hypotheses of things you could do to improve becomes enormous if you allow for a large window of time-delay and conjunctions of hypotheses. I find a useful thinking technique for looking at complicated hypotheses is to ask “would I expect to succeed if the ground-truth were in a hypothesis-class of this level of complexity, and with no incremental improvements by implementing only parts of the plan?”. I don’t feel like I’ve ever had luck testing conjunctions, but timescales are trickier—a physicists estimate would be to take whatever is the variability timescale of symptom severity, and look in this range.
As an overall comment on the app idea: definitely a good idea and I’d love to use it! And super-duper would double-love to have a data-set over “mysterious chronic illnesses like mine”. I think there could ba a lot of value-added also in a different aspect of what you’re talking about accomplishing—specifically, having a well-curated list of “things people have found success by tracking in the past”, and “types of hypothesis which people have found success by testing in the past” might be more valuable than the ability to do a lot of statistics on your data (I’ve found that any hypothesis which is complex enough to need statistics fails my ‘conditioning-on-success’ test)
Hope there’s something useful in here; just something I think about a lot, so sorry if I went on too long ahah. I expect this advice is biased towards reflecting what actually worked for me—food-eliminations rather than other interventions, named-disease-diagnoses, etc. so feel free to correct as you see fit.
There are also a number of good posts about self-experimentation on slimemoldtimemold.com, and a few more good ones at acesounderglass.com.
From one playing the same game, best wishes in making things better using the scientific method :)