Part of me strongly agrees with you. At the same time… I don’t think “how much work would it have been to address [single bad thing]?” is the right unit of measurement. The actual question is “what would the cost/benefit analysis of address every single thing in this class of problems?”, and that’s a lot less clear cut.
Some things that make it difficult:
There are just a lot of potential problems.
There are an infinite number of things they didn’t research. Every time they add one to the explicit list of unresearched things they make the list as a whole harder to read (-> people are more likely to miss warnings relevant to them) while also conveying more safety and confidence. God forbid people decide anything not on the list of “things we didn’t check” was in fact checked.
You can say they should estimate the danger from each of these and only explicitly disavow research over a certain level of importance, but that’s an inherently noisy process people are likely to disagree on, and no matter what there has to be something just below the line. So again adding warnings can leave people with an inaccurately increased feeling of safety.
When I share medical info, I always preface it with “anything with a real effect can hurt you, think this through for yourself”. I think that’s a little better than what SMTM did. But I don’t see any easy additions to their statements of “If at any point you get sick or begin having side-effects, stop the diet immediately. We can still use your data up to that point, and we don’t want anything to happen to you” and “We are not doctors. We are 20 rats in a trenchcoat. eee! eee! eee!” that make things obviously better.
You note that you felt obliged to keep going in order to provide better data. Maybe there was pressure in private communication, but AFAICT they explicitly spoke against that in the blog post. I had a similar thing happen in my experiment- despite explicitly saying people could drop out at any time and would still be paid, one person was extremely reluctant to do so despite bad side effects and needed a strong push from me. In some sense I don’t think this is fair to put on experimenters, people shouldn’t even need to be told “stop if you feel bad”. But this seems like a pattern, and therefore should be taken into account even if it’s not objectively fair.
“what would the cost/benefit analysis of address every single thing in this class of problems?”,
I mean, that is why I included “More research to shrink our unknown unknowns” as a general category. I do not think the research needs to be thorough, 3-4 very broad general area would suffice, but even if that does not fit into the points you mention, a statement along the lines you mentioned could work. In general, I do not think that more than one-two hours should be spent on writing this warning.
As for “If at any point you get sick or begin having side effects, stop the diet immediately”, that is indeed a good first step. My problem with it is that I am now understanding that you can develop side effects that indicate one should stop, that are not apparent unless you really track several variables intently. A general instruction to, in addition to doing one’s own research, decide what to look for in advance could have worked.
You note that you felt obliged to keep going in order to provide better data
I want to note three things:
I did not feel pressured by SMTM either in their public or private communications at any point
That is not quite correct. I wanted to keep going for the data to be included as a data point, especially if the diet turned out not to work, which I thought it would not.
I had not noticed any side effects or problems at that point. I was tired of potatoes, but it was manageable.
Maybe I should better emphasize on what points I feel SMTM is responsible and on what points I do not. I do not feel that SMTM is responsible for my safety and what could have happened, or the impact of the diet on me. In fact, I do not believe SMTM to be responsible for anything me-related: I made my own choices, and the fact that I did not stop the diet sooner was my own mistake I am owning up to.
What I do feel is that SMTM had a very loose methodology in how they conducted their studies, were more trying to confirm their hypothesis than really challenge it, and as a result the data is quite muddled and probably not that useful.
The safety part is related in the sense that:
It might be that many potato diet’s benefits are actually just coming from malnutrition. But if there were evidences of safety, that would not turn up to be a problem
It is relevant about what kind of preemptive research they made to decide whether the potato diet was worth it
For me at least, it goes with the general notion that it was very unclear what they were looking for
Part of me strongly agrees with you. At the same time… I don’t think “how much work would it have been to address [single bad thing]?” is the right unit of measurement. The actual question is “what would the cost/benefit analysis of address every single thing in this class of problems?”, and that’s a lot less clear cut.
Some things that make it difficult:
There are just a lot of potential problems.
There are an infinite number of things they didn’t research. Every time they add one to the explicit list of unresearched things they make the list as a whole harder to read (-> people are more likely to miss warnings relevant to them) while also conveying more safety and confidence. God forbid people decide anything not on the list of “things we didn’t check” was in fact checked.
You can say they should estimate the danger from each of these and only explicitly disavow research over a certain level of importance, but that’s an inherently noisy process people are likely to disagree on, and no matter what there has to be something just below the line. So again adding warnings can leave people with an inaccurately increased feeling of safety.
When I share medical info, I always preface it with “anything with a real effect can hurt you, think this through for yourself”. I think that’s a little better than what SMTM did. But I don’t see any easy additions to their statements of “If at any point you get sick or begin having side-effects, stop the diet immediately. We can still use your data up to that point, and we don’t want anything to happen to you” and “We are not doctors. We are 20 rats in a trenchcoat. eee! eee! eee!” that make things obviously better.
You note that you felt obliged to keep going in order to provide better data. Maybe there was pressure in private communication, but AFAICT they explicitly spoke against that in the blog post. I had a similar thing happen in my experiment- despite explicitly saying people could drop out at any time and would still be paid, one person was extremely reluctant to do so despite bad side effects and needed a strong push from me. In some sense I don’t think this is fair to put on experimenters, people shouldn’t even need to be told “stop if you feel bad”. But this seems like a pattern, and therefore should be taken into account even if it’s not objectively fair.
I mean, that is why I included “More research to shrink our unknown unknowns” as a general category. I do not think the research needs to be thorough, 3-4 very broad general area would suffice, but even if that does not fit into the points you mention, a statement along the lines you mentioned could work. In general, I do not think that more than one-two hours should be spent on writing this warning.
As for “If at any point you get sick or begin having side effects, stop the diet immediately”, that is indeed a good first step. My problem with it is that I am now understanding that you can develop side effects that indicate one should stop, that are not apparent unless you really track several variables intently. A general instruction to, in addition to doing one’s own research, decide what to look for in advance could have worked.
I want to note three things:
I did not feel pressured by SMTM either in their public or private communications at any point
That is not quite correct. I wanted to keep going for the data to be included as a data point, especially if the diet turned out not to work, which I thought it would not.
I had not noticed any side effects or problems at that point. I was tired of potatoes, but it was manageable.
Maybe I should better emphasize on what points I feel SMTM is responsible and on what points I do not. I do not feel that SMTM is responsible for my safety and what could have happened, or the impact of the diet on me. In fact, I do not believe SMTM to be responsible for anything me-related: I made my own choices, and the fact that I did not stop the diet sooner was my own mistake I am owning up to.
What I do feel is that SMTM had a very loose methodology in how they conducted their studies, were more trying to confirm their hypothesis than really challenge it, and as a result the data is quite muddled and probably not that useful.
The safety part is related in the sense that:
It might be that many potato diet’s benefits are actually just coming from malnutrition. But if there were evidences of safety, that would not turn up to be a problem
It is relevant about what kind of preemptive research they made to decide whether the potato diet was worth it
For me at least, it goes with the general notion that it was very unclear what they were looking for