The last few months I have been stumbling onto RCTs conducted in developing countries, with great results in treating Covid-19. The evaluated treatments seem apriori unlikely to generate these great results, e.g. this gargling study from Bangladesh, or this honey+black cumin seeds study from Pakistan, or Bromhexine from Iran all have results that would revolutionary-if-true. FWIW, and on the face of it, none of them can be described as garbage (decent sample size, randomized, p- preregistered for the Pakistani study, etc.)
This question however is not about these particular studies, but which base rate I should be using to evaluate such studies. I am coming up empty with prior instances of clinical trials from developing countries with promising results, good sample sizes but which later turned out to be false. Does anybody know of examples or can recommend search terms?
The rapid availability of third world originated studies to internet searches by those living elsewhere, which has happened because of internet publishing and especially in the last three years because of openly searchable preprint sites such as medrxiv, is a new enough thing in science that I doubt there has been enough time to get a valid such likely future failure rate, especially if you are including preprints.
To get an approximate base rate you would need to check the years prior to open internet medical paper publishing. At minimum you would need to read the older years of paper versions of those nation’s journals, likely in their medical schools, and count the studies fitting your criteria over the past ten years or more, ideally even including the studies rejected by such journals if you are including preprints in your questions about the Covid-19 studies. Not an easy project, so best to take the studies you mention as proposing a hypothesis, perhaps to be tested in the future via reproducibility. Just like most other clinical studies published by anyone else anywhere.
Thanks for the answer, and I understand where you are coming from. But concluding that “we need to do more studies” is not useful in this case, when most of the future damage of the pandemic will happen in the next few months, and publishing new studies takes close to a year.