I guess I misunderstood you. I figured that without “regression coefficients,” the sentence would be a bit tautological: “the point of randomized controlled trial is to avoid [a] non-randomized sample,” and there were other bits that made me think you had an issue with both selection bias (agree) and regressions (disagree).
I share your overall takeaway, but at this point I am just genuinely curious why the self-selection is presumed to be such a threat to internal validity here. I think we need more attention to selection effects on the margin, but I also think there is a general tendency for people to believe that once they’ve identified a selection issue the results are totally undermined. What is the alternative explanation for why semaglutide would disincline people who would have had small change scores from participating or incline people who have large change scores to participate (remember, this is within-subjects) in the alcohol self-administration experiment? Maybe those who had the most reduced cravings wanted to see more of what these researchers could do? But that process would also occur among placebo, so it’d work via the share of people with large change scores being greater in the semaglutide group, which is...efficacy. There’s nuance there, but hard to square with lack of efficacy.
That said, still agree that the results are no slam dunk. Very specific population, very specific outcomes affected, and probably practically small effects too.
> What is the alternative explanation for why semaglutide would disincline people who would have had small change scores from participating or incline people who have large change scores to participate (remember, this is within-subjects) in the alcohol self-administration experiment?
I’m a bit unsure what the non-alternative explanation is here. But imagine that semaglutide does not reduce the urge to drink but—I don’t know—makes people more patient, or makes them more likely to agree to do things doctors ask them to do, or makes them more greedy. Then take the “marginal” person, who is just on the border of participating or not. If those marginal people drink less on average, then semaglutide would look good purely due to changing selection rather than actually reducing drinking.
Now, I don’t claim that the above story is true. It’s possible, but lots of other stories are also possible, including ones where the bias could go in the other way.
I also think there is a general tendency for people to believe that once they’ve identified a selection issue the results are totally undermined.
I expected this sentence to be followed by you praising me for explicitly disavowing such a view and stating that, since the bias could be in either direction, the lab experiment does provide some evidence in favor of semaglutide. :) (Just very weak evidence.)
I guess I misunderstood you. I figured that without “regression coefficients,” the sentence would be a bit tautological: “the point of randomized controlled trial is to avoid [a] non-randomized sample,” and there were other bits that made me think you had an issue with both selection bias (agree) and regressions (disagree).
I share your overall takeaway, but at this point I am just genuinely curious why the self-selection is presumed to be such a threat to internal validity here. I think we need more attention to selection effects on the margin, but I also think there is a general tendency for people to believe that once they’ve identified a selection issue the results are totally undermined. What is the alternative explanation for why semaglutide would disincline people who would have had small change scores from participating or incline people who have large change scores to participate (remember, this is within-subjects) in the alcohol self-administration experiment? Maybe those who had the most reduced cravings wanted to see more of what these researchers could do? But that process would also occur among placebo, so it’d work via the share of people with large change scores being greater in the semaglutide group, which is...efficacy. There’s nuance there, but hard to square with lack of efficacy.
That said, still agree that the results are no slam dunk. Very specific population, very specific outcomes affected, and probably practically small effects too.
(Sorry for the slow reply—just saw this.)
> What is the alternative explanation for why semaglutide would disincline people who would have had small change scores from participating or incline people who have large change scores to participate (remember, this is within-subjects) in the alcohol self-administration experiment?
I’m a bit unsure what the non-alternative explanation is here. But imagine that semaglutide does not reduce the urge to drink but—I don’t know—makes people more patient, or makes them more likely to agree to do things doctors ask them to do, or makes them more greedy. Then take the “marginal” person, who is just on the border of participating or not. If those marginal people drink less on average, then semaglutide would look good purely due to changing selection rather than actually reducing drinking.
Now, I don’t claim that the above story is true. It’s possible, but lots of other stories are also possible, including ones where the bias could go in the other way.
I expected this sentence to be followed by you praising me for explicitly disavowing such a view and stating that, since the bias could be in either direction, the lab experiment does provide some evidence in favor of semaglutide. :) (Just very weak evidence.)