Getting a large and statistically unbiased control group would seem to be practically impossible in many cases, yet having one would be utterly essential.
An old scam: “Send us $50, and we will use our connections to get your child admitted at the college of their choice! Guaranteed, with a full refund given in the event of a rejection!” Then the scammers do nothing, refund the portion of the money sent in by applicants who were later rejected, and keep the portion of the money sent in by applicants who were later accepted.
People are usually wary enough to avoid distributing their money this way, but for votes and sociological beliefs it seems to be a more tried-and-true approach. “Having control of the most prominent political office when a business cycle goes boom” and “having control when a business cycle goes bust” seem to be reliable ways for political parties to gain and lose mindshare even when it is very unlikely that such a rapid and strong connection exists between that office and that economic status. Similarly, every time crime rates go significantly up or down, everyone is eager to blame contemporaneous policy X or credit contemporaneous policy Y, usually despite prominent examples of neighboring polities which didn’t implement X/Y but saw the same type of crime rate change at the same time.
I agree that having a good control group would be very important. I also agree that for many charitable activities this would be impossible. However, in many other activities, it would be possible. A good number of randomized controlled trials are being run to measure the effectiveness of health, educational and even political/governance activities, so they can be done. They might be costly, but that’s a separate question, and, as I said, there could be very large benefits in improved knowledge from doing them.
I think your scam analogy is somewhat off-target. A closer analogy would be that, for donations at the moment, they ask you to send $50 and don’t offer any refund if your child is rejected. You just have to trust them, or examine them closely enough, to be confident they are doing something effective. A payment for results arrangement with a badly constructed control group might be like your analogy, but even this might be some sort of improvement, because there is now at least an incentive to reduce the number of rejections, even if some of the money is going for things that would have happened anyway. And, with a good control group, you would only be paying for things you could be confident were the results of the charity’s activity.
An old scam: “Send us $50, and we will use our connections to get your child admitted at the college of their choice! Guaranteed, with a full refund given in the event of a rejection!” Then the scammers do nothing, refund the portion of the money sent in by applicants who were later rejected, and keep the portion of the money sent in by applicants who were later accepted.
I saw that used as a joke on Facebook (though the actual example was “I will use sorcery to predict your expected baby’s sex for $1000, and I will refund you if I’m wrong”), but I didn’t think anyone had used that in real life. (But I really oughtn’t be surprised this much.)
Getting a large and statistically unbiased control group would seem to be practically impossible in many cases, yet having one would be utterly essential.
An old scam: “Send us $50, and we will use our connections to get your child admitted at the college of their choice! Guaranteed, with a full refund given in the event of a rejection!” Then the scammers do nothing, refund the portion of the money sent in by applicants who were later rejected, and keep the portion of the money sent in by applicants who were later accepted.
People are usually wary enough to avoid distributing their money this way, but for votes and sociological beliefs it seems to be a more tried-and-true approach. “Having control of the most prominent political office when a business cycle goes boom” and “having control when a business cycle goes bust” seem to be reliable ways for political parties to gain and lose mindshare even when it is very unlikely that such a rapid and strong connection exists between that office and that economic status. Similarly, every time crime rates go significantly up or down, everyone is eager to blame contemporaneous policy X or credit contemporaneous policy Y, usually despite prominent examples of neighboring polities which didn’t implement X/Y but saw the same type of crime rate change at the same time.
I agree that having a good control group would be very important. I also agree that for many charitable activities this would be impossible. However, in many other activities, it would be possible. A good number of randomized controlled trials are being run to measure the effectiveness of health, educational and even political/governance activities, so they can be done. They might be costly, but that’s a separate question, and, as I said, there could be very large benefits in improved knowledge from doing them.
I think your scam analogy is somewhat off-target. A closer analogy would be that, for donations at the moment, they ask you to send $50 and don’t offer any refund if your child is rejected. You just have to trust them, or examine them closely enough, to be confident they are doing something effective. A payment for results arrangement with a badly constructed control group might be like your analogy, but even this might be some sort of improvement, because there is now at least an incentive to reduce the number of rejections, even if some of the money is going for things that would have happened anyway. And, with a good control group, you would only be paying for things you could be confident were the results of the charity’s activity.
I saw that used as a joke on Facebook (though the actual example was “I will use sorcery to predict your expected baby’s sex for $1000, and I will refund you if I’m wrong”), but I didn’t think anyone had used that in real life. (But I really oughtn’t be surprised this much.)