My worry is, these heuristics are only “time-tested” in that they’ve (apparently) had good consequences relative to not-super-large-scale goals, in not-super-unfamiliar contexts. Many people working on the kinds of problems you’re pointing at do so for impartial altruistic reasons. So I suspect that that track record is only weak evidence that such heuristics will work relative to their goals, in unfamiliar contexts — like, say, ASI takeoff.
“Sharing scientific knowledge freely” contributed to the industrial revolution, for example, without which factory farming wouldn’t have happened, nor the development of technologies that pose x-risks. Granted, the implications of these examples are debatable. But they seem prima facie concerning enough to me that calling big-world heuristics “robust to uncertainty” sounds way too strong (for impartial altruistic goals at least).
depend on your specific causal argument being correct & your conceptualizations apt to reality. Often it’s a fairly conjunctive argument full of lots of beliefs about what people will do
I don’t think the extrapolation from “good on local scales / familiar contexts” to “good on the cosmic scale / very unfamiliar contexts” escapes really conjunctive causal mechanisms. It just hides them at a higher level of abstraction. When making that extrapolation, we’re predicting that the same sorts of mechanisms that led to good consequences on the local scales will be at play on the cosmic scale. So we’re baking in a lot of implicit beliefs about those mechanisms.
Thanks for clarifying!
My worry is, these heuristics are only “time-tested” in that they’ve (apparently) had good consequences relative to not-super-large-scale goals, in not-super-unfamiliar contexts. Many people working on the kinds of problems you’re pointing at do so for impartial altruistic reasons. So I suspect that that track record is only weak evidence that such heuristics will work relative to their goals, in unfamiliar contexts — like, say, ASI takeoff.
“Sharing scientific knowledge freely” contributed to the industrial revolution, for example, without which factory farming wouldn’t have happened, nor the development of technologies that pose x-risks. Granted, the implications of these examples are debatable. But they seem prima facie concerning enough to me that calling big-world heuristics “robust to uncertainty” sounds way too strong (for impartial altruistic goals at least).
I don’t think the extrapolation from “good on local scales / familiar contexts” to “good on the cosmic scale / very unfamiliar contexts” escapes really conjunctive causal mechanisms. It just hides them at a higher level of abstraction. When making that extrapolation, we’re predicting that the same sorts of mechanisms that led to good consequences on the local scales will be at play on the cosmic scale. So we’re baking in a lot of implicit beliefs about those mechanisms.