when you are working on a hard technical problem that’s a long way from solved, and could have both benefits and harms to humanity if it were solved, you don’t really have to worry about “what if we succeed and it’s bad?” or “what if this information gets into the hands of the wrong people?” You’re a long way from winning, worrying about winning too much is silly, and heuristics like “do a good job on important work” and “share scientific knowledge freely” are more trustworthy than galaxy-brained arguments for why in this particular case the utilitarian calculus works out the other way.
I’m not sure why exactly you believe this, can you say more? (Calling the arguments for backfire “galaxy-brained” kinda unduly stacks the deck, I think.)
Heuristics like “doing a good job on important work benefits the world” and “it’s beneficial to share scientific knowledge freely” are time-tested (at least over the past several centuries) and have a lot of empirical evidence in their favor. “The Baconian project has been working” is one of the only sweeping historical/social claims I am highly confident in. And you don’t even need that to believe a more common-sense thing like “working to solve problems that people find helpful to have solved is beneficial on net.” Like, “productivity is good” is something that could have been intuitive in antiquity or prehistory. There’s just tons of individual cases, life experience, and logical extrapolation that all point in the same direction.
“THIS time if I invent/improve this technology it’ll be harmful” or “THIS time if I disseminate this discovery it’ll be harmful” fly in the face of those heuristics and depend on your specific causal argument being correct & your conceptualizations apt to reality. Often it’s a fairly conjunctive argument full of lots of beliefs about what people will do, made without specific knowledge of the kinds of people in question. Doesn’t mean it’s wrong, but it’s risky.
My worry is, these heuristics are only “time-tested” in that they’ve (apparently) had good consequences relative to not-super-large-scale goals, in not-super-unfamiliar contexts. Many people working on the kinds of problems you’re pointing at do so for impartial altruistic reasons. So I suspect that that track record is only weak evidence that such heuristics will work relative to their goals, in unfamiliar contexts — like, say, ASI takeoff.
“Sharing scientific knowledge freely” contributed to the industrial revolution, for example, without which factory farming wouldn’t have happened, nor the development of technologies that pose x-risks. Granted, the implications of these examples are debatable. But they seem prima facie concerning enough to me that calling big-world heuristics “robust to uncertainty” sounds way too strong (for impartial altruistic goals at least).
depend on your specific causal argument being correct & your conceptualizations apt to reality. Often it’s a fairly conjunctive argument full of lots of beliefs about what people will do
I don’t think the extrapolation from “good on local scales / familiar contexts” to “good on the cosmic scale / very unfamiliar contexts” escapes really conjunctive causal mechanisms. It just hides them at a higher level of abstraction. When making that extrapolation, we’re predicting that the same sorts of mechanisms that led to good consequences on the local scales will be at play on the cosmic scale. So we’re baking in a lot of implicit beliefs about those mechanisms.
I’m not sure why exactly you believe this, can you say more? (Calling the arguments for backfire “galaxy-brained” kinda unduly stacks the deck, I think.)
Heuristics like “doing a good job on important work benefits the world” and “it’s beneficial to share scientific knowledge freely” are time-tested (at least over the past several centuries) and have a lot of empirical evidence in their favor. “The Baconian project has been working” is one of the only sweeping historical/social claims I am highly confident in. And you don’t even need that to believe a more common-sense thing like “working to solve problems that people find helpful to have solved is beneficial on net.” Like, “productivity is good” is something that could have been intuitive in antiquity or prehistory. There’s just tons of individual cases, life experience, and logical extrapolation that all point in the same direction.
“THIS time if I invent/improve this technology it’ll be harmful” or “THIS time if I disseminate this discovery it’ll be harmful” fly in the face of those heuristics and depend on your specific causal argument being correct & your conceptualizations apt to reality. Often it’s a fairly conjunctive argument full of lots of beliefs about what people will do, made without specific knowledge of the kinds of people in question. Doesn’t mean it’s wrong, but it’s risky.
Thanks for clarifying!
My worry is, these heuristics are only “time-tested” in that they’ve (apparently) had good consequences relative to not-super-large-scale goals, in not-super-unfamiliar contexts. Many people working on the kinds of problems you’re pointing at do so for impartial altruistic reasons. So I suspect that that track record is only weak evidence that such heuristics will work relative to their goals, in unfamiliar contexts — like, say, ASI takeoff.
“Sharing scientific knowledge freely” contributed to the industrial revolution, for example, without which factory farming wouldn’t have happened, nor the development of technologies that pose x-risks. Granted, the implications of these examples are debatable. But they seem prima facie concerning enough to me that calling big-world heuristics “robust to uncertainty” sounds way too strong (for impartial altruistic goals at least).
I don’t think the extrapolation from “good on local scales / familiar contexts” to “good on the cosmic scale / very unfamiliar contexts” escapes really conjunctive causal mechanisms. It just hides them at a higher level of abstraction. When making that extrapolation, we’re predicting that the same sorts of mechanisms that led to good consequences on the local scales will be at play on the cosmic scale. So we’re baking in a lot of implicit beliefs about those mechanisms.