If you want probabilities for these things to be backed up by mathematics, you’re going to be disappointed, because there aren’t any. The best probabilities—or rather, the only probabilities we have here, were produced using human intuition. You can break down the possibilities into small pieces, generate probabilities for the pieces, and get an overall probability that way, but at the base of the calculations you just have order-of-magnitude estimates. You can’t provide formal, strongly defensible probabilities for the sub-events, because there just isn’t any data—and there won’t be any data until after the danger of AGI has destroyed us, or passed without destroying us. And that, I think, is the reason why SIAI doesn’t provide any numbers: since they’d only be order of magnitude estimates, they’d give the people who had already made up their minds something to attack.
I’m not asking for defensible probabilities that would withstand academic peer review. I’m asking for decision procedures including formulas with variables that allow you to provide your own intuitive values to eventually calculate your own probabilities. I want the SIAI to provide a framework that gives a concise summary of the risks in question and a comparison with other existential risks. I want people to be able to carry out results analysis and distinguish risks posed by artificial general intelligence from other risks like global warming or grey goo.
There aren’t any numbers for a lot of other existential risks either. But one is still able to differentiate between those risks and that from unfriendly AI based on logical consequences of other established premises like the Church–Turing–Deutsch principle. Should we be equally concerned with occultists trying to summon world-changing supernatural powers?
Unfortunately, this is a common conversational pattern.
Q. You have given your estimate of the probability of FAI/cryonics/nanobots/FTL/antigravity. In support of this number, you have here listed probabilities for supporting components, with no working shown. These appear to include numbers not only for technologies we have no empirical knowledge of, but particular new scientific insights that have yet to occur. It looks very like you have pulled the numbers out of thin air. How did you derive these numbers?
A. Bayesian probability calculations.
Q. Could you please show me your working? At least a reasonable chunk of the Bayesian network you derived this from? C’mon, give me something to work with here.
A.(tumbleweeds)
Q. I remain somehow unconvinced.
If you pull a number out of thin air and run it through a formula, the result is still a number pulled out of thin air.
If you want people to believe something, you have to bother convincing them.
If you want probabilities for these things to be backed up by mathematics, you’re going to be disappointed, because there aren’t any. The best probabilities—or rather, the only probabilities we have here, were produced using human intuition. You can break down the possibilities into small pieces, generate probabilities for the pieces, and get an overall probability that way, but at the base of the calculations you just have order-of-magnitude estimates. You can’t provide formal, strongly defensible probabilities for the sub-events, because there just isn’t any data—and there won’t be any data until after the danger of AGI has destroyed us, or passed without destroying us. And that, I think, is the reason why SIAI doesn’t provide any numbers: since they’d only be order of magnitude estimates, they’d give the people who had already made up their minds something to attack.
I’m not asking for defensible probabilities that would withstand academic peer review. I’m asking for decision procedures including formulas with variables that allow you to provide your own intuitive values to eventually calculate your own probabilities. I want the SIAI to provide a framework that gives a concise summary of the risks in question and a comparison with other existential risks. I want people to be able to carry out results analysis and distinguish risks posed by artificial general intelligence from other risks like global warming or grey goo.
There aren’t any numbers for a lot of other existential risks either. But one is still able to differentiate between those risks and that from unfriendly AI based on logical consequences of other established premises like the Church–Turing–Deutsch principle. Should we be equally concerned with occultists trying to summon world-changing supernatural powers?
+1
Unfortunately, this is a common conversational pattern.
Q. You have given your estimate of the probability of FAI/cryonics/nanobots/FTL/antigravity. In support of this number, you have here listed probabilities for supporting components, with no working shown. These appear to include numbers not only for technologies we have no empirical knowledge of, but particular new scientific insights that have yet to occur. It looks very like you have pulled the numbers out of thin air. How did you derive these numbers?
A. Bayesian probability calculations.
Q. Could you please show me your working? At least a reasonable chunk of the Bayesian network you derived this from? C’mon, give me something to work with here.
A. (tumbleweeds)
Q. I remain somehow unconvinced.
If you pull a number out of thin air and run it through a formula, the result is still a number pulled out of thin air.
If you want people to believe something, you have to bother convincing them.