You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
I wonder why it took so long until you came along posting this comment.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
So what’s the alternative?
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
This isn’t quite what you’re talking about, but a relatively accessible intro doc:
http://singinst.org/riskintro/index.html
This seems like a summary of the idea of there being significant risk:
Anna Salamon at Singularity Summit 2009 - “Shaping the Intelligence Explosion”
http://www.vimeo.com/7318055
Good comment.
However,
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.