No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we’d need even more, if that.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
There are also serious concerns about the end-game
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Hey, three days have passed and I want that post!
I have an excuse, I got a cold!
Okay hurry up then, you’re wasting lives in our future light cone.
“Shut up and do the temporarily inconvenient!”
Three more days have passed.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.