I assign a probability of less than 10^(-9) to [Eliezer] succeeding in playing a critical role on the Friendly AI project that [he’s] working on.
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we’d need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you’ve made in the above comment?
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
It doesn’t compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I’d rather you make the intended meaning explicit yourself.)
Eliezer is probably at about 1 in 10000 level of ability [of G]
Agreed, and I like to imagine that he reads that and thinks to himself “only 10000? thanks a lot!” :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is—eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we’d need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer’s ability to contribute to the solution. But it’s OK, informal conclusions differ; what’s not OK is that in the other comment you seem to contradict your claim.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we’d need even more, if that.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
There are also serious concerns about the end-game
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.
You wrote elsewhere in the thread:
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we’d need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you’ve made in the above comment?
It doesn’t compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I’d rather you make the intended meaning explicit yourself.)
Agreed, and I like to imagine that he reads that and thinks to himself “only 10000? thanks a lot!” :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is—eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
I’ll explain my thinking on these matters later.
Yes, strictly speaking we’d need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer’s ability to contribute to the solution. But it’s OK, informal conclusions differ; what’s not OK is that in the other comment you seem to contradict your claim.
Edit: I was not thinking clearly here.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Hey, three days have passed and I want that post!
I have an excuse, I got a cold!
Okay hurry up then, you’re wasting lives in our future light cone.
“Shut up and do the temporarily inconvenient!”
Three more days have passed.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.