This post suffers from lumping together orthogonal issues and conclusions from them. Let’s consider individually the following claims:
The world is in danger, and the feat of saving the world (if achieved) would be very important, more so than most other things we can currently do.
Creating FAI is possible.
Creating FAI, if possible, will be conductive to saving the world.
If FAI is possible, person X’s work contributes to developing FAI.
Person X’s work contributes to saving the world.
Most people’s work doesn’t contribute to saving the world.
Person X’s activity is more important than that of most other people.
Person X believes their activity is more important than that of most other people.
Person X suffers from delusions of grandeur.
A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn’t automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn’t:
The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity.
Thus, “doing something which very visibly and decisively alters the fate of humanity” is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can’t require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.
In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.
Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X’s work doesn’t in fact contribute to FAI (assuming it’s possible to contribute to FAI).
Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.
Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:
Success is not assured. I’m not sure what’s meant by confessing to being “ambitious”. Is it like being “optimistic”?
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don’t have much confidence is that each time I “level up”, last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don’t expect to solve the whole problem (and I don’t expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer’s work on communicating the problem statement of FAI and new SIAI’s work on spreading the word) whose contributions will be more significant.
Your analysis is very careful and I agree with almost everything that you say.
I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises—we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc.
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
I don’t believe that Eliezer can credibly hold the view that he’s the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
I assign a probability of less than 10^(-9) to [Eliezer] succeeding in playing a critical role on the Friendly AI project that [he’s] working on.
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we’d need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you’ve made in the above comment?
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
It doesn’t compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I’d rather you make the intended meaning explicit yourself.)
Eliezer is probably at about 1 in 10000 level of ability [of G]
Agreed, and I like to imagine that he reads that and thinks to himself “only 10000? thanks a lot!” :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is—eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we’d need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer’s ability to contribute to the solution. But it’s OK, informal conclusions differ; what’s not OK is that in the other comment you seem to contradict your claim.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we’d need even more, if that.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
There are also serious concerns about the end-game
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.
Most people’s work doesn’t contribute to saving the world.
I’d argue that a lot of people’s work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.
Obviously, you need to interpret that statement as “Any given person’s work doesn’t significantly contribute to saving the world”. In other words, if we “subtract” that one person, the future (in the aspect of the world not ending) changes insignificantly.
Are you also amending 4) to have the significant clause?
Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.
I’m not amending, I’m clarifying. (4) doesn’t have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn’t apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people).
As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).
Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
(4 here being “If FAI is possible, person X’s work contributes to developing FAI.”) This seems be a weak part of your argument. A successful FAI attempt will obviously have to use lots of philosophical and technical results that were not developed specifically with FAI in mind. Many people may be contributing to FAI, without consciously intending to do so. For example when I first started thinking about anthropic reasoning I was mainly thinking about human minds being copyable in the future and trying to solve philosophical puzzles related to that.
Another possibility is that the most likely routes to FAI go through intelligence enhancement or uploading, so people working in those fields are actually making more contributions to FAI than people like you and Eliezer.
Generally speaking, your argument isn’t very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it. (EDIT: To clarify slightly after your response, I mean to point out that you didn’t directly mention these particular assumptions, and that I think many people take issue with them.)
My personal, rather uninformed belief is that FAI would be a source of enormous good, but it’s not necessary for humanity to continue to grow and to overcome x-risk (so 3 is weaker); X may be contributing to the development of FAI, but not that much (so 4 is weaker); and other people engaged in productive pursuits are also contributing a non-zero amount to “save the world” (so 6 is weaker.)
As such, I have a hard time concluding that X’s activity is anywhere near the “most important” using your reasoning, although it may be quite important.
Generally speaking, your argument isn’t very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it.
The argument I gave doesn’t include justification of things it assumes (that you referred to). It only serves to separate the issues with claims about a person from issues with claims about what’s possible in the world. Both kinds of claims (assumptions in the argument I gave) could be argued with, but necessarily separately.
OK, I now see what your post was aimed at, a la this other post you made. I agree that criticism ought to be toward person X’s beliefs about the world, not his conclusions about himself.
This post suffers from lumping together orthogonal issues and conclusions from them. Let’s consider individually the following claims:
The world is in danger, and the feat of saving the world (if achieved) would be very important, more so than most other things we can currently do.
Creating FAI is possible.
Creating FAI, if possible, will be conductive to saving the world.
If FAI is possible, person X’s work contributes to developing FAI.
Person X’s work contributes to saving the world.
Most people’s work doesn’t contribute to saving the world.
Person X’s activity is more important than that of most other people.
Person X believes their activity is more important than that of most other people.
Person X suffers from delusions of grandeur.
A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn’t automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn’t:
Thus, “doing something which very visibly and decisively alters the fate of humanity” is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can’t require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.
In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.
Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X’s work doesn’t in fact contribute to FAI (assuming it’s possible to contribute to FAI).
Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.
Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don’t have much confidence is that each time I “level up”, last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don’t expect to solve the whole problem (and I don’t expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer’s work on communicating the problem statement of FAI and new SIAI’s work on spreading the word) whose contributions will be more significant.
Your analysis is very careful and I agree with almost everything that you say.
I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises—we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc.
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
I don’t believe that Eliezer can credibly hold the view that he’s the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
You wrote elsewhere in the thread:
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we’d need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you’ve made in the above comment?
It doesn’t compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I’d rather you make the intended meaning explicit yourself.)
Agreed, and I like to imagine that he reads that and thinks to himself “only 10000? thanks a lot!” :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is—eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
I’ll explain my thinking on these matters later.
Yes, strictly speaking we’d need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer’s ability to contribute to the solution. But it’s OK, informal conclusions differ; what’s not OK is that in the other comment you seem to contradict your claim.
Edit: I was not thinking clearly here.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it’s fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister’s remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of “collaborating with existing institutions” is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say “Secrecy in science? Nonsense!”, it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it’s not what we ought to do.
I’m currently ambivalent on this point, but it seems to me that at least preference theory (I’ll likely have a post on that on my blog tomorrow) doesn’t directly increase the danger, as it’s about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won’t be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I’m not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a “weapon of math destruction”.
Hey, three days have passed and I want that post!
I have an excuse, I got a cold!
Okay hurry up then, you’re wasting lives in our future light cone.
“Shut up and do the temporarily inconvenient!”
Three more days have passed.
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
I agree with this and believe that this could justify secrecy, but I think that it’s very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I’m pursuing my current trajectory of postings.
I’d argue that a lot of people’s work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.
Obviously, you need to interpret that statement as “Any given person’s work doesn’t significantly contribute to saving the world”. In other words, if we “subtract” that one person, the future (in the aspect of the world not ending) changes insignificantly.
Are you also amending 4) to have the significant clause?
Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.
I’m not amending, I’m clarifying. (4) doesn’t have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn’t apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people).
As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).
That makes me wonder who will replace Norman Borlaug, or lets say any particular influential writer or thinker.
Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.
(4 here being “If FAI is possible, person X’s work contributes to developing FAI.”) This seems be a weak part of your argument. A successful FAI attempt will obviously have to use lots of philosophical and technical results that were not developed specifically with FAI in mind. Many people may be contributing to FAI, without consciously intending to do so. For example when I first started thinking about anthropic reasoning I was mainly thinking about human minds being copyable in the future and trying to solve philosophical puzzles related to that.
Another possibility is that the most likely routes to FAI go through intelligence enhancement or uploading, so people working in those fields are actually making more contributions to FAI than people like you and Eliezer.
Person X believes that their activity is more important than all other people, and that no other people can do it.
Person X also believes that only this project is likely to save the world.
Person X also believes that FAI will save the world on all axes, including political and biological.
--JRM
Generally speaking, your argument isn’t very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it. (EDIT: To clarify slightly after your response, I mean to point out that you didn’t directly mention these particular assumptions, and that I think many people take issue with them.)
My personal, rather uninformed belief is that FAI would be a source of enormous good, but it’s not necessary for humanity to continue to grow and to overcome x-risk (so 3 is weaker); X may be contributing to the development of FAI, but not that much (so 4 is weaker); and other people engaged in productive pursuits are also contributing a non-zero amount to “save the world” (so 6 is weaker.)
As such, I have a hard time concluding that X’s activity is anywhere near the “most important” using your reasoning, although it may be quite important.
The argument I gave doesn’t include justification of things it assumes (that you referred to). It only serves to separate the issues with claims about a person from issues with claims about what’s possible in the world. Both kinds of claims (assumptions in the argument I gave) could be argued with, but necessarily separately.
OK, I now see what your post was aimed at, a la this other post you made. I agree that criticism ought to be toward person X’s beliefs about the world, not his conclusions about himself.