For those complaining about references to terms not defined within the Overcoming Bias sequence, see:
Coherent Extrapolated Volition (what does a “Friendly” AI do?) KnowabilityOfFAI (why it looks theoretically possible to specify the goal system of a self-modifying AI; I plan to post from this old draft document into Overcoming Bias and thereby finish it, so you needn’t read the old version right now, unless demand immediate answers).
@Vladimir Nesov: Good reply, I read it and wondered “Who’s channeling me?” before I got to the byline.
@Shane Legg: After studying FAI for a few years so that you actually have some idea of what the challenges are, and how many automatic failures are built into the problem, and seeing people say “We’ll just go full steam ahead and try out best”, and knowing that these people are not almost at the goal but ten lightyears short of it; then you learn to blank rivals out of your mind, and concentrate on the wall. That’s the only way you can see the wall, at all.
@Ben Goertzel: Who is there that says “I am working on Artificial General Intelligence?” and is doing theoretical research? AFAICT there’s plenty of theoretical research on AI, but it’s by people who no longer see themselves as coding an AGI at the end of it—it just means you’re working on narrow AI now.
@Yvain: To first order and generalizing from one data point, figure that Eliezer_2000 is demonstrably as smart and as knowledgeable as you can possibly get while still being stupid enough to try and charge full steam ahead into Unfriendly AI. Figure that Eliezer_2002 is as high as it gets before you spontaneously stop trying to build low-precision Friendly AI. Both of these are smart enough to be dangerous and not smart enough to be helpful, but they were highly unstable in terms of how long they stayed that way; Eliezer_2002 had less than four months left on his clock when he finished “Levels of Organization in General Intelligence”. I would not be intimidated by either of them into giving up, even though they’re taking holding themselves to much lower standards. They will charge ahead taking the quick and easy and vague and imprecise and wasteful and excruciatingly frustrating path. That’s going to burn up a lot of their time.
Those of AGI who stay in suicidal states, for years, even when I push on them externally, I find even less intimidating than the prospect of going up against an Eliezer_2002 who permanently stayed bounded at the highest suicidal level.
An AGI wannabe could theoretically have a different intellectual makeup that allows them to get farther and be more dangerous than Eliezer_2002, without passing the Schwarzschild bound and collapsing into an FAI programmer; but I see no evidence that this has ever actually happened.
To put it briefly: There really is an upper bound on how smart you can be, and still be that stupid.
So the state of the gameboard is not good, but the day is not already lost. You draw a line with all the sloppy suicides on one side, and those who slow down for precise standards on the other, and you hope that no one sufficiently intelligent + knowledgeable can stay on the wrong side of the line for long.
That’s the last thread on which our doom now hangs.
@Yvain: To first order and generalizing from one data point, figure that Eliezer_2000 is demonstrably as smart and as knowledgeable as you can possibly get while still being stupid enough to try and charge full steam ahead into Unfriendly AI. Figure that Eliezer_2002 is as high as it gets before you spontaneously stop trying to build low-precision Friendly AI. Both of these are smart enough to be dangerous and not smart enough to be helpful…
To put it briefly: There really is an upper bound on how smart you can be, and still be that stupid.
I think this line of argument should provide less comfort that it seems to.
Firstly, intelligent people can meaningfully have different values. Not all intelligences value the same things and not all human intelligences value the same things. Some people might be willing to take more risk with other people’s lives than you. Example: Oil company executives. There is strong reason to believe they are very intelligent and effective; they seem to achieve their goals in the world with a higher frequency than most other groups. Yet they also seem more likely to take actions with high risks to third parties.
Second, an intelligent moral individual could be bound up in an institution which exerts pressure on them to act in a way that satisfies the institutions values rather than their own. It is commonly said (although I don’t have a source, so grain of salt needed) that some members of the Manhattan project were not Certain that the reaction would not just continue indefinitely. It seems plausible that some of those physicists might have been over what has been described as the “upper bound on how smart you can be, and still be that stupid.”
For those complaining about references to terms not defined within the Overcoming Bias sequence, see:
Coherent Extrapolated Volition (what does a “Friendly” AI do?) KnowabilityOfFAI (why it looks theoretically possible to specify the goal system of a self-modifying AI; I plan to post from this old draft document into Overcoming Bias and thereby finish it, so you needn’t read the old version right now, unless demand immediate answers).
@Vladimir Nesov: Good reply, I read it and wondered “Who’s channeling me?” before I got to the byline.
@Shane Legg: After studying FAI for a few years so that you actually have some idea of what the challenges are, and how many automatic failures are built into the problem, and seeing people say “We’ll just go full steam ahead and try out best”, and knowing that these people are not almost at the goal but ten lightyears short of it; then you learn to blank rivals out of your mind, and concentrate on the wall. That’s the only way you can see the wall, at all.
@Ben Goertzel: Who is there that says “I am working on Artificial General Intelligence?” and is doing theoretical research? AFAICT there’s plenty of theoretical research on AI, but it’s by people who no longer see themselves as coding an AGI at the end of it—it just means you’re working on narrow AI now.
@Yvain: To first order and generalizing from one data point, figure that Eliezer_2000 is demonstrably as smart and as knowledgeable as you can possibly get while still being stupid enough to try and charge full steam ahead into Unfriendly AI. Figure that Eliezer_2002 is as high as it gets before you spontaneously stop trying to build low-precision Friendly AI. Both of these are smart enough to be dangerous and not smart enough to be helpful, but they were highly unstable in terms of how long they stayed that way; Eliezer_2002 had less than four months left on his clock when he finished “Levels of Organization in General Intelligence”. I would not be intimidated by either of them into giving up, even though they’re taking holding themselves to much lower standards. They will charge ahead taking the quick and easy and vague and imprecise and wasteful and excruciatingly frustrating path. That’s going to burn up a lot of their time.
Those of AGI who stay in suicidal states, for years, even when I push on them externally, I find even less intimidating than the prospect of going up against an Eliezer_2002 who permanently stayed bounded at the highest suicidal level.
An AGI wannabe could theoretically have a different intellectual makeup that allows them to get farther and be more dangerous than Eliezer_2002, without passing the Schwarzschild bound and collapsing into an FAI programmer; but I see no evidence that this has ever actually happened.
To put it briefly: There really is an upper bound on how smart you can be, and still be that stupid.
So the state of the gameboard is not good, but the day is not already lost. You draw a line with all the sloppy suicides on one side, and those who slow down for precise standards on the other, and you hope that no one sufficiently intelligent + knowledgeable can stay on the wrong side of the line for long.
That’s the last thread on which our doom now hangs.
I think this line of argument should provide less comfort that it seems to. Firstly, intelligent people can meaningfully have different values. Not all intelligences value the same things and not all human intelligences value the same things. Some people might be willing to take more risk with other people’s lives than you. Example: Oil company executives. There is strong reason to believe they are very intelligent and effective; they seem to achieve their goals in the world with a higher frequency than most other groups. Yet they also seem more likely to take actions with high risks to third parties.
Second, an intelligent moral individual could be bound up in an institution which exerts pressure on them to act in a way that satisfies the institutions values rather than their own. It is commonly said (although I don’t have a source, so grain of salt needed) that some members of the Manhattan project were not Certain that the reaction would not just continue indefinitely. It seems plausible that some of those physicists might have been over what has been described as the “upper bound on how smart you can be, and still be that stupid.”