Eliezer: “and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever).”
Why do the values freeze? Because there is no more competition? And if that’s the problem, why not try to plan a transition from pre-AI to an ecology of competing AIs that will not converge to a singleton? Or spell out the problem clearly enough that we can figure whether one can achieve a singleton that doesn’t have that property?
(Not that Eliezer hasn’t heard me say this before. I made a bit of a speech about AI ecology at the end of the first AGI conference a few years ago.)
Robin: “In a foom that took two years, if the AI was visible after one year, that might give the world a year to destroy it.”
Yes. The timespan of the foom is important largely because it changes what the AI is likely to do, because it changes the level of danger that the AI is in and the urgency of its actions.
Eliezer: “When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background.”
There are many sociological parallels between Eliezer’s “movement”, and early 20th-century communism.
Eliezer: “I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.”
I wonder if you’re thinking that I meant that. You can see that I didn’t in my first comment on Visions of Heritage. But I do think you’re going one level too few meta. And I think that CEV would make it very hard to escape the non-meta philosophies of the programmers. It would be worse at escaping them than the current, natural system of cultural evolution is.
Numerous people have responded to some of my posts by saying that CEV doesn’t restrict the development of values (or equivalently, that CEV doesn’t make AIs less free). Obviously it does. That’s the point of CEV. If you’re not trying to restrict how values develop, you might as well go home and watch TV and let the future spin out of control. One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.” Is it “meta-level value stasis”?
I think that evolution and competition have been pretty good at causing value development. (That’s me going one more level meta.) Having competition between different subpopulations with different values is a key part of this. Taking that away would be disastrous.
Not to mention the fact that value systems are local optima. If you’re doing search, it might make sense to average together some current good solutions and test the results out, in competition with the original solutions. It is definitely a bad idea to average together your current good solutions and replace them with the average.
Eliezer: “and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever).”
Why do the values freeze? Because there is no more competition? And if that’s the problem, why not try to plan a transition from pre-AI to an ecology of competing AIs that will not converge to a singleton? Or spell out the problem clearly enough that we can figure whether one can achieve a singleton that doesn’t have that property?
(Not that Eliezer hasn’t heard me say this before. I made a bit of a speech about AI ecology at the end of the first AGI conference a few years ago.)
Robin: “In a foom that took two years, if the AI was visible after one year, that might give the world a year to destroy it.”
Yes. The timespan of the foom is important largely because it changes what the AI is likely to do, because it changes the level of danger that the AI is in and the urgency of its actions.
Eliezer: “When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background.”
There are many sociological parallels between Eliezer’s “movement”, and early 20th-century communism.
Eliezer: “I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.”
I wonder if you’re thinking that I meant that. You can see that I didn’t in my first comment on Visions of Heritage. But I do think you’re going one level too few meta. And I think that CEV would make it very hard to escape the non-meta philosophies of the programmers. It would be worse at escaping them than the current, natural system of cultural evolution is.
Numerous people have responded to some of my posts by saying that CEV doesn’t restrict the development of values (or equivalently, that CEV doesn’t make AIs less free). Obviously it does. That’s the point of CEV. If you’re not trying to restrict how values develop, you might as well go home and watch TV and let the future spin out of control. One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.” Is it “meta-level value stasis”?
I think that evolution and competition have been pretty good at causing value development. (That’s me going one more level meta.) Having competition between different subpopulations with different values is a key part of this. Taking that away would be disastrous.
Not to mention the fact that value systems are local optima. If you’re doing search, it might make sense to average together some current good solutions and test the results out, in competition with the original solutions. It is definitely a bad idea to average together your current good solutions and replace them with the average.