Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.
Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.