What I am saying that being enormously powerful and useful does not determine the meaning of a word. Yes, something that optimizes can be enormously useful. That doesn’t make it intelligent, just like it doesn’t make it blue or green. And for the same reason: neither “intelligent” nor “blue” means “optimizing.” And your case of evolution proves that; evolution is not intelligent, even though it was enormously useful.
“This claim doesn’t follow from your premise at all.” Not as a logical deduction, but in the sense that if you pay attention to what I was talking about, you can see that it would be true. For example, precisely because they have general knowledge, human beings can pursue practically any goal, whenever something or someone happens to persuade them that “this is good.” AIs will have general knowledge, and therefore they will be open to pursuing almost any goal, in the same way and for the same reasons.
What I am saying that being enormously powerful and useful does not determine the meaning of a word. Yes, something that optimizes can be enormously useful. That doesn’t make it intelligent, just like it doesn’t make it blue or green. And for the same reason: neither “intelligent” nor “blue” means “optimizing.” And your case of evolution proves that; evolution is not intelligent, even though it was enormously useful.
“This claim doesn’t follow from your premise at all.” Not as a logical deduction, but in the sense that if you pay attention to what I was talking about, you can see that it would be true. For example, precisely because they have general knowledge, human beings can pursue practically any goal, whenever something or someone happens to persuade them that “this is good.” AIs will have general knowledge, and therefore they will be open to pursuing almost any goal, in the same way and for the same reasons.