Omohundro has a paper on instrumental goals that many/most intelligences would converge on. For instance, they would strive to model themselves, to expand their capabilities, to represent their goals in terms of utility functions, to protect their utility functions against change, etc. None of these are universally true because we can just posit a pathological intelligence whose terminal goal is to not do these things. (And to some extent e.g. humans do in fact behave pathologically like that.)
We can say very little “optimizers over possible futures” in full generality, because that concept can be very broad if you define “optimizer” sufficiently broadly. Is a thermostat an intelligence, with the goal of achieving some temperature? Or consider a rock—we can see it as a mind with the goal of continuing its existence, and thus decides to be hard.
It seems that the paper discusses the inside view of intelligence, not the ways to detect one by its non-human-like artifacts.
I agree that it is hard to tell an intelligence without pattern-matching it to humans, that’s why I asked the question in the first place. But there hopefully should be at least some way to be convinced that a rock is not very intelligent, even if you can’t put yourself in its crystalline shoes.
Omohundro has a paper on instrumental goals that many/most intelligences would converge on. For instance, they would strive to model themselves, to expand their capabilities, to represent their goals in terms of utility functions, to protect their utility functions against change, etc. None of these are universally true because we can just posit a pathological intelligence whose terminal goal is to not do these things. (And to some extent e.g. humans do in fact behave pathologically like that.)
We can say very little “optimizers over possible futures” in full generality, because that concept can be very broad if you define “optimizer” sufficiently broadly. Is a thermostat an intelligence, with the goal of achieving some temperature? Or consider a rock—we can see it as a mind with the goal of continuing its existence, and thus decides to be hard.
It seems that the paper discusses the inside view of intelligence, not the ways to detect one by its non-human-like artifacts.
I agree that it is hard to tell an intelligence without pattern-matching it to humans, that’s why I asked the question in the first place. But there hopefully should be at least some way to be convinced that a rock is not very intelligent, even if you can’t put yourself in its crystalline shoes.