Saying that there is an agent refers (in my view; definition for this thread) to a situation where future events are in some sense expected to be optimized according to some goals, to the extent certain other events (“actions”) control those future events. There might be many sufficient conditions for that in terms of particular AI designs, but they should amount to this expectation.
So an agent is already associated with goals in terms of its actual effect on its environment. Given that agent’s own future state (design) is an easily controlled part of the environment, it’s one of the things that’ll be optimized, and given that agents are particularly powerful incantations, it’s a good bet that future will retain agent-y patterns, at least for a start. If future agent has goals different from the original, this by the same definition says that the future will be optimized for different goals, and yet in a way controllable by original agent’s actions (through the future agent). This contradicts that the original agent is an agent (with original goals). And since the task of constructing future agent includes specification of goals, original agent needs to figure out what they are.
And since the task of constructing future agent includes specification of goals . . .
There seems to be a leap, here. An agent, qua agent, has goals. But is it clear that the historical way in which the future-agent is constructed by the original agent must pass through an explicit specification of the future-agent’s goals? The future-agent could be constructed that way, but must it? (Analogously, a composite integer has factors, but a composite can be constructed without explicitly specifying its factors.)
Goals don’t need to be specified explicitly, all that’s required is that it’s true that future agent has goals similar to original agent’s. However, since construction of future agent is part of original agent’s behavior that contributes to original agent’s goals (by my definition), it doesn’t necessarily make sense for the agent to prove that goals are preserved, it just needs to be true that they are (to some extent), more as an indication that we understand original agent correctly than a consideration that it takes into account.
For example, original agent might be bad at accomplishing its “normative” goals, and even though it’s true that it optimizes the environment to some extent, it doesn’t do it very well, so definition of “normative” goals (related in my definition to actual effect on environment) doesn’t clearly derive from original agent’s construction, except specifically for its tendency to construct future agents with certain goals (assuming it can do that true to the “normative” goals), in which case future agent’s goals (as parameters of design) are closer to the mark (actual effect on environment and “normative” goals) than original agent’s (as parameters of design).
However, since construction of future agent is part of original agent’s behavior that contributes to original agent’s goals (by my definition), it doesn’t necessarily make sense for the agent to prove that goals are preserved, it just needs to be true that they are (to some extent), more as an indication that we understand original agent correctly than a consideration that it takes into account.
(Emphasis added.) For that sense of “specify”, I agree.
Specification of goals doesn’t need to be explicit, the only thing that’s necessary is showing that new goals are sufficiently close to original goals. Since “goals” are already a rather abstract property of the way an agent behaves, saying the they are “specified” is close to just saying that the agent (in environment) is specified, and showing in what way that relates to the implicit goals.
So an agent is already associated with goals in terms of its actual effect on its environment. Given that agent’s own future state (design) is an easily controlled part of the environment, it’s one of the things that’ll be optimized...
If you added general intelligence and consciousness to IBM Watson, where does the urge to refine or protect its Jeopardy skills come from? Why would it care if you pulled the plug on it? I just don’t see how optimization and goal protection are inherent features of general intelligence, agency or even consciousness.
He seems to be arguing around the definition of an agent using BDI or similar logic; BDI stands for beliefs-desires-intentions, and the intentions are goals. In this framework (more accurately, set of frameworks) agents necessarily, by definition have goals. More generally, though, I have difficulty envisioning anything that could realistically be called an “agent” that does not have goals. Without goals you would have a totally reactive intelligence, but it could not do anything without being specifically instructed, like a modern computer.
ADDED: Thinking further, such a “goal-less” intelligence couldn’t even try to foresee questions in order to have answers ready, or take any independent action. You seem to be arguing for an un-intelligent, in any real meaning of the word, intelligence.
Consider someone with a larger inferential distance, e.g. a potential donor. The title “The Basic AI Drives” seems to be a misnomer, given the amount of presuppositions inherent in your definition. There exist a vast amount of possible AI designs, that would appear to be agents, that would have no incentive to refine or even protect their goals.
Saying that there is an agent refers (in my view; definition for this thread) to a situation where future events are in some sense expected to be optimized according to some goals
Omohundro’s paper says:
Researchers have explored a wide variety of architectures for building intelligent systems: neural networks, genetic algorithms, theorem provers, expert systems, Bayesian networks, fuzzy logic, evolutionary programming, etc. Our arguments apply to any of these kinds of system as long as they are sufficiently powerful.
It’s not obvious to me why any of these systems would be “agents” under your definition. So I guess your definition is too strong. My question stands.
The “sufficiently powerful” clause seems to me like something that should translate as roughly my definition, making implementation method irrelevant for essentially the same reasons. In context, “powerful” means “powerful as a consequentialist agent”, and that’s just what I unpacked (a little bit) in my definition.
(It’s unknown how large the valley is between a hacked together AI that can’t get off the ground and a hacked together AI that is at least as reflective as, say, Vladimir Nesov. Presumably Vladimir Nesov would be very wary of locking himself into a decision algorithm that was as unreflective as many synax-manipulator/narrow-AI-like imagined AGIs that get talked about by default around here/SingInst.)
Saying that there is an agent refers (in my view; definition for this thread) to a situation where future events are in some sense expected to be optimized according to some goals, to the extent certain other events (“actions”) control those future events. There might be many sufficient conditions for that in terms of particular AI designs, but they should amount to this expectation.
So an agent is already associated with goals in terms of its actual effect on its environment. Given that agent’s own future state (design) is an easily controlled part of the environment, it’s one of the things that’ll be optimized, and given that agents are particularly powerful incantations, it’s a good bet that future will retain agent-y patterns, at least for a start. If future agent has goals different from the original, this by the same definition says that the future will be optimized for different goals, and yet in a way controllable by original agent’s actions (through the future agent). This contradicts that the original agent is an agent (with original goals). And since the task of constructing future agent includes specification of goals, original agent needs to figure out what they are.
There seems to be a leap, here. An agent, qua agent, has goals. But is it clear that the historical way in which the future-agent is constructed by the original agent must pass through an explicit specification of the future-agent’s goals? The future-agent could be constructed that way, but must it? (Analogously, a composite integer has factors, but a composite can be constructed without explicitly specifying its factors.)
Goals don’t need to be specified explicitly, all that’s required is that it’s true that future agent has goals similar to original agent’s. However, since construction of future agent is part of original agent’s behavior that contributes to original agent’s goals (by my definition), it doesn’t necessarily make sense for the agent to prove that goals are preserved, it just needs to be true that they are (to some extent), more as an indication that we understand original agent correctly than a consideration that it takes into account.
For example, original agent might be bad at accomplishing its “normative” goals, and even though it’s true that it optimizes the environment to some extent, it doesn’t do it very well, so definition of “normative” goals (related in my definition to actual effect on environment) doesn’t clearly derive from original agent’s construction, except specifically for its tendency to construct future agents with certain goals (assuming it can do that true to the “normative” goals), in which case future agent’s goals (as parameters of design) are closer to the mark (actual effect on environment and “normative” goals) than original agent’s (as parameters of design).
(Emphasis added.) For that sense of “specify”, I agree.
Specification of goals doesn’t need to be explicit, the only thing that’s necessary is showing that new goals are sufficiently close to original goals. Since “goals” are already a rather abstract property of the way an agent behaves, saying the they are “specified” is close to just saying that the agent (in environment) is specified, and showing in what way that relates to the implicit goals.
If you added general intelligence and consciousness to IBM Watson, where does the urge to refine or protect its Jeopardy skills come from? Why would it care if you pulled the plug on it? I just don’t see how optimization and goal protection are inherent features of general intelligence, agency or even consciousness.
He seems to be arguing around the definition of an agent using BDI or similar logic; BDI stands for beliefs-desires-intentions, and the intentions are goals. In this framework (more accurately, set of frameworks) agents necessarily, by definition have goals. More generally, though, I have difficulty envisioning anything that could realistically be called an “agent” that does not have goals. Without goals you would have a totally reactive intelligence, but it could not do anything without being specifically instructed, like a modern computer.
ADDED: Thinking further, such a “goal-less” intelligence couldn’t even try to foresee questions in order to have answers ready, or take any independent action. You seem to be arguing for an un-intelligent, in any real meaning of the word, intelligence.
Consider someone with a larger inferential distance, e.g. a potential donor. The title “The Basic AI Drives” seems to be a misnomer, given the amount of presuppositions inherent in your definition. There exist a vast amount of possible AI designs, that would appear to be agents, that would have no incentive to refine or even protect their goals.
Omohundro’s paper says:
It’s not obvious to me why any of these systems would be “agents” under your definition. So I guess your definition is too strong. My question stands.
The “sufficiently powerful” clause seems to me like something that should translate as roughly my definition, making implementation method irrelevant for essentially the same reasons. In context, “powerful” means “powerful as a consequentialist agent”, and that’s just what I unpacked (a little bit) in my definition.
(It’s unknown how large the valley is between a hacked together AI that can’t get off the ground and a hacked together AI that is at least as reflective as, say, Vladimir Nesov. Presumably Vladimir Nesov would be very wary of locking himself into a decision algorithm that was as unreflective as many synax-manipulator/narrow-AI-like imagined AGIs that get talked about by default around here/SingInst.)