Note that Omohundro doesn’t assume that the AGI would actually have a utility function: he only assumes that the AGI is capable of understanding the microeconomic argument for why it would be useful for it to act as if it did have one. His earlier 2007 paper is clearer on this point.
Excellent point. But I think the assumptions about goal-directedness are still too strong. Omohundro writes:
Self-improving systems do not yet exist but we can predict how they might play chess. Initially, the rules of chess and the goal of becoming a good player would be supplied to the system in a formal language such as first order predicate logic1. Using simple theorem proving, the system would try to achieve the specified goal by simulating games and studying them for regularities. [...] As its knowledge grew, it would begin doing
“meta-search”, looking for theorems to prove about the game and discovering useful concepts such as “forking”. Using this new knowledge it would redesign its position representation and its strategy for learning from the game simulations.
That’s all good and fine, but doesn’t show that the system has a “goal of winning chess games” in the intuitive sense of that phrase. Unlike a human being or other mammal or bird, say, its pursuit of this “goal” might turn out to be quite fragile. That is, changing the context slightly might have the system happily solving some other, mathematically similar problem, oblivious to the difference. It could dramatically fail to have robust semantics for key “goal” concepts like “winning at chess”.
For example, a chess playing system might choose U to be the total number of games that it wins in a universe history.
That seems highly unlikely. More likely, the system would be programmed to maximize the percentage of its games that end in a win, conditional on the number of games it expects to play and the resources it has been given. It would not care how many games were played nor how many resources it was allotted.
On the other hand, Omohundro is making things too convenient for me by his choice of example. So let’s say we have a system intended to play the stock market and to maximize profits for XYZ Corporation. Further let’s suppose that the programmers do their best to make it true that the system has a robust semantics for the concept “maximize profits”.
OK, so they try. The question is, do they succeed? Bear in mind, again, that we are considering a “heuristic soup” approach.
Even at the risk of sounding like someone who’s arguing by definition, I don’t think that a system without any strongly goal-directed behavior qualifies as an AGI; at best it’s an early prototype on the way towards AGI. Even an oracle needs the goal of accurately answering questions in order to do anything useful, and proposals of “tool AGI” sound just incoherent to me.
Of course, that raises the question of whether a heuristic soup approach can be used to make strongly goal-directed AGI. It’s clearly not impossible, given that humans are heuristic soups themselves; but it might be arbitrarily difficult, and it could turn out that a more purely math-based AGI was far easier to make both tractable and goal-oriented. Or it could turn out that it’s impossible to make a tractable and goal-oriented AGI by the math route, and the heuristic soup approach worked much better. I don’t think anybody really knows the answer to that, at this point, though a lot of people have strong opinions one way or the other.
Note that Omohundro doesn’t assume that the AGI would actually have a utility function: he only assumes that the AGI is capable of understanding the microeconomic argument for why it would be useful for it to act as if it did have one. His earlier 2007 paper is clearer on this point.
Excellent point. But I think the assumptions about goal-directedness are still too strong. Omohundro writes:
That’s all good and fine, but doesn’t show that the system has a “goal of winning chess games” in the intuitive sense of that phrase. Unlike a human being or other mammal or bird, say, its pursuit of this “goal” might turn out to be quite fragile. That is, changing the context slightly might have the system happily solving some other, mathematically similar problem, oblivious to the difference. It could dramatically fail to have robust semantics for key “goal” concepts like “winning at chess”.
That seems highly unlikely. More likely, the system would be programmed to maximize the percentage of its games that end in a win, conditional on the number of games it expects to play and the resources it has been given. It would not care how many games were played nor how many resources it was allotted.
On the other hand, Omohundro is making things too convenient for me by his choice of example. So let’s say we have a system intended to play the stock market and to maximize profits for XYZ Corporation. Further let’s suppose that the programmers do their best to make it true that the system has a robust semantics for the concept “maximize profits”.
OK, so they try. The question is, do they succeed? Bear in mind, again, that we are considering a “heuristic soup” approach.
Even at the risk of sounding like someone who’s arguing by definition, I don’t think that a system without any strongly goal-directed behavior qualifies as an AGI; at best it’s an early prototype on the way towards AGI. Even an oracle needs the goal of accurately answering questions in order to do anything useful, and proposals of “tool AGI” sound just incoherent to me.
Of course, that raises the question of whether a heuristic soup approach can be used to make strongly goal-directed AGI. It’s clearly not impossible, given that humans are heuristic soups themselves; but it might be arbitrarily difficult, and it could turn out that a more purely math-based AGI was far easier to make both tractable and goal-oriented. Or it could turn out that it’s impossible to make a tractable and goal-oriented AGI by the math route, and the heuristic soup approach worked much better. I don’t think anybody really knows the answer to that, at this point, though a lot of people have strong opinions one way or the other.