That’s funny. What you described in the second paragraph is something like a 2-player bimatrix game played across time and space in which the players aren’t even sure of their opponents’ existence and in which our player’s strategy is which decision theory he uses.
Very interesting, and great food for thought. But again, the complication comes from the possible existence of another player. I would argue that it’s reasonable to assume ourselves some ‘breathing room’ of one to two million years before we have to deal with other players. Then in that case, why not build a ‘naive’ FAI which operates under the assumption that there is no other player, let it grow, and then when it has some free time, let it think of a decision theory for you? (I don’t know whether you speak for the SIAI, cousin_it, but I think it would be fair for an outsider to wonder why Yudkowsky thinks this route in particular has the greatest cost/benefit in terms of achieving FAI as fast as possible.)
I’m not affiliated with SIAI in any way. Just like you, I’m an outsider trying to clear up these topics for my own satisfaction :-)
Many people here think that we must get FAI right on the first try, because after it gains power it will resist our attempts to change it. If you code into the AI the assumption that it’s the only player, it won’t believe in other players even when it sees them, and will keep allocating resources to building beautiful gardens even as alien ships are circling overhead (metaphorically speaking). When you ask it to build some guns, it will see you as promoting a suboptimal strategy according to its understanding of what’s likely to work.
It might be preferable to build a less rigid AI that would be open to further amendments from humanity, rather than maximizing its initial utility function no matter what. But we don’t know any mathematical formalism that can express that. The first AIs are likely to be expected utility maximizers just because maximization of expected utility is mathematically neat.
The issue of rigidity is broad and important topic which has been insufficiently addressed on this site. A ‘rigid’ AI cannot be considered rational, because all rational beings are aware that their reasoning processes are prone to error. I would go on further to say that a rigid FAI can be just as dangerous (in the long-term) as a paperclip maximizer. However, the problem of implementing a ‘flexible’ AI would indeed be difficult. Such an AI would be a true inductive agent—even its confidence in the solidity of mathematical proof would be based on empirical evidence. Thus it would be difficult to predict how such an AI might function—there is a risk that the AI would ‘go insane’ as it loses confidence in the validity of the core assumptions underlying its cognitive processes. But this is already taking us far afield of the original subject of discussion.
That’s funny. What you described in the second paragraph is something like a 2-player bimatrix game played across time and space in which the players aren’t even sure of their opponents’ existence and in which our player’s strategy is which decision theory he uses.
Very interesting, and great food for thought. But again, the complication comes from the possible existence of another player. I would argue that it’s reasonable to assume ourselves some ‘breathing room’ of one to two million years before we have to deal with other players. Then in that case, why not build a ‘naive’ FAI which operates under the assumption that there is no other player, let it grow, and then when it has some free time, let it think of a decision theory for you? (I don’t know whether you speak for the SIAI, cousin_it, but I think it would be fair for an outsider to wonder why Yudkowsky thinks this route in particular has the greatest cost/benefit in terms of achieving FAI as fast as possible.)
I’m not affiliated with SIAI in any way. Just like you, I’m an outsider trying to clear up these topics for my own satisfaction :-)
Many people here think that we must get FAI right on the first try, because after it gains power it will resist our attempts to change it. If you code into the AI the assumption that it’s the only player, it won’t believe in other players even when it sees them, and will keep allocating resources to building beautiful gardens even as alien ships are circling overhead (metaphorically speaking). When you ask it to build some guns, it will see you as promoting a suboptimal strategy according to its understanding of what’s likely to work.
It might be preferable to build a less rigid AI that would be open to further amendments from humanity, rather than maximizing its initial utility function no matter what. But we don’t know any mathematical formalism that can express that. The first AIs are likely to be expected utility maximizers just because maximization of expected utility is mathematically neat.
+1 great explanation.
The issue of rigidity is broad and important topic which has been insufficiently addressed on this site. A ‘rigid’ AI cannot be considered rational, because all rational beings are aware that their reasoning processes are prone to error. I would go on further to say that a rigid FAI can be just as dangerous (in the long-term) as a paperclip maximizer. However, the problem of implementing a ‘flexible’ AI would indeed be difficult. Such an AI would be a true inductive agent—even its confidence in the solidity of mathematical proof would be based on empirical evidence. Thus it would be difficult to predict how such an AI might function—there is a risk that the AI would ‘go insane’ as it loses confidence in the validity of the core assumptions underlying its cognitive processes. But this is already taking us far afield of the original subject of discussion.