It’s more of a model for FAI meeting an alien AI in space. Suppose each side then has the choice of doing an arms buildup or not, and the payoffs for these choices are analogous to PD. (If one side builds up while the other doesn’t, it can attack and conquer the other. If both sides build up, it’s a stalemate and just wastes resources.) BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
I guess if both sides are using your decision theory, then whether the human FAI plays C or D against the alien AI depends on how much logical correlation the FAI thinks exists between its human designer and the alien AI’s designer, which does make sense (assuming we solve the problem of the FAI just simulating its designer and already knowing what their output is).
It’s more of a model for FAI meeting an alien AI in space.
Makes sense.
BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
Frankly, I didn’t have a specific realistic scenario in mind. I came up with the anti-Newcomb problem as simple semi-artificial problem demonstrating the problems with quining in UDT. The reason I started thinking about these problems is that it doesn’t seem “classical” UDT can be translated to realistic AGI architecture. UDT takes a finite number of bits and produces a finite number of bits whereas a realistic AGI has continuous input and output streams. Such an AGI has to somehow take into account a formal specification of its own hardware, and the natural way of introducing such a specification seems to me to go through introducing a precursor, specifically a precursor which is a Solomonoff average over all “theories of physics” containing the formal specification.
It’s more of a model for FAI meeting an alien AI in space. Suppose each side then has the choice of doing an arms buildup or not, and the payoffs for these choices are analogous to PD. (If one side builds up while the other doesn’t, it can attack and conquer the other. If both sides build up, it’s a stalemate and just wastes resources.) BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
I guess if both sides are using your decision theory, then whether the human FAI plays C or D against the alien AI depends on how much logical correlation the FAI thinks exists between its human designer and the alien AI’s designer, which does make sense (assuming we solve the problem of the FAI just simulating its designer and already knowing what their output is).
Makes sense.
Frankly, I didn’t have a specific realistic scenario in mind. I came up with the anti-Newcomb problem as simple semi-artificial problem demonstrating the problems with quining in UDT. The reason I started thinking about these problems is that it doesn’t seem “classical” UDT can be translated to realistic AGI architecture. UDT takes a finite number of bits and produces a finite number of bits whereas a realistic AGI has continuous input and output streams. Such an AGI has to somehow take into account a formal specification of its own hardware, and the natural way of introducing such a specification seems to me to go through introducing a precursor, specifically a precursor which is a Solomonoff average over all “theories of physics” containing the formal specification.