Designing a true 0 sum game situation is quite straightforward. Or at least a situation which both AI’s think is zero sum, and don’t try to cooperate. Consider both AI’s to be hypercomputers with a cartesian boundary. The rest of the world is some initially unknown Turing machine. Both agents are the obvious 2 player generalization of AIXI, The reward signal is shared after the magic incorruptible Cartesian boundary.
This is something that could be programmed on an indestructible hypercomputer.
I also suspect that some of the easiest shared 0 sum goals to make might be really wierd. Like maximise the number of ones on the right side of the tape head in a Turing machine representation of the universe.
You could even have two delusional AI’s that were both certain that phlogisten existed, one a phlogisten maximizer, the other a phlogisten minimizer. If they come up with the same crazy theories about where the phlogisten is hiding, they will act 0 sum.
I don ’t think this is straightforward in practice—and putting a cartesian boundary in place is avoiding exactly the key problem. Any feature of the world used as the item to minimize/maximize is measured, and uncorruptable measurement systems seems like a non-trivial problem. For instance, how do I get my GAI to maximize blue in an area instead of maximizing the blue input into their sensor when pointed at that area? We need to essentially solve value loading and understand a bunch of embedded agent issues to really talk about this.
Designing a true 0 sum game situation is quite straightforward. Or at least a situation which both AI’s think is zero sum, and don’t try to cooperate. Consider both AI’s to be hypercomputers with a cartesian boundary. The rest of the world is some initially unknown Turing machine. Both agents are the obvious 2 player generalization of AIXI, The reward signal is shared after the magic incorruptible Cartesian boundary.
This is something that could be programmed on an indestructible hypercomputer.
I also suspect that some of the easiest shared 0 sum goals to make might be really wierd. Like maximise the number of ones on the right side of the tape head in a Turing machine representation of the universe.
You could even have two delusional AI’s that were both certain that phlogisten existed, one a phlogisten maximizer, the other a phlogisten minimizer. If they come up with the same crazy theories about where the phlogisten is hiding, they will act 0 sum.
I don ’t think this is straightforward in practice—and putting a cartesian boundary in place is avoiding exactly the key problem. Any feature of the world used as the item to minimize/maximize is measured, and uncorruptable measurement systems seems like a non-trivial problem. For instance, how do I get my GAI to maximize blue in an area instead of maximizing the blue input into their sensor when pointed at that area? We need to essentially solve value loading and understand a bunch of embedded agent issues to really talk about this.