Stupid question… Can one construct ImpudentBot, specifically designed to defeat the analysis done by PrudentBot, say, by forcing it to never halt or something? Or PrudeBot, which resists all attempts at uncovering its true appearance?
PrudentBot halts on all inputs. More accurately, I should say, the modal form of PrudentBot has an output that’s always decidable in PA+2, while the algorithm version only looks for proofs up to a certain pre-specified length, and thus always halts. In either case, PrudentBot defects by default if it cannot prove anything.
So ImpudentBot is not possible, and PrudeBot earns a big fat D from PrudentBot.
Incidentally, this is the big distinction between “programs that try and run other programs’ code” and “programs that try to prove things about other programs’ output”: Löb’s Theorem allows you to avoid the danger of infinite regress.
I wouldn’t. PrudentBot is unexploitable, so the best thing I could do against it is achieve mutual cooperation.
But that doesn’t mean PrudentBot is optimal in any strong sense, because the same statement above is true about FairBot as well, and FairBot wastes utility by mutually cooperating with CooperateBot. PrudentBot makes similar miscalculations, but they’re all against complicated agents that you wouldn’t realistically submit, and the mistakes are of the form “getting (D,D) when I could have gotten (C,C)” and “getting (C,C) when I could have gotten (D,C)”.
the mistakes are of the form “getting (D,D) when I could have gotten (C,C)” and “getting (C,C) when I could have gotten (D,C)”.
Interesting. How hard is it to construct this anti-prudent bot?
Relatedly, so the only way for PrudentBot to lose an open-source tournament is when there are many imprudent entries against which some other algorithm can get (D,C) where PrudentBot gets (C,C)?
You can’t get more points than PrudentBot in one-to-one combat, but you can try doing better against the environment (other players).
There is always an environment where agent X gets more points than PrudentBot. Specifically, an environment of ServantOfX bots that recognize the source code of X, cooperate with X, and defect against everyone else.
Technically, there is a difference between “maximizing your score” and “beating PrudentBot”. (Focusing on “beating PrudentBot” makes it a zero-sum game.) If you know in advance that there will be more copies of you than copies of PrudentBot, you could just decide to defect against PrudentBot, which will harm it more than it will harm you. But it will harm you. You will defeat PrudentBot, but get less utility than you could get otherwise. (You will prove suboptimality of PrudentBot by becoming suboptimal yourself.)
Actually, you prove that there does not exist an optimal strategy for the game “Score more points that everybody else in a given competition.” The game “Score the maximum number of points possible” is subtly different.
The “maximum number of points” optimizing strategy maximizes the number of points in some average environment. In a sufficiently perverse environment, it will score low.
So, perhaps the question is how to define the “average” environment. Is it an environment containing bots with probability corresponding to their (Kolmogorov) complexity of code?
Stupid question… Can one construct ImpudentBot, specifically designed to defeat the analysis done by PrudentBot, say, by forcing it to never halt or something? Or PrudeBot, which resists all attempts at uncovering its true appearance?
PrudentBot halts on all inputs. More accurately, I should say, the modal form of PrudentBot has an output that’s always decidable in PA+2, while the algorithm version only looks for proofs up to a certain pre-specified length, and thus always halts. In either case, PrudentBot defects by default if it cannot prove anything.
So ImpudentBot is not possible, and PrudeBot earns a big fat D from PrudentBot.
Incidentally, this is the big distinction between “programs that try and run other programs’ code” and “programs that try to prove things about other programs’ output”: Löb’s Theorem allows you to avoid the danger of infinite regress.
Thanks! So, were you given a task of defeating PrudentBot, where would you start?
I wouldn’t. PrudentBot is unexploitable, so the best thing I could do against it is achieve mutual cooperation.
But that doesn’t mean PrudentBot is optimal in any strong sense, because the same statement above is true about FairBot as well, and FairBot wastes utility by mutually cooperating with CooperateBot. PrudentBot makes similar miscalculations, but they’re all against complicated agents that you wouldn’t realistically submit, and the mistakes are of the form “getting (D,D) when I could have gotten (C,C)” and “getting (C,C) when I could have gotten (D,C)”.
Interesting. How hard is it to construct this anti-prudent bot?
Relatedly, so the only way for PrudentBot to lose an open-source tournament is when there are many imprudent entries against which some other algorithm can get (D,C) where PrudentBot gets (C,C)?
You can’t get more points than PrudentBot in one-to-one combat, but you can try doing better against the environment (other players).
There is always an environment where agent X gets more points than PrudentBot. Specifically, an environment of ServantOfX bots that recognize the source code of X, cooperate with X, and defect against everyone else.
Technically, there is a difference between “maximizing your score” and “beating PrudentBot”. (Focusing on “beating PrudentBot” makes it a zero-sum game.) If you know in advance that there will be more copies of you than copies of PrudentBot, you could just decide to defect against PrudentBot, which will harm it more than it will harm you. But it will harm you. You will defeat PrudentBot, but get less utility than you could get otherwise. (You will prove suboptimality of PrudentBot by becoming suboptimal yourself.)
Actually, you prove that there does not exist an optimal strategy for the game “Score more points that everybody else in a given competition.” The game “Score the maximum number of points possible” is subtly different.
The “maximum number of points” optimizing strategy maximizes the number of points in some average environment. In a sufficiently perverse environment, it will score low.
So, perhaps the question is how to define the “average” environment. Is it an environment containing bots with probability corresponding to their (Kolmogorov) complexity of code?
I was going to say that there was a program which would score more than any other program in any given environment.
But when I write that out, it’s trivial to falsify.
There does not exist a strategy which is dominant in all environments.