(Sorry for the delayed reply.) I’m a little confused: it’s true that we can easily compare heuristics for world()s that depend on the agent only through easily identifiable call sites, but if we’re only considering this class of problems, what does your algorithm add over the simpler algorithm of replacing all call sites to agent() by a constant, running the result, and choosing the constant that gives the highest value?
(I suppose I was hoping for more direct game theoretical applications than you, so I had situations in mind where the (rest of) the world does depend on the source of the agent.)
Regarding examples, what I had in mind was: Surely your literal algorithm of enumerating all proofs is not efficient enough to run even on toy problems like your implementation of Newcomb! Also, surely for any real-world problem, it won’t be feasible to find the optimal solution. So at some point, we will want to use the theory as a background for looking for practical (and thus necessarily heuristic) algorithms.
what does your algorithm add over the simpler algorithm of replacing all call sites to agent() by a constant, running the result, and choosing the constant that gives the highest value?
Admittedly, not much. If world() calls agent2() that provably returns the same value as agent(), agent() will notice it. But this wasn’t really the point of the post. The point was to show that having perfect knowledge of yourself doesn’t necessarily stop you from considering counterfactuals about yourself. Of course it’s all completely impractical.
Perhaps a closer approximation to reality would be a “listener” algorithm that accepts proofs (e.g. by reading game theory books), instead of trying to find them.
(Sorry for the delayed reply.) I’m a little confused: it’s true that we can easily compare heuristics for world()s that depend on the agent only through easily identifiable call sites, but if we’re only considering this class of problems, what does your algorithm add over the simpler algorithm of replacing all call sites to agent() by a constant, running the result, and choosing the constant that gives the highest value?
(I suppose I was hoping for more direct game theoretical applications than you, so I had situations in mind where the (rest of) the world does depend on the source of the agent.)
Regarding examples, what I had in mind was: Surely your literal algorithm of enumerating all proofs is not efficient enough to run even on toy problems like your implementation of Newcomb! Also, surely for any real-world problem, it won’t be feasible to find the optimal solution. So at some point, we will want to use the theory as a background for looking for practical (and thus necessarily heuristic) algorithms.
Admittedly, not much. If world() calls agent2() that provably returns the same value as agent(), agent() will notice it. But this wasn’t really the point of the post. The point was to show that having perfect knowledge of yourself doesn’t necessarily stop you from considering counterfactuals about yourself. Of course it’s all completely impractical.
Perhaps a closer approximation to reality would be a “listener” algorithm that accepts proofs (e.g. by reading game theory books), instead of trying to find them.