AI alignment researcher supported by MIRI and LTFF. Working on the learning-theoretic agenda. Based in Israel. See also LinkedIn.
E-mail: vanessa DOT kosoy AT {the thing reverse stupidity is not} DOT org
Vanessa Kosoy(Vanessa Kosoy)
Karma: 8,293
Identity and quining in UDT
Hi Nate, thx for commenting!
...every decision theory has an evil problem.
It seems to me this problem can be avoided by allowing access to random bits. See my reply to KnaveOfAllTrades and my reply to V_V. Formally, we should allow pi in (4′) to be a random algorithm.
...The above reasoning process selects a decision theory that logically-causes the worse outcome, and I don’t think that’s the right move.
I don’t think “logical causation” in the sense you are using here is the right way to think about the anti-Newcomb problem. From the precursor’s point of view, there is no loss in utility due of choosing XDT over UDT.
If you’re only trying to make agents that work well on “fair” games (where the obvious formalization of “fair” is “extensional” as defined above), then you should probably make that much more explicit.
Of course. I didn’t attempt to formalize “fairness” at that post but the idea is approaching optimality for decision-determined problems in the sense of Yudkowsky 2010.
I have the impression that the agents you define in this post, while interesting, aren’t really attacking the core of the problem, which is this: how can one reason under false premises?
I realize that the logical expectation values I’m using are so far mostly wishful thinking. However, I think there is benefit in attacking the problems from both ends: understanding the usage of logical probabilities may shed light on the desirada they should satisfy.
...UDT choosing strategies without regard for its inputs is the mechanism by which it is able to trade with counterfactual versions of itself.
Consider two UDT agents A & B with identical utility functions living in different universes. Each of the agents is charged with making a certain decision, while receiving no input. If both agents are aware of each other’s existence, we expect [in the sense of “hope” rather than “are able to prove” :)] them to make decisions that will maximizing overall utility, even though that on the surface, each agent is only maximizing over its own decisions rather than the decisions of both agents.
What is the difference between this scenario and the scenario of a single agent existing in both universes which receives a single bit of input that indicates in which universe the given copy is?
… I’m not quite sure why you’re so concerned with avoiding quining here.
See my reply to Wei Dai.
...how do we resolve the problem where sometimes it seems like agents with less computing power have some sort of logical-first-mover-advantage?
You’re referring to the agent-simulates-predictor problem? Actually, I think my (4′) may contain a clue for solving it. As I commented, the logical expectation values should only use about as much computer power as the precursor has rather than as much computing power as the successor has. Therefore, if the predictor is as at least strong as the precursor, the successor wins by choosing a policy pi which is a UDT agent symmetric to the predictor.
There are certainly some places where our thinking has diverged...
Hopefully, further discussion will lead us to a practical demonstration of Aumann’s theorem :)
A complexity theoretic approach to logical uncertainty (Draft)
Hi Patrick, thx for reading the paper and commenting!
I agree that the introduction section should be expanded.
Regarding examples of problems in , see section 7.
Optimal predictors and conditional probability
This is an updated version of the optical predictors paper which contains Theorem 4.5 and Theorem 4.6. I mentioned these results without proof during the logical uncertainty workshop in May.
There is a crucial difference between the setting of Theorem 4.4 and setting of Theorems 4.5, 4.6. In Theorems 4.5, 4.6 we consider the intersection of two languages in which case . In Theorem 4.4 we consider the Cartesian product of two languages. is not the same thing as . Moreover, the diagonal embedding of into is not a valid reduction for the purpose of Theorem 6.1 since condition (ii) is violated (diagonal elements are rare within the product ensemble).
Optimal predictors and propositional calculus
Exactly. It’s such a simple tweak that it’s embarrassing I haven’t noticed it in the first place. It’s just that in average complexity theory negligible errors seem much more popular. The price to pay is that some theorems require stronger assumptions namely something that had to be bounded by a polynomial now has to be bounded by a constant. On the other hand Theorems 9 demonstrates that there is no chance optimal predictors cover all of (for example) whereas quasi-optimal predictors might work. Also, I think that in there is a universal construction for uniform quasi-optimal predictors (as opposed to optimal predictors) similar to Levin’s universal search although I haven’t fleshed out the details yet (anyhow such a construction would be theoretically valid but highly inefficient in practice).
This is in the vein of the approach I call “epsitemic boxing”. The probability distribution over questions corresponds to the external inputs part of the stochastic model of the box.
EDIT: Corrected Example 5.2 and added Note 2 (previous Note 2 renamed to Note 3)
There is a typo in the text: “We say that S is an ??? with probability p.” I guess this is supposed to be “irreducible pattern”?
Btw, it seems that the definition makes sense for arbitrary promise problems, you don’t have to consider provability in particular.
Typos: “expressible it * bits” should be “expressible in * bits”
It appears to me that there is a natural analogue of the concept of irreducible pattern in the language of average case complexity. Moreover, the calibration theorem for optimal predictor schemes implies they pass the Benford test in the associated sense. I’ll write it down carefully and post...
Optimal predictor schemes pass a Benford test
I haven’t understood the details yet, but just a basic question. Your “randomized Turing machines”, do they have a random oracle or a random advice? In other words, do you assume the lookup of takes time or ?
EDIT: Deleted examples of generatable problems since they are wrong. They are in fact examples of a weaker notion which I think also admits uniform OPS but this should be explored elsewhere.
Hi Benja, thx for commenting!
I agree that it’s best to work on fully specified models. Hopefully, soon I will write about my own approach to logical uncertainty via complexity theory.