# AlexMennen

Karma: 3,139

NewTop

I don’t think that specifying the property of importance is simple and helps narrow down S. I think that in order for predicting S to be important, S must be generated by a simple process. Processes that take large numbers of bits to specify are correspondingly rarely occurring, and thus less useful to predict.

Suppose that I just specify a generic feature of a simulation that can support life + expansion (the complexity of specifying “a simulation that can support life” is also paid by the intended hypothesis, so we can factor it out). Over a long enough time such a simulation will produce life, that life will spread throughout the simulation, and eventually have some control over many features of that simulation.

Oh yes, I see. That does cut the complexity overhead down a lot.

Once you’ve specified the agent, it just samples randomly from the distribution of “strings I want to influence.” That has a way lower probability than the “natural” complexity of a string I want to influence. For example, if 1/quadrillion strings are important to influence, then the attackers are able to save log(quadrillion) bits.

I don’t understand what you’re saying here.

I didn’t mean that an agenty Turing machine would find S and then decide that it wants you to correctly predict S. I meant that to the extent that predicting S is commonly useful, there should be a simple underlying reason why it is commonly useful, and this reason should give you a natural way of computing S that does not have the overhead of any agency that decides whether or not it wants you to correctly predict S.

This reasoning seems to rely on there being such strings S that are useful to predict far out of proportion to what you would expect from their complexity. But a description of the circumstance in which predicting S is so useful should itself give you a way of specifying S, so I doubt that this is possible.

I think decision problems with incomplete information are a better model in which to measure optimization power than deterministic decision problems with complete information are. If the agent knows exactly what payoffs it would get from each action, it is hard to explain why it might not choose the optimal one. In the example I gave, the first agent could have mistakenly concluded that the .9-utility action was better than the 1-utility action while making only small errors in estimating the consequences of each of its actions, while the second agent would need to make large errors in estimating the consequences of its actions in order to think that the .1-utility action was better than the 1-utility action.

I’m not convinced that the probability of S’ could be pushed up to anything near the probability of S. Specifying an agent that wants to trick you into predicting S’ rather than S with high probability when you see their common prefix requires specifying the agency required to plan this type of deception (which should be quite complicated), and specifying the common prefix of S and S’ as the particular target for the deception (which, insofar as it makes sense to say that S is the “correct” continuation of the prefix, should have about the same “natural” complexity as S). That is, specifying such an agent requires all the information required to specify S, plus a bunch of overhead to specify agency, which adds up to much more complexity than S itself.

The multi-armed bandit problem is a many-round problem in which actions in early rounds provide information that is useful for later rounds, so it makes sense to explore to gain this information. That’s different from using exploration in one-shot problems to make the counterfactuals well-defined, which is a hack.

Some undesirable properties of C-score:

It depends on how the space of actions are represented. If a set of very similar actions that achieve the same utility for the agent are merged into one action, this will change the agent’s C-score.

It does not depend on the magnitudes of the agent’s preferences, only on their orderings. Compare 2 agents: the first has 3 available actions, which would give it utilities 0, .9, and 1, respectively, and it picks the action that would give it utility .9. The second has 3 available actions, which would give it utilities 0, .1, and 1, respectively, and it picks the action that would give it utility .1. Intuitively, the first agent is a more successful optimizer, but both agents have the same C-score.

# A comment on the IDA-AlphaGoZero metaphor; capabilities versus alignment

Logical induction does not take the outputs of randomized algorithms into account. But it does listen to deterministic algorithms that are defined by taking a randomized algorithm but making it branch pseudo-randomly instead of randomly. Because of this, I expect that modifying logical induction to include randomized algorithms would not lead to a significant gain in performance.

Oh, I see. But for any particular value of n, the claim that there are n! permutations of n objects is something we can know in advance is resolvable (even if we haven’t noticed why this is always true), because we can count the permutations and check.

Are you making the point that we often reason about how likely a sentence is to be true and then use our conclusion as evidence about how likely it is to have a proof of reasonable length? I think this is a good point. One possible response is that if we’re doing something like logical induction in that it listens to many heuristics and pays more attention to the ones that have been reliable, then some of those heuristics can involve performing computations that look like trying to estimate how likely a sentence is to be true, in the process of estimating how likely it is to have a short proof, and then we can just pay attention to the probabilities suggested for existence of a short proof. A possible counterobjection is that if you want to know how to be such a successful heuristic, rather than just how to aggregate successful heuristics, you might want to reason about probabilities of mathematical truth yourself. A possible response to this counterobjection is that yes, maybe you should think about how likely a mathematical claim is to be true, but it is not necessary for your beliefs about it to be expressed as a probability, since it is impossible to resolve bets made about the truth of a mathematical sentence, absent a procedure for checking it.

# Logical uncertainty and Mathematical uncertainty

2. No, neither classical logic nor probability theory as the extension of classical propositional logic assumes anything about observers, or their numbers, or experiments, or what may happen in the future.

Right, probability theory itself makes no mention of observers at all. But the development of probability theory and the way that it is applied in practice were guided by implicit assumptions about observers.

3. Selection effects are routinely handled within the framework of standard probability theory. You don’t need to go beyond standard probability theory for this.

You seemed to argue in your first post that selection effects were not routinely handled within standard probability theory. Unless perhaps you see a significant difference between the selection effect that suggests that the coin has a

^{1}⁄_{3}chance of having landed heads in the Sleeping Beauty problem and other selection effects? I was attempting to concede for the sake of argument that accounting for selection effects as typically practiced depart from standard probability theory, not advance it as an argument of my own.1. …

Certainly agreed as to logic (which does not include probability theory). As for probability theory, it should not be a priori surprising if a formalism that we had strong intuitive reasons for being very general, in which we made certain implicit assumptions about observers (which do not appear explicitly in the formalism) in these intuitive justifications, turned out not to be so generally applicable in situations in which those implicit assumptions were violated. As for whether probability theory does actually lack generality in this way, I’m going to wait to address that until you clarify what you mean by applying standard probability theory, since you offered a fairly narrow view of what this means in your original post, and seemed to contradict it in your point 3 in the comment. My position is that “the information available” should not be interpreted as simply the existence of at least one agent making the same observations you are, while declining to make any inferences at all about the number of such agents (beyond that it is at least 1). I take no position on whether this position violates “standard probability theory”.

I agree that it is difficult to see things from the perspective of people in such a world, but we should at least be able to think about whether certain hypotheses about how they’d think are plausible. That may still be difficult, but ordinary reasoning is not easy to do reliably in these cases either; if it was, then presumably there would be a consensus on how to address the Sleeping Beauty problem.

The questions of whether two duplicates are actually different people and of whether they count twice in moral calculations are different questions, and would likely be answered differently. People often answer these questions differently in the real world: people are usually said to remain the same person over time, but I think if you ask whether it is better to improve the daily quality of life of someone who’s about to die tomorrow, or improve the daily quality of life of someone who will go on to live a long life by the same amount, I think most people would agree that the second one is better because the beneficiary will get more use out of it, despite each time-slice of the beneficiary benefiting just as much in either case. Anyway, I was specifically talking about the extent to which experiences being differentiated would influence the subjective probability beliefs of such people. If they find it useful to assign probabilities in ways that depend on differentiating copies of themselves, this is probably because the extent to which they care about future copies of themselves depends on how those copies are differentiated from each other, and I can’t see why they might decide that the existence of a future copy of them decreases the marginal value of an additional identical copy down to 0 while having no effect on the marginal value of an additional almost identical copy.

Sleeping Beauty may be less fantastical, but it is still fantastical enough that such problems did not influence the development of probability theory. As I said, even testing hypotheses that correlate with how likely you are to survive to see the result of the test are too fantastical to influence the development of probability theory, despite such things actually occurring in real life. My point was that people who see Sleeping Beauty-like problems as a normal part of everyday life would likely have a better perspective on the problem than we do, so it might be worth trying to think from their perspective. The fact that Sleeping Beauty-type problems being normal is more fantastical than a Sleeping Beauty-type problem happening once doesn’t change this.

I think your criticism that the usual thirder arguments fail to properly apply probability theory misses the mark. The usual thirder arguments correctly avoid using the standard formalization of probability, which was not designed with anthropic reasoning in mind. It is usually taken for granted that the number of copies of you that will be around in the future to observe the results of experiments is fixed at exactly 1, and that there is thus no need to explicitly include observation selection effects in the formalism. Situations in which the number of future copies of you around could be 0, and in which this is correlated with hypotheses you might want to test, do occur in real life, but they are rare enough that they did not influence the development of probability theory. The fact that you can fit anthropic reasoning into a formalism that was not designed to accommodate it by identifying your observation with the existence of at least one observer making that observation, but that there is some difficulty fitting anthropic reasoning into the formalism in a way that weights observations by the number of observers making that observation, is not conclusive evidence that the former is an appropriate thing to do.

Imagine a world where people getting split into multiple copies and copies getting deleted is a daily occurrence, in which most events will correlate with the number of copies of you around in the near future, so that these people would not think it makes sense to assume for simplicity that the hypotheses they are interested in are independent of the number of copies of them that will be around. How would people in this world think about probability. I suspect they would be thirders. Or perhaps they would be halfers, or this would still be a controversial issue for them, or they’d come up with something else entirely. But one thing I am confident they would not do is think about what sources of randomness are available to them that might make different copies have slightly different experiences. This behavior is not useful. Do you disagree that people in this world would be very unlikely to handle anthropic reasoning in the way you suggest? Do you disagree that people in this world are better positioned to think about anthropic reasoning than we are?

I don’t find your rebuttal to travisrm89 convincing. Your response amounts to reiterating that you are identifying observations with the existence of at least one observer making that observation. But each particular observer only sees one bit or the other, and which bit it sees is not correlated with anything interesting, so all the observers finding a random bit shouldn’t do any of them any good.

And I have a related objection. The way of handling anthropic reasoning you suggest is discontinuous, in that it treats two identical copies of an observer much differently from two almost identical copies of an observer, no matter how close the latter two copies get to being identical. In an analog world, this makes no sense. If Sleeping Beauty knows that on Monday, she will see a dot in the exact center of her field of vision, but on Tuesday, if she wakes at all, she will see a dot just slightly to the right of the center of her field of vision, this shouldn’t make any difference if the dot she sees on Tuesday is far too close to the center for her to have any chance of noticing that it is off-center. But there also shouldn’t be some precise nonzero error tolerance that is the cutoff between “the same experience” and “not the same experience” (if there were, “the same” wouldn’t even be transitive). A sharp distinction between identical experiences and similar experiences should not play any role in anthropic reasoning.

In any particular structure, each proposition is simply true or false. But one proposition can be true in some structure and false in another structure. The universe could instantiate many structures, with non-indexical terms being interpreted the same way in each of them, but indexical terms being interpreted differently. Then sentences not containing indexical terms would have the same truth value in each of these structures, and sentences containing indexical terms would not. None of this contradicts using classical logic to reason about each of these structures.

I’m sympathetic to the notion that indexical language might not be meaningful, but it does not conflict with classical logic.

Good point that there can be fairly natural finite measures without there being a canonical or physically real measure. But there’s also a possibility that there is no fairly natural finite measure on the universe either. The universe could be infinite and homogeneous in some sense, so that no point in space is any easier to point to than any other (and consequently, none of them can be pointed to with any finite amount of information).

Ok, I see what you’re getting at now.