To clarify, I mean failures should not lead to a change of probability away from the prior probability; of course they do result in a different probability estimate than if the LHC succeeded and we survived.
simon2
Karma: 83
Actually, failures of the LHC should never have any effect at all on our estimate of the probability that if it did not fail it would destroy Earth.
This is because the ex ante probability of failure of the LHC is independent of whether or not if it turned on it would destroy Earth. A simple application of Bayes’ rule.
Now, the reason you come to a wrong conclusion is not because you wrongly applied the anthropic principle, but because you failed to apply it (or applied it selectively). You realized that the probability of failure given survival is higher under the hypothesis that the LHC would destroy the Earth if it did not fail, but you didn’t take into account the fact that the probability of survival is itself lower under that hypothesis (i.e. the anthropic principle).
As previously mentioned, there are tricky aspects to this. You can’t say: “You see those humans over there? Whatever desire is represented in their brains, is therefore right.” This, from a moral perspective, is wrong—wanting something doesn’t make it right—and the conjugate failure of the AI is that it will reprogram your brains to want things that are easily obtained in great quantity. If the humans are PA, then we want the AI to be PA+1, not Self-PA… metaphorically speaking.
Before reading this post, if I had been programming a friendly AI I would have attempted to solve this issue by programming the AI it to take into account only minds existing at the moment it makes its decisions. (the AI still cares about the future, but only to the extent that currently existing minds, if extrapolated, would care about the future). This technique has the flaw that it would be likely to fail in the event that time travel is easy (the AI invents it before it reprograms itself to eliminate the bug). But I think this would be easier to get right, and what is the chance of time travel being easy compared to the chance of getting the “right” solution wrong?
It’s not clear to me where you are going with it.
To argue that a proof is being made concluding ?C using the assumption ?(◻C → C) given the theory PA, to which proof we can apply the deduction theorem to get (PA |- ”?(◻C → C) → ?C”) (i.e. my interpretation of Löb’s Theorem)
We use 10 steps, 9 of which are proofs inside of PA
But the proof uses an additional assumption which is the antecedent of an implication, and comes to a conclusion which is the consequent of the implication. To get the implication, we must use the deduction theorem or something like it, right?
the fact that if PA |- X then PA |- “PA |- X”
Is this fact a theorem of first order logic without any additional assumptions, or is it merely a theorem of PA? I admit I don’t know, as I’m not very familiar with first order logic, but it intuitively seems to me that if first order logic were powerful enough on its own to express concepts like “PA proves X” it would probably be powerful enough to express arithmetic, in which case the qualification in Gödel’s theorem that it only applies to theories that express arithmetic would be superfluous.
Hmm. I was thinking that Löb’s Theorem was a theorem in PA, in which case the step going from
PA + ?(?C → C) |- ?(?L → C)
to
PA + ?(?C → C) |- ?(?(?L → C))
seems legitimate given
PA |- (?X → ?(?X))
which we ought to be able to use since PA is part of the theory before the |- symbol.
If we don’t have PA on the left, can we use all the “ingredients” without adding additional assumptions?
In any case, if we do not use the deduction theorem to derive the implication in Löb’s Theorem, what do we use?
We don’t have PA + X proving anything for any X.
It seems to me that we do have (PA + ”?(◻C → C)” |- ”?C”)
from which the deduction theorem gives: (PA |- ”?(◻C → C) → ?C”) which is Löb’s Theorem itself.
The hypothesis was that PA proves that “if PA proves C, then C” This enabled it to be proved that “PA proves C” So I think what we actually get applying the deduction theorem is ?((◻C)->C)->?C
I don’t think you’re talking about my sort of view* when you say “morality-as-preference”, but:
Why do people seem to mean different things by “I want the pie” and “It is right that I should get the pie”? Why are the two propositions argued in different ways?
A commitment to drive a hard bargain makes it more costly for other people to try to get you to agree to something else. Obviously an even division is a Schelling point as well (which makes a commitment to it more credible than a commitment to an arbitrary division).
When and why do people change their terminal values? Do the concepts of “moral error” and “moral progress” have referents? Why would anyone want to change what they want?
I think humans tend not to have very clean divisions between instrumental and terminal values. Although there is no absolute moral progress or error, some moralities may be better or worse than others by almost any moral standard a human would be likely to use. Through moral hypocrisy, humans can signal loyalty to group values while disobeying them. Since humans don’t self modify easily, a genuine desire to want to change may be a cost-effective way to improve the effectiveness of this strategy.
Why and how does anyone ever “do something they know they shouldn’t”, or “want something they know is wrong”? Does the notion of morality-as-preference really add up to moral normality?
See above on signaling and hypocrisy.
*moral nihilist with instrumental view of morality as tool for coordinating behaviour.
“From a utilitarian perspective”, where does the desire to do things better than can be done with the continued existence of humans come from? If it comes from humans, should not the desire to continue to exist also be given weight?
Also, if AI researchers anchor their expectations for AI on the characteristics of the average human then we could be in big trouble.
Gordon, humans respond in kind to hatred because we are programmed to by evolution, not because it is a universal response of all “ghosts”. But of course an AI won’t have compassion etc. either unless programmed to do so.
The last student should use a fuctional language. He’s right, the computer could easily be programmed to handle any order (as long as the IO is in the right sequence, and each variable is only assigned one value). So it’s reasonable for him to expect that it would be.
Michael: Eliezer has at least in the past supported coherent extrapolated volition. I don’t know if this is up-to-date with his current views.
Unless there is a surprising amount of coherence between worlds with different lottery outcomes, this mangled worlds model should still be vulnerable to my lottery winning technique (split the world a bunch of times if you win).
I guess I was too quick to assume that mangled worlds involved some additional process. Oops.
Eliezer: OK, so you object to branching indifference.
Here is what I was going to reply until I remembered that you support mangled worlds:
“So, I guess I’ll go buy a lottery ticket, and if I win, I’ll conduct an experiment that branches the universe 10^100 times (eg. single electron Stern-Gerlach repeated less than 1000 times). That way I’ll be virtually certain to win.”
Now, I suppose with mangled worlds and a low cutoff you can’t quite rule out your point of view experimentally this way. But you’re still proposing a rule in which if you have a world which splits into world A and world B, they have probability 1⁄2 each, and then when world B splits into B1 and B2, it changes the probability of A to 1⁄3 - until an unobserved physical process turns the probability of A back to 1⁄2. Seems a little odd, no?
Sorry for the impulsive unhelpful bit of my previous comment. Of course if you have a number ambiguity between subjectively identical minds, then you might have problems if you apply an indifference principle to determine probabilities. But please explain if you have any other problem with this.
I haven’t yet devised a way to express my appreciation of the orderliness of the universe, which doesn’t involve counting people in orderly states as compared to disorderly states.
What do you mean by that?
Frankly, I’m not sure what it is that you’re complaining about. Even in ordinary life humans have number ambiguity: if you split the connection between the halves of the brain, you get what seems to be two minds, but why should this be some great problem?
But unfortunately there’s that whole thing with the squared modulus of the complex amplitude giving the apparent “probability” of “finding ourselves in a particular blob”.
I hope you will at least acknowledge the existence of the point of view of Wallace/Saunders/Deutsch that the Born rule can be derived from quantum mechanics without it plus only very reasonable outside assumptions, if you won’t agree with it.
anonymous, the rate of change in amplitude at a location depends only on the derivatives at that location (and the derivative of a function at a point depends only on the values near that point).
Eliezer, I am on the whole inclined to agree with Psy-Kosh, but I sometimes suspect (wild unsupported speculation) that perhaps a locality rule is fundamental and spacetime itself is not, but derived from the locality.
In classical configuration spaces, you can take a single point in the configuration space, and the single point describes the entire state of a classical system. So you can take a single point in classical configuration space, and ask how the corresponding system develops over time. You can take a single point in classical configuration space, and ask, “Where does this one point go?”
The development over time of quantum systems depends on things like the second derivative of the amplitude distribution. Our laws of physics describe how amplitude distributions develop into new amplitude distributions. They do not describe, even in principle, how one configuration develops into another configuration.
Instead of viewing the wavefunction as some kind of structure encompassing many points in configuration space, you can view the wavefunction as a whole as a single point in configuration space. Then the evolution in configuration space does indeed depend only on the point itself, not its neighbourhood.
MIND IS FUNDAMENTAL AFTER ALL! CONSCIOUS AWARENESS DETERMINES OUR EXPERIMENTAL RESULTS!
You can still read this kind of stuff. In physics textbooks.
I hope this is just a strawman of the Copenhagen interpretation. If not, what textbooks are you reading?
Allan, I am of course aware of that (actually, it would probably take time, but even if the annihilation were instantaneous the argument would not be affected).
There are 4 possibilities:
The LHC would destroy Earth, but it fails to operate
The LHC destroys Earth
The LHC would not destroy Earth, but it fails anyway
The LHC works and does not destroy Earth
The fact that conditional on survival possibility 2 must not have happened has no effect on the relative probabilities of possibility 1 and possibility 3.