The AI can’t trick you that way, because it can’t tamper with the real you and the only unplug-decider who matters is the real you. The AI gains nothing by simulating versions of yourself who have been modified to make the wrong decision.
Strilanc
Sounds like we’re in agreement. I only meant that specific trick.
Suppose you were in a universe where particles really did have tags and you really could check them. How do you prove that no two particles have the same tag, implying they are truly interchangeable? As with everything else, there is no true proof either way. (Just evidence)
“Certainty” is an algorithmic optimization your brain performs (and evolution performed on brains) to avoid computing the effects of unlikely scenarios. It is an implementation detail of a decision making algorithm, not some guaranteed facet of decision theory delivered from on high.
It’s kind of strange, when you think about it, how much we project our brain’s strategies for working with the world onto reality itself. As if a chess program would perceive the universe as having some intangible “eightness” or “fourness” to it, because the chess program happens to use a search depth of 4 self moves + 4 opponent moves.
I had a similar experience with my girlfriend, except the symptoms were significantly more alarming. She was, among other things, unable to remember many common nouns. I would point and say ’What is that swinging room separator?” and she would be unable to figure out “door”.
I was aware from the start that the symptoms might have been due to a migraine aura, having looked up the symptoms on Wikipedia, but was advised by 811 to take her to the hospital immediately. The symptoms were gone before we arrived. Five hours later (a strong hint that at least the triage people thought it wasn’t an emergency), a doctor had diagnosed it as a silent migraine.
Well, I don’t really remember the exact boundary between Friend is Optimal vs Caelum est Conterrens, but...
People who don’t want to upload are eternally harassed as society crumbles around them until they give in or die.
People are lied to and manipulated constantly.
Everything non-human (puppies, trees, stars, aliens capable of radio communication) is destroyed.
The uploading process seemed to be destructive only for convenience’s sake.
Nitpick:
and some sacrifice cognitive ability to other pleasures (BBJ+03)), and many turn their backs on high-powered careers.
What part of “expected utility maximizer” don’t you understand?
It’s a bit confusing to quote across a bracket boundary like that. The bit about sacrificing cognitive ability for other pleasures is an example of “they don’t often try to improve their rationality”, whereas turning backs on careers was about expected utility maximization.
I agree that turning your back on a high-powered career is not a good example of failing to maximize utility, but trading cognition for pleasure seems like a reasonable example of not valuing, or failing to act on the value of, being more rational.
I’m going to nitpick on Section 3.8:
If there are several “hard steps” in the evolution of intelligence, then planets on which intelligent life does evolve should expect to see the hard steps spaced about equally across their history, regardless of each step’s relative difficulty. [...]
[...] [...]
[...] the time interval from Australopithecus to Homo sapiens is too short to be a plausible hard step.
I don’t think this argument is valid. Assuming there’s a last hard step, you’d expect intelligence to show up soon after it was made (because there’s no more hard steps).
In terms of the analogy to lock picking, it’s inappropriate to set the clock to timeout now. The clock should timeout when it’s too late for intelligence to succeed (e.g. when the sun has aged too much and is evaporating the oceans away).
Also, I tested if the claim about the even spacing was correct. The locks do appear to have the same distribution when you condition on being below the time limit. However, the picking times don’t appear to be evenly spaced particularly often. With a timeout of 1000 and clocks with pick-chances of 10^-3 through 10^-8 I get average standard deviations between picking times of ~300 which I think implies that the most common situation is for a lock or two to be picked quickly with the other locks consuming all the slack.
edit Now when I read the sentence I see something slightly different. Did you mean “the undertaking of the step should take a long time” or that “the amount of time since the last step was made should be a long time”?
Harry should be screaming at Dumbledore to use his time-turner. There are a lot of options, constrained mostly by the necessity of seeing a Hermione-looking-thing die.
“I’ve already used it six times today, Harry...”
[Two boxers] are interested in what aspects of the agent’s winning can be attributed to their decision and they say that we can attribute the agent’s winning to their decision if this is caused by their decision. This strikes me as quite a reasonable way to apportion the credit for various parts of the winning.
What do you mean by “the agent’s winning can be attributed to their decision”? The agent isn’t winning! Calling losing winning strikes me as a very unreasonable way to apportion credit for winning.
It would be helpful to me if you defined how you’re attributing winning to decisions. Maybe taboo the words winning and decision. At the moment I really can’t get my head around what you’re trying to say.
I still don’t follow. The causal effect of two-boxing is getting 1000$ instead of 1000000$. That’s bad. How are you interpreting it, so that it’s good? Because they’re following a rule of thumb that’s right under different circumstances?
In that case: the two-boxer isn’t just wrong, they’re double-wrong. You can’t just come up with some related-but-different function (“caused gain”) to maximize. The problem is about maximizing the money you receive, not “caused gain”.
For example, I’ve seen some two-boxers justify two-boxing as a moral thing. They’re willing to pay 999000$ for the benefit of throwing being predicted in the predictors face, somehow. Fundamentally, they’re making the same mistake: fighting the hypothetical by saying the payoffs are different than what was stated in the problem.
Constraint programming would be extremely powerful. They aren’t a silver bullet, but they would make a ton of hard problems a lot easier. Particularly when it comes to proving whole-program correctness properties like “does not use more than X memory”.
For example, when a human programmer is designing a public key cryptosystem, they’re trying to satisfy properties like “there is no tractable Turing Machine that takes a public key as input and outputs the corresponding private key with non-negligible probability”. Doing that is really, really hard compared to just writing down what we want to satisfy.
Verifying proofs and writing programs are isomorphic. A smarter-than-human theorem prover is likely to also be a better-than-human programmer, and you can ask it to do things like find good theorem provers...
It’s true that it would need to be insanely powerful. Even clever humans haven’t managed to prove that one-way hash functions exist (though we know a function that is one-way if and only if they exist...).
Note theorem proving is isomorphic to programming. Even if it wasn’t quite so equivalent, you could just ask for a constructive proof of a program satisfying whatever and the proof would contain the program.
I agree that the constants might be related in ways we don’t know, which would allow compression. I’m more interested in an upper bound on the complexity than an exact value (which is likely incomputable for halting problem reasons), so I’m willing to be over by 100 bits because we fail to see a pattern.
As far variable constants: Sure, we can estimate the kolmogorov complexity where the “constants” are inputs. Or we can estimate the kolmogorov complexity of the current laws plus the state 13.7 billion years ago at the big bang. Or we can estimate the complexity of a program that runs all programs. All of these questions are interesting. But right now I want the answer to the one I asked, not the others.
edit clarified response
I’m fine with a guess being included in the text, even one that isn’t elaborated on. The post wasn’t about making that particular estimate, and elaborating on it there would have been counter-productive.
However, that doesn’t really answer my question. What factors need to be accounted for when estimating the kolmogorov complexity of physics? Which parts cost the most? Which parts cost almost nothing?
Assuming you’re right about it only being a few more lines of code, and that you didn’t use a lot of external libraries, that puts an upper bound at… a few dozen kilobits? I’m guessing that could be made a lot smaller, since you were likely not focused on minimizing the lines of code at all costs.
First, remember that Kolmogorov complexity is only well-defined up to a constant, which is determined by your model of computation. [...]
Right, if you do something like (say) take away the addition instruction then the shortest program might get longer because it has to emulate addition using subtraction and negation (or use a totally different approach, or include a translation pass that expands the additions into negate+subtract or include a universal turing machine that runs the original machine or… yeah).
Second, the article addresses why you can have the paradoxical situation of the Universe being low-complexity while specific things in the Universe are of high complexity.
In this case I care about the complexity of the laws that govern the change in state (positions of atoms or values of the wave functions) with respect to time. I will not make you pay for the presumably absurd amount of data required for the initial state of >10^(82) atoms. That would make talking about the complexity of the laws laughably negligible. I realize that this is, in a sense, an arbitrary distinction but… that’s the question I want to know the answer to.
According to string theory (which is a Universal theory in the sense that it is Turing-complete) the landscape of possible Universes is 2^500 or so, which leads to 500 bits of information. Perhaps this is where Eliezer got the figure from (though I admit that I don’t exactly know where he got it from either).
Interesting guess at where the number could have come from. I just assumed he tallied up various aspects of the standard model somehow and got an answer between 450 and 510.
They typically produce a program that, when run, plays video and audio in real time after a loading period.
Data with lots of redundancy does, in a certain sense, contain a lot of junk. Junk that, although it helps reliably transmit the data, doesn’t change the meaning of the data (or doesn’t change it by much).