No, he says “you’re the first person who etc...”
Allan_Crossman
Karma: 188
Is this a “failed utopia” because human relationships are too sacred to break up, or is it a “failed utopia” because the AI knows what it should really have done but hasn’t been programmed to do it?
that can support the idea that the much greater incidence of men committing acts of violence is “natural male aggression” that we can’t ever eliminate.
The whole point of civilisation is to defeat nature and all its evils.
… how isn’t atheism a religion? It has to be accepted on faith, because we can’t prove there isn’t a magical space god that created everything.
I think there’s a post somewhere on this site that makes the reasonable point that “is atheism a religion?” is not an interesting question. The interesting question is “what reasons do we have to believe or disbelieve in the supernatural?”
My issue with this is that we don’t, actually, have a philosophical/rational/scientific vision of capital-T Truth yet, despite all of our efforts. (Descartes, Spinoza, Kant, etc.)
Truth is whatever describes the world the way it is.
Even the capital-T Truth believers will admit that we don’t know how to achieve an understanding of that truth, they’ll just say that it’s possible because there really is this kind of truth.
Do you mean an understanding of the way the world is, or an understanding of what truth is?
Isn’t it the case, then that your embracing this kind of objective truth is itself a “true because it’s useful” kind of thinking, not a “true because it’s true” kind of thinking?
You can of course define “truth” however you like—it’s just a word. If you’re expecting some sort of actual relationship to hold between—say—ink on a page saying “Jupiter has four large moons” and the moons of Jupiter themselves, then of course there’s no such thing; the universe is just made of protons, electrons, and such mundane objects.
But there still really are four large moons of Jupiter.
Paul, that’s a good point.
Eliezer: If all I want is money, then I will one-box on Newcomb’s Problem.
Mmm. Newcomb’s Problem features the rather weird case where the relevant agent can predict your behaviour with 100% accuracy. I’m not sure what lessons can be learned from it for the more normal cases where this isn’t true.
If a serial killer comes to a confessional, and confesses that he’s killed six people and plans to kill more, should the priest turn him in? I would answer, “No.” If not for the seal of the confessional, the serial killer would never have come to the priest in the first place.
It’s important to distinguish two ways this argument might work. The first is that the consequences of turning him in are bad, because future killers will be (or might be) less likely to seek advice from priests. That’s a fairly straightforward utilitarian argument.
But the second is that turning him in is somehow bad, regardless of the consequences, because the world in which every “confessor” did as you do is a self-defeating, impossible world. This is more of a Kantian line of thought.
Eliezer, can you be explicit which argument you’re making? I thought you were a utilitarian, but you’ve been sounding a bit Kantian lately. :)
Benja: But it doesn’t follow that you should conclude that the other people are getting shot, does it?
I’m honestly not sure. It’s not obvious to me that you shouldn’t draw this conclusion if you already believe in MWI.
(Clearly you learned nothing about that, because whether or not they get shot does not affect anything you’re able to observe.)
It seems like it does. If people are getting shot then you’re not able to observe any decision by the guards that results in you getting taken away. (Or at least, you don’t get to observe it for long—I’m don’t think the slight time lag matters much to the argument.)
Benja: Allan, you are right that if the LHC would destroy the world, and you’re a surviving observer, you will find yourself in a branch where LHC has failed, and that if the LHC would not destroy the world and you’re a surviving observer, this is much less likely. But contrary to mostly everybody’s naive intuition, it doesn’t follow that if you’re a surviving observer, LHC has probably failed.
I don’t believe that’s what I’ve been saying; the question is whether the LHC failing is evidence for the LHC being dangerous, not whether surviving is evidence for the LHC having failed.
Allan: your intuition is wrong here too. Notice that if Zeus were to have independently created a zillion people in a green room, it would change your estimate of the probability, despite being completely unrelated.
I don’t see how, unless you’re told you could also be one of those people.
Simon: As I say above, I’m out of my league when it comes to actual probabilities and maths, but:
P(W|F) = P(F|W)P(W)/P(F)
Note that none of these probabilities are conditional on survival.
Is that correct? If the LHC is dangerous and MWI is true, then the probability of observing failure is 1, since that’s the only thing that gets observed.
An analogy I would give is:
You’re created by God, who tells you that he has just created 10 people who are each in a red room, and depending on a coin flip God made, either 0 or 10,000,000 people who are each in a blue room. You are one of these people. You turn the lights on and see that you’re one of the 10 people in a red room. Don’t you immediately conclude that there are almost certainly only 10 people, with nobody in a blue room?
The red rooms represent Everett worlds where the LHC miraculously and repeatedly fails. The blue rooms represent Everett worlds where the LHC works. God’s coin flip is whether or not the LHC is dangerous.
i.e. You conclude that there are no people in worlds where the LHC works (blue rooms), because they’re all dead. The reasoning still works even if the coin is biased, as long as it’s not too biased.
Benja, I’m not really smart enough to parse the maths, but I can comment on the intuition:
The very small number of Everett branches that have the LHC non-working due to a string of random failures is the same in both cases [of LHC dangerous vs. LHC safe]
I see that, but if the LHC is dangerous then you can only find yourself in the world where lots of failures have occurred, but if the LHC is safe, it’s extremely unlikely that you’ll find yourself in such a world.
Thus, if all you know is that you are in an Everett branch in which the LHC is non-working due to a string of random failures, you have no information about whether the other Everett branches have the LHC happily chugging ahead, or dead.
The intuition on my side is that, if you consider yourself a random observer, it’s amazing that you should find yourself in one of the extremely few worlds where the LHC keeps failing, unless the LHC is dangerous, in which case all observers are in such a world.
(I would like to stress for posterity that I don’t believe the LHC is dangerous.)
Simon: the ex ante probability of failure of the LHC is independent of whether or not if it turned on it would destroy Earth.
But—if the LHC was Earth-fatal—the probability of observing a world in which the LHC was brought fully online would be zero.
(Applying anthropic reasoning here probably makes more sense if you assume MWI, though I suspect there are other big-world cosmologies where the logic could also work.)
Oh God I need to read Eliezer’s posts more carefully, since my last comment was totally redundant.
First collisions aren’t scheduled to have happened yet, are they? In which case, the failure can’t be seen as anthropic evidence yet, since we might as well be in a world where it hasn’t failed, since such a world wouldn’t have been destroyed yet in any case.
But if I’m not mistaken, even old failures will become evidence retrospectively once first collisions are overdue, since (assuming the unlikely case of the LHC actually being dangerous) all observers still alive would be in a world where the LHC failed; when it failed being irrelevant.
As much as the AP fascinates me, it does my head in. :)
From your perspective, you should chalk this up to the anthropic principle: if I’d fallen into a true dead end, you probably wouldn’t be hearing from me on this blog.
I’m not sure that can properly be called anthropic reasoning; I think you mean a selection effect. To count as anthropic, my existence would have to depend upon your intellectual development; which it doesn’t, yet. :)
(Although I suppose my existence as Allan-the-OB-reader probably does so depend… but that’s an odd way of looking at it.)
I’m interested in the inconsistency of those who accept defection as the rational equilibrium in the one-shot PD, but find excuses to reject it in the finitely iterated known-horizon PD.
[...] What if neither party to the IPD thinks there’s a realistic chance that the other party is stupid—if they’re both superintelligences, say?
It’s never worthwhile to cooperate in the one shot case, unless the two players’ actions are linked in some Newcomb-esque way.
In the iterated case, if there’s even a fairly small chance that the other player will try to establish cooperation, then it’s worthwhile to cooperate on move 1. And since both players are superintelligences, surely they both realise that there is indeed a sufficiently high chance, since they’re both likely to be thinking this. Is this line of reasoning really an “excuse”?
One more thing; could something like the following be made respectable?
The prior odds of the other guy defecting in round 1 are .999
But if he knows that I know fact #1, the odds become .999 x .999
But if he knows that I know facts #1 and #2, the odds become .999 x .999 x .999
Etc...
Or is this nonsense?
Carl—good point.
I shouldn’t have conflated perfectly rational agents (if there are such things) with classical game-theorists. Presumably, a perfectly rational agent could make this move for precisely this reason.
Probably the best situation would be if we were so transparently naive that the maximizer could actually verify that we were playing naive tit-for-tat, including on the last round. That way, it would cooperate for 99 rounds. But with it in another universe, I don’t see how it can verify anything of the sort.
(By the way, Eliezer, how much communication is going on between us and Clippy? In the iterated dilemma’s purest form, the only communications are the moves themselves—is that what we are to assume here?)
Vladimir: In case of prisoner’s dilemma, you are penalized by ending up with (D,D) instead of better (C,C) for deciding to defect
Only if you have reason to believe that the other player will do whatever you do. While that’s the case in Simpleton’s example, it’s not the case in Eliezer’s.
… and stuns Akon (or everyone). He then opens a channel to the Superhappies, and threatens to detonate the star—thus preventing the Superhappies from “fixing” the Babyeaters, their highest priority. He uses this to blackmail them into fixing the Babyeaters while leaving humanity untouched.