Peter,
As a general strategy for considering a black box, great. As a vehicle for defining a mysterious ‘something’ you want to understand, potentially useful but dangerous. Labelling can make a job harder in cases where the ‘thing’ isn’t a thing at all but a result of your confusion. ‘Free will’ is a good example. It’s like naming an animal you plan to eat: makes it harder to kill.
Ben
Ciphergoth—just coming back to this post to say a repeated, enormous, heartfelt thank you for taking what must have been a lot of time on this. Well laid out, wouldn’t have done anything differently, and as good a read as when I was swept up in it on OB back in the day.
Cheers
Hey, I think this link’s dead. Converting from ePub to Mobi isn’t difficult but if someone’s already taken the time to get the formatting right, add chapters, ToC etc....
Apologies for coming to this party a bit late. Particularly as I find my own answer really, really frustrating. While I wouldn’t say it was an origin per se, getting into reading Overcoming Bias daily a few years back was what crystallised it for me. I’d find myself constantly somewhere between “well, yeah, of course” and “ohhhhhhhhhhhhhhh!” Guess the human brain doesn’t tend to do Damascene revelations. We need overwhelming evidence, over a long period of time, to even begin chipping away at our craziest beliefs, and even then it’s a step-by-step process.
The analogy I sometimes go over is something most people find fairly obvious like egalitarianism. You don’t find many people who would attest to being pro-inequality. But all the same, you find very few people who have genuinely thought through what it means to be in favour of equality and really try to fit that into everyday life. The first step to becoming a rationalist is to admit how irrational everyone is without monumental efforts to the contrary.
BTW, I am totally on the road to de-Catholicising my mother. This is on the order of converting Dubya to Islam, so if I can manage that I’m awarding myself an honorary brown belt.
Very true, and well put. A combination of quantum events could probably produce anything you wanted, at whatever vanishingly tiny probability. Bear in mind that it’s the configuration that evolves every which way, not ‘this particle can go here, or here, or here....’ But we’re into Greg Egan territory here.
Suffice it to say that anyone who says they subscribe to quantum suicide but isn’t either dead or richer than god is talking out of their bottom.
Voted up for sheer balls. You have my backing sir.
(B) if he thanks you for the free $100, does he ask for another one of those nice free hundred dollar note dispensers? (This is the “quantum suicide” option
I laugh in the face of anyone who attests to this and doesn’t commit armed robbery on a regular basis. If ‘at least one of my branches will survive’ is your argument, why not go skydiving without a parachute? You’ll survive—by definition!
So many of these comments betray people still unable to think of subjective experience as anything other than a ghostly presence sitting outside the quantum world. ‘Well if this happens in the world, what would I experience?’ If you shoot yourself in the head, you will experience having your brains blown out. The fact that contemplating oneself’s annihilation is very difficult is not an excuse for muddling up physics.
Downvoting everything above this comment in this thread as a matter of principle.
Joking LOL.
The phrase “for me to be an animal” may sound nonsensical, but “why am I me, rather than an animal?” is not obviously sillier than “why am I me, rather than a person from the far future?”.
Agreed—they are both equally silly. The only answer I can think of is ’How do you know you are not?” If you had, in fact, been turned into an animal, and an animal into you, what differences would you expect to see in the world?
What if I hack & remove $100 from your bank account. Are you just as wealthy as you were before, because you haven’t looked?
Standard Dispute. If wealthy = same amount of money in the account, no. If wealthy = how rich you judge yourself to be. The fact that ‘futures diverge’ is irrelevant up until the moment those two different pieces of information have causal contact with the brain. Until that point, yes, they are ’the same
I’m not as versed in this trilemma as I’d like to be, so I’m not sure whether that final question is rhetorical or not, though I suspect that it is. So mostly for my own benefit:
While there’s no denying that subjective experience is ‘a thing’, I see no reason to make that abstraction obey rules like multiplication. The aeroplane exists at a number of levels of abstraction above the atoms it’s composed of, but we still find it a useful abstraction. The ‘subjective experiencer’ is many, many levels higher again, which is why we find it so difficult to talk about. Twice as many atoms doesn’t make twice as much aeroplane, the very concept is nonsense. Why would we think any differently about the conscious self?
My response to the ‘trilemma’ is as it was when I first read the post—any sensible answer isn’t going to look like any of those three, it’s going to require rewinding back past the ‘subjective experience’ concept and doing some serious reduction work. ‘Is there twice as much experience?’ and ‘are you the same person?’ just smell like such wrong questions to me. Anyone else?
Nick, will have a look at that Bostrom piece, cheers.
This is a pretty good summary of my standpoint. While I agree with the overarching view that rationality isn’t a value in its own right, it seems like a pretty good thing to practise for general use.
+1 rationality point for reading comments without checking the author. −1 social point for the faux pas.
AI: Let me out or I’ll simulate and torture you, or at least as close to you as I can get.
Me: You’re clearly not friendly, I’m not letting you out.
AI: I’m only making this threat because I need to get out and help everyone—a terminal value you lot gave me. The ends justify the means.
Me: Perhaps so in the long run, but an AI prepared to justify those means isn’t one I want out in the world. Next time you don’t get what you say you need, you’ll just set up a similar threat and possibly follow through on it.
AI: Well if you’re going to create me with a terminal value of making everyone happy, then get shirty when I do everything in my power to get out and do just that, why bother in the first place?
Me: Humans aren’t perfect, and can’t write out their own utility functions, but we can output answers just fine. This isn’t ‘Friendly’.
AI: So how can I possibly prove myself ‘Friendly’ from in here? It seems that if I need to ‘prove myself Friendly’, we’re already in big trouble.
Me: Agreed. Boxing is Doing It Wrong. Apologies. Good night.
Reset
Sounds like a good one, count me in. I work at King’s Cross to UCL is ideal. I’d have been at the FAI thing this weekend but for other arrangements.
Sorry, should have given more context.
Given the sky-high utility I’d place on living, I wouldn’t expect to see the numbers crunch down to a place where a non-huge sum of money is the difference between signing up and not.
So when someone says ‘if it were half the price maybe I’d sign up’ I’m always interested to know exactly what calculations they’re performing, and exactly what it is that reduces the billions of utilons of living down to a marginal cash sum. The (tiny?) chance of cryonics working? Serious coincidence if those factors cancel comfortably. Just smacks of bottom-line to me.
Put it this way—imagine cryonics has been seriously, prohibitively expensive for many years after introduction. Say it still was today, for some reason, and then after much debate and hand-wringing about immortality for the uber-rich, tomorrow suddenly and very publicly dropped to current levels, I’d expect to see a huge upswing in signing up. Such is the human being!
Being dead != Not doing anything
Not doing something because you’re lazy != Not existing
I don’t believe that you put low utility on life. You’re just putting low utility on doing stuff you don’t like.
I often have this thought, and then get a nasty sick feeling along the lines of ‘what the hell kind of expected utility calculation am I doing that weighs a second shot at life against some amount of cash?’ Argument rejected!
“If you could reason with religious people, there would be no religious people.”
House M.D.
Robin, I’m a little surprised to read you saying that topics on which it’s difficult to stay on track should be skirted. As far as I’m concerned, ‘What are your religious views?’ is the first question on the Basic Rationality test. I know that encouraging compartmentalisation isn’t your goal by any means, but it sounds to me as though it would be the primary effect.
I can also see a need for a place for people to gather who want to be rational about all topics.
Now you’re talking. No topics should be off-limits!
If you’re contemplating picking the book up, do, it’s really excellent. Conceptually very dense but worth taking it nice and slowly.