I am not quite sure what this story is getting at. I’d guess it’s saying that we need to understand how human morality arises on a more fundamental (computable/programmable?) level before we can be sure that we can program AIs that will adhere to it, but the basis of human morality is (presumably) so much more complicated than the “prime numbers = good” presented here that the analogy is a bit strained. I may be interpreting this entirely wrongly.
ShardPhoenix
It seems to me that moral reasoning is only a computation in the sense that all human thought processes are computations. In other words, I’m not sure how helpful this is for AI purposes, other than a reminder that such a thing is possible.
I’m not sure it’s possible to extricate the complete underlying rules of human morality from all the other elements of human thought. I don’t think it’s necessarily impossible either, it just seems like we aren’t much closer to the solution.
You come across as very intelligent when you stick to your areas of expertise, like probability theory, AI and cognitive biases, but some of your more tangential stuff can seem a little naive. Compared to the other major poster on this blog, Robin, I’d say you come across as smarter but less “wise”, if that means anything to you. I’m not even a huge fan of the notion of “wisdom”, but if there’s something you’re missing, I think that’s it.
I thought I already knew all this, but this post has made me realize that I’ve still, deep down, been thinking as you describe—that the universe can’t be that unfair, and that the future isn’t really at risk. I guess the world seems like a bit scarier of a place now, but I’m sure I’ll go back to being distracted by day-to-day life in short order ;).
As for cryonics, I’m a little interested, but right now I have too many doubts about it and not enough spare money to go out and sign up immediately.
Why do people, including you apparently, always hide the price for this kind of thing? Market segmentation? Trying to get people to mentally commit before they find out how expensive it is? Maintaining a veneer of upper-class distaste for the crassness of money (or similarly, a “if you have to ask how much it is, you can’t afford it” type thing)?
Sorry, I just hate it when I’m looking at some product or service for sale on a webpage and they won’t tell you the damn price. I usually assume that it must be overpriced.
I think some of the above posts have a point—what’s the difference between “fun” and wireheading if what you’re doing for fun has no impact on the external world (because you’ve already set up your physical situation more-or-less as well as it can be given the laws of physics)?
Maybe if we can somehow reach other universes or some sort of (physical) “higher planes of existence” then there will always be something that “needs to be done”, but otherwise it seems that there will come a point where there is nothing to do but await the heat-death of the universe.
Honestly, reading these articles might be having the opposite of the intended effect on me, making me more nihilistic. You keep talking about “fun” and I keep wondering what the point is when the desire for fun isn’t really different from hunger or other evolution-driven desires that would presumably be (optionally?) eliminated in a friendly singularity scenario.
If you never get bored, do you care about or need to have fun?
@James:
Doesn’t the choice of a perfect external regulator amount to the same thing as directly imposing restrictions on yourself, thereby going back to the original problem? I suppose such a regulator, on indeed any stabilizing self-modification, could have the advantage of being publicly available and widely used, and therefore be well-tested and having thoroughly understood operations and consequences.
Another way to do it might be to create many copies of yourself (I’m assuming this scenario takes place inside a computer) and let majority (or 2/3s majority or etc) rule when it comes to “rescuing” copies that have made un-self-recoverable errors.
Anyway I suppose this is all somewhat beside the point since such a scenario was chosen as an example of what Eliezer expects a successful future to not look like.
@michael vassar:
So, are you saying that lying about your beliefs can be good because it allows you to freely believe some non-PC or otherwise unpopular idea (that your reason leads you to believe is the truth), without having to worry about the social consequences of being discovered to have such a belief?
I’m not sure if I agree with or not but it’s worth thinking about.
Jayson Virissimo:
To put my own spin on a famous quote, there are no “rights”. There is do, or do not.
I guess another way of thinking about it is that you decide on what terminal (possibly dynamic) state you want, then take measures to achieve that. Floating “rights” have no place.
(To clarify, “rights” can serve as a useful heuristic in practical discussions, but they’re not fundamental enough to figure into this kind of deep philosophical issue.)
Eliezer, what do you have against “Excession”? It’s been a while since I last read them, but I thought it was the 2nd best of the Culture books after “Use of Weapons”. I do agree that “Player of Games” is the best place to start though (I started with Consider Phlebas but found it a little dry).
Anyway, as for your actual point, I think it sounds reasonable at least on the surface, but I think considering this stuff too deeply may be putting the cart ahead of the horse somewhat when we’re not even very sure what causes consciousness in the first place, or what the details of its workings are, and therefore to what extent a non-conscious yet correctly working FAI is even possible or desirable.
The Book of the New Sun is/are my favourite book(s), but I’m not really sure what the quote you chose means. Perhaps, in the context of this blog at least, you could interpret it to support the idea of a person being a pattern of information, or even the idea that it doesn’t matter whether a mind-model is conscious or not.
“It’s premature optimization”
Thanks, I was trying to think of exactly how to describe this series of posts, and that phrase seems concise enough. It’s not that it’s not interesting in it’s own way, but even for an already pretty speculative blog, you’re really building castles on air here.
To make yet another analogy, you’re trying to build 100th floor of a huge house of cards here, when you’re not even sure what the 5th floor should be like yet (I was going to say the 1st floor, but I think you’ve at least gotten off to a decent start).
“I couldn’t disagree more. This kind of thinking is very important—not because we need to know RIGHT NOW in order to make some immediate and pressing policy decision, but because humans like to know where things are heading, what we are eventually aiming for. Suppose someone rejects cryonics or life extension research and opts for religion on the grounds that eternity in heaven will be “infinitely” good, but human life on earth, even technologically enhanced life, is necessarily mediocre. What can one say to such objections other than something like this series of posts?”
I’d say that if they’re willing to believe something just because it sounds nice rather than because it’s true, they’ve already given up on rationality. Is the goal to be rational and spread the truth, or to recruit people to the cause with wildly speculative optimism? I’d think just the idea of creating a super-intelligent AI that doesn’t destroy the world (if that’s even an issue—and I think there’s a good chance that it is) is a good incentive already—there’s no need to postulate a secular heaven that depends on so many things that we aren’t at all sure about yet.
“Any less than that, and your motivation to go on working can still be torpedoed by not being able to visualize a light at the end of the tunnel.”
I understand, and I’m hardly a perfect rationalist myself, but to me it seems that you don’t need to go so far as this to motivate people. You can easily come up with post-FAI scenarios that are much preferable to the modern day without having to speculate about hedon representation in the year 12000, when we don’t even know exactly what a hedon is or even what we really want in the long term. And if someone is convinced that post-singularity life can’t help but be horrible (or even is just a bit dubious about the whole scenario), then I doubt such “crazy sounding” ideas are going to make them listen.
On a side note, a lot of the stuff in this post seems very closely related to wireheading to me—not that I’m necessarily against that, but I know you are, and this post almost seems to be leading up to wireheading by another name.
“I cannot think of any strategy game where fog of war was a good idea”
Fog of War can generate interesting gameplay in both the need for active scouting and in the need to make decisions based on limited information (like in poker).
I feel that a lot of your discussion about Fun Theory is a bit too abstract to have an emotional appeal in terms of looking forward to the future. I think for at least some people (even smart, rational ones), it may be more effective to point out the possibility of more concrete, primitive, “monkey with a million bananas” type scenarios, even if those are not the most likely to actually occur.
Even if you know that the future probably won’t be specifically like that, you can imagine how good that would be in a more direct and emotionally compelling way, and then reason that a Fun Theory compatible future would be even better than that, even if you can’t visualize what it would be like so clearly.
I’d like to see some examples of types of large institutions that you believe should exist, but don’t due to lack of coordination.
I don’t see how the size of the universe makes any difference—isn’t it only the density of weird events that matters?
No matter how badly you’re being tricked, there still must be something there to be tricked. So Cogito Ergo Sum applies without any need for further logical justification than the fact that you thought of it (whether naturally or induced by the deceiving demon, and regardless of the validity any further reasoning or justification) in the first place.
A more generalized idea might be “something exists”.