you think that there’s something more dangerous than the human race, who can’t quite decide whether global warming should be mitigated against, trying to build an AI, where you have to get the answer pretty close to perfect first time, whilst also preventing all other groups from rushing to beat you and building uFAI?
I do think that we could build something more dangerous to civilization than the human race is at that time—but that seems like a rather obvious thing to think—and the fact that it is possible does not necessarily mean that it is likely.
Roko and I don’t see eye to eye on this issue. From my POV, we have had 50 years of unsuccessful attempts. That is not exactly “getting it right the first time”.
Google was not the first search engine, Microsoft was not the first OS maker—and Diffie–Hellman didn’t invent public key crypto.
Being first does not necessarily make players uncatchable—and there’s a selection process at work in the mean time, that weeds out certain classes of failures.
From my perspective, this is mainly a SIAI confusion. Because their funding is all oriented around the prospect of them saving the world from imminent danger, the execution of their mission apparently involves exaggerating the risks associated with that—which has the effect of stimulating funding from those who they convince that DOOM is imminent—and that the SIAI can help with averting in.
Humans will most likely get the machines they want—because people will build them to sell them—and because people won’t buy bad machines.
Tim, I think that what worries me is the “detailed reliable inheritance from human morals and meta-morals” bit. The worry that there will not be “detailed reliable inheritance from human morals and meta-morals” is robust to what specific way you think the future will go. Ems can break the inheritance. The first, second or fifteenth AGI system can break it. Intelligence enhancement gone wrong can break it. Any super-human “power” that doesn’t explicitly preserve it will break it.
All the examples you cite differ in the substantive dimension: the failure of attempt number 1 doesn’t preclude the success of attempt number two.
In the case of the future of humanity, the first failure to pass the physical representation of human morals and metamorals on to the next timeslice of the universe is game over.
The other thing to say is that there’s an important sense in which most modern creatures don’t value anything—except for their genetic heritage—which all living things necessarily value.
Contrast with a gold-atom maximiser. That values collections of pure gold atoms. It cares about something besides the survival of its genes (which obviously it also cares about—no genes, no gold). It strives to leave something of value behind.
Most modern organisms don’t leave anything behind—except for things that are inherited—genes and memes. Nothing that they expect to last for long, anyway. They keep dissipating energy gradients until everything is obliterated in high-entropy soup.
Those values are not very difficult to preserve—they are the default state.
If ecosystems cared about creating some sort of low-entropy state somewhere, then that property would take some effort to preserve (since it is vulnerable to invasion by creatures who use that low-entropy state as fuel). However, with the current situation, there aren’t really any values to preserve—except for those of the replicators concerned.
The idea has been called variously: goal system zero, god’s utility function, Shiva’s values.
Even the individual replicators aren’t really valued in themselves—except by themselves. There’s a parliament of genes, and any gene is expendable, on a majority vote. Genes are only potentially immortal. Over time, the representation of the original genes drops. Modern refactoring techniques will mean it will drop faster. There is not really a floor to the process—eventually, all may go.
I figure a fair amount of modern heritable information (such as morals) will not be lost. Civilization seems to be getting better at keeping and passing on records. You pretty-much have to hypothesize a breakdown of civilization for much of genuine value to be lost—an unprecedented and unlikely phenomenon.
However, I expect increasing amounts of it to be preserved mostly in history books and museums as time passes. Over time, that will probably include most DNA-based creatures—including humans.
Evolution is rather like a rope. Just as no strand in a rope goes from one end to the other, most genes don’t tend to do that either. That doesn’t mean the rope is weak, or that future creatures are not—partly—our descendants.
Museums have some paperclips in them. You have to imagine future museums as dynamic things that recreate and help to visualise the past—as well as preserving artefacts.
If you were an intelligence only cared about the number of paperclips in the universe, you would not build a museum to the past, because you could make more paperclips with the resources needed to create such a museum.
This is not some clever, convoluted argument. This is the same as saying that if you make your computer execute
10: GOTO 20
20: GOTO 10
then it won’t at any point realize the program is “stupid” and stop looping. You could even give the computer another program which is capable of proving that the first one is an infinite loop, but it won’t care, because its goal is to execute the first program.
That’s a different question—and one which is poorly specified:
If insufficient look-ahead is used, such an agent won’t bother to remember its history—prefering instead the gratification of instant paperclips.
On the other hand, if you set the look-ahead further out, it will. That’s because most intelligent agents are motivated to remember the past—since only by remembering the past can they predict the future.
Understanding the history of their own evolution may well help them to understand the possible forms of aliens—which might well help them avoid being obliterated by alien races (along with all the paper clips they have made so far). Important stuff—and well worth building a few museums over.
Remebering the past is thus actually an proximate goal for a wide range of agents. If you want to argue paperclip-loving agents won’t build museums, you need to be much more specific about which paperclip-loving agents you are talking about—because some of them will.
Once you understand this you should be able to see what nonsense the “value is fragile” post is.
At this point, I’m only saying this to ensure you don’t take any new LWers with you in your perennial folly, but your post has anthropomorphic optimism written all over it.
This has nothing to do with anthropomorphism or optimism—it is a common drive for intelligent agents to make records of their pasts—so that they can predict the consequences of their actions in the future.
Once information is lost, it is gone for good. If information might be valuable in the future, a wide range of agents will want to preserve it—to help them attain their future goals. These points do not seem particularly complicated.
I hope at least that you now realise that your “loop” analogy was wrong. You can’t just argue that paperclipping agents will not have preserving the past in museums as a proximate goal—since their ultimate goal involves making paperclips. There is a clear mechanism by which preserving their past in museums might help them attain that goal in the long term.
A wide class of paperclipping agents who are not suffering from temporal myopia should attempt to conquer the universe before wasting precious time and resources with making any paperclips. Once the universe is securely in their hands—then they can get on with making paperclips. Otherwise they run a considerable risk of aliens—who have not been so distracted with useless trivia—eating them, and their paperclips. They will realise that they are in an alien race—and so they will run.
I haven’t downvoted, but I assume it’s because he’s conflating ‘sees the value in storing some kinds of information’ with ‘will build museums’. Museums don’t seem to be particularly efficient forms of data-storage, to me.
Future “museums” may not look exactly like current ones—and sure—some information will be preserved in “libraries”—which may not look exactly like current ones either—and in other ways.
‘Museum’ and ‘library’ both imply, to me at least, that the data is being made available to people who might be interested in it. In the case of a paperclipper, that seems rather unlikely—why would it keep us around, instead of turning the planet into an uninhabitable supercomputer that can more quickly consider complex paperclip-maximization strategies? The information about what we were like might still exist, but probably in the form of the paperclipper’s ‘personal memory’ - and more likely than not, it’d be tagged as ‘exploitable weaknesses of squishy things’ rather than ‘good patterns to reproduce’, which isn’t very useful to us, to say the least.
I see. We have different connotations of the word, then. For me, a museum is just a place where objects of historical interest are stored.
When I talked about humans being “preserved mostly in history books and museums”—I was intending to conjour up an institution somewhat like the Jurassic park theme park. Or perhaps—looking further out—something like The Matrix. Not quite like the museum of natural history as it is today—but more like what it will turn into.
Regarding the utility of existence in a museum—it may be quite a bit better than not existing at all.
Regarding the reason for keeping objects of historical around—that is for much the same reason as we do today—to learn from them, and to preserve them for future generations to study. They may have better tools for analysing things with in the future. If the objects of study are destroyed, future tools will not be able to access them.
The direct ancestors are perhaps not the most illustrative examples but they will do. (I downvoted them on their perceived merit completely independently of the name.)
A pathetic example, IMHO. Those were perfectly reasonable comments attempting to dispel a poster’s inaccurate beliefs about the phenomenon in question.
Those were perfectly reasonable comments attempting to dispel a poster’s inaccurate beliefs about the phenomenon in question.
I disagree. That was what you were trying to do. You aren’t a troll, you are just quite bad at thinking so your posts often get downvoted. This reduces the likelyhood that you successfully propagate positions that are unfounded.
Yet another vague accusation that is not worth replying to.
I’m getting bored with this pointless flamewar. I can see that the mere breath of dissent causes the community to rise up in arms to nuke the dissenter. Great fun for you folk, I am sure—but I can’t see any good reason for me to play along with your childish games.
Yet another vague accusation that is not worth replying to.
It’s really not. Nothing good can come of this exchange, least of all to you.
I’m getting bored with this pointless flamewar.
People ask questions. People get answers. You included.
I can see that the mere breath of dissent causes the community to rise up in arms to nuke the dissenter.
No, you’re actually just wrong and absurdly so. Clippy doesn’t need you for his museum.
Great fun for you folk, I am sure
It isn’t wise for me to admit it but yes, there is a certain amount of satisfaction to be derived from direct social competition. I’m human, I’m male.
but I can’t see any good reason for me to play along with your childish games.
I agree (without, obviously, accepting the label). You are better off sticking to your position and finding ways to have your desired influence that avoid unwanted social penalties.
That is something we worry about from time to time, but in this case I think the downvotes are justified. Tim Tyler has been repeating a particular form of techno-optimism for quite a while, which is fine; it’s good to have contrarians around.
However, in the current thread, I don’t think he’s taking the critique seriously enough. It’s been pointed out that he’s essentially searching for reasons that even a Paperclipper would preserve everything of value to us, rather than just putting himself in Clippy’s place and really asking for the most efficient way to maximize paperclips. (In particular, preserving the fine details of a civilization, let alone actual minds from it, is really too wasteful if your goal is to be prepared for a wide array of possible alien species.)
I feel (and apparently, so do others) that he’s just replying with more arguments of the same kind as the ones we generally criticize, rather than finding other types of arguments or providing a case why anthropomorphic optimism doesn’t apply here.
In any case, thanks for the laugh line:
You went over some peoples heads.
My analysis of Tim Tyler in this thread isn’t very positive, but his replies seem quite clear to me; I’m frustrated on the meta-level rather than the object-level.
It’s been pointed out that he’s essentially searching for reasons that even a Paperclipper would preserve everything of value to us, rather than just putting himself in Clippy’s place and really asking for the most efficient way to maximize paperclips.
I don’t think that a paperclip maximiser would “preserve everything of value to us” in the first place. What I actually said at the beginning was:
TT: I figure a fair amount of modern heritable information (such as morals) will not be lost.
Not everything. Things are constantly being lost.
In particular, preserving the fine details of a civilization, let alone actual minds from it, is really too wasteful if your goal is to be prepared for a wide array of possible alien species.
TT: it is a common drive for intelligent agents to make records of their pasts—so that they can predict the consequences of their actions in the future.
We do, in fact, have detailed information about how much our own civilisation is prepared to spend on preserving its own history. We preserve many things which are millions of years old—and which take up far more resources than a human. For example, see how this museum dinosaur dwarfs the humans in the foreground. We have many such exhibits—and we are still a planet-bound civilisation. Our descendants seem likely to have access to much greater resources—and so may devote a larger quantity of absolute resources to museums.
So: that’s the basis of my estimate. What is the basis of your estimate?
The real dichotomy here is
“maximising evaluation function”
versus
“maximising probability of positive evaluation function”
In paperclip making, or better, the game of Othello/Reversi, there are choices like this:
80% chance of winning 60-0, versus
90% chance of winning 33-31.
The first maximises the winning, and is similar to a paperclip maker consuming the entire universe.
The second maximises the probability of succeeding, and is similar to a paperclip maker avoiding being annihilated by aliens or other unknown forces.
Mathematically, the first is similar to finding the shortest program in Kolmogorov Complexity, while the second is similar to integrating over programs.
So, friendly AI is surely of the second kind, while insane AI is of the first kind.
I guess you down-voters of me felt quite rational when doing so.
And this is precisely the reason I seldom post here, and only read a few posters that I know are rational from their own work on the net, not from what they write here:
There are too many fake rationalists here. The absence of any real arguments either way to my article above, is evidence of this.
My Othello/Reversi example above was easy to understand, and a very central problem in AI systems, so it should be of interest to real rationalists interested in AI, but there is only negative reaction instead, from people I guess have not even made a decent game playing AI, but nevertheless have strong opinions on how they must be.
So, for getting intelligent rational arguments on AI, this community is useless, as opposed to Yudkowsky, Schmidhuber, Hansen, Tyler, etc. which has shown on their own sites that they have something to contribute.
To get real results in AI and rationality, I do my own math and science.
Your Othello Reversi example is fundamentally flawed, but it may not seem like it unless you realize that at LW the tradition is to say that utility is linear in paperclips to Clippy. That may be our fault, but there’s your explanation. “Winning 60-0”, to us using our jargon, is equivalent to one paperclip, not 60. And “winning 33-31″ is also equivalent to one paperclip, not 33. (or they’re both equivalent to x paperclips, whatever)
So when I read your example, I read it as “80% chance of 1 paperclip, or 90% chance of 1 paperclip”.
I’m sure it’s very irritating to have your statement miscommunicated because of a jargon difference (paperclip = utility rather than f(paperclip) = utility)! I encourage you to post anyway, and begin with the assumption that we misunderstand you rather than the assumption that we are “fake rationalists”, but realize that in the current environment (unfortunately or not, but there it is) the burden of communication is on the poster.
While most of this of this seems sensible, I don’t understand how your last sentence follows. I have heard similar strategies suggested to reduce the probability of paperclipping, but it seems like if we actually succeed in producing a true friendly AI, the quantity it tries to maximize (expected winning, P(winning), or something else) will depend on how we evaluate outcomes.
This made some sense to me, at least to the point where I’d expect an intelligent refutation from disagreers, and seems posted in good faith. What am I missing about the voting system? Or about this post.
Your use of “get together” brings to mind some sort of Less Wrong cabal who gathered to make a decision. This is of course the opposite of the truth, which is that each downvote is the result of someone reading the thread and deciding to downvote the comment. They’re not necessarily uncorrelated, but “get together” is completely the wrong way to think about how these downvotes occur.
Actually, that’s what I was meaning to evoke. I read his recent comments, and while I didn’t agree with all of them, didn’t find them to be in bad faith. I found it odd that so many of them would be at −3, and wondered if I missed something.
In seriousness, why would you deliberately evoke a hypothesis that you know is wildly unrealistic? Surely whatever the real reasons for the downvoting pattern are, they are relevant to your enquiry?
Perhaps “cabal who gathered to make a decision [to downvote]” is an overly ominous image.
However, we’ve seen cases where every one of someone’s comments has been downvoted in a short span of time, which is clearly not the typical reason for a downvoting.
It is possible the first downvote tends to attract further downvotes (by priming, for example), but an equally parsimonious explanation is that there are several people refreshing the comments page at a time and a subset of them dislike the content independently.
But you can still be very confident that actual collusion wasn’t involved, so you shouldn’t be talking as if it might have been.
EDIT: as always I’m keen to know why the downvote—thanks! My current theory is that they come across as hostile, which they weren’t meant to, but I’d value better data than my guesses.
One hypothesis is that people can’t offer counter-arguments—but they don’t like the conclusions—because they are contrary to the received wisdom. That creates cognitive dissonance in them—and they have to find an outlet.
Really, never write comments proffering self-aggrandising explanations of why your comments are being badly received. You are way too smart and thoughtful to go all green ink on us like this.
It may amuse you to know, I am told by my lawyer partners that a really astonishing proportion of crazy people who write to you really do use green ink. It seems astonishing to imagine that there really could be a correlation between sanity and favoured ink colour.
I should be astonished, but I’m not. A statistically unlikely proportion of the autistics I know have a strong preference for the color purple (and I don’t think I know any NTs with that preference), so the idea that color preference is a function of neurotype doesn’t seem too odd to me.
Incidentally, I hope you don’t mean the “self-aggrandising” / “green ink” comments literally!
Disagreeing with majorities is often a bad sign. Delusional individuals may create “green ink” explanations of why others are foolish enough to disagree with them. However, critics may also find themselves disagreeing with majorities—for example when in the company of the associates of those being criticised. That is fairly often my role here. I am someone not in the thrall of the prevailing reality distortion field. Under those circumstances disagreements do not have the same significance.
Disagreeing with majorities is often a bad sign. Delusional individuals may create “green ink” explanations of why others are foolish enough to disagree with them. However, critics may also find themselves disagreeing with majorities—for example when in the company of the associates of those being criticised. That is fairly often my role here. I am someone not in the thrall of the prevailing reality distortion field. Under those circumstances disagreements do not have the same significance.
The indicated sections are green ink—claims which are easy to make regardless of the rectitude of your opinion, and which therefore are made by fools with higher-than-normal frequency.
Arguing that fools make statement X with greater-than-average frequency is a rather feeble argument that someone making statement X is a fool. I am not sure why you are even bothering to present it.
Well, the first bold section is a true, general and relevant statement.
I won’t say what my estimate of a person’s rationality would be, given only the information that they had written the second bold section somewhere on the internet; but it wouldn’t be 100% crank, either.
Well, the first bold section is a true, general and relevant statement.
That doesn’t mean the ink isn’t green. In this particular case, he is persistently claiming that his remarks are being attacked due to various sorts of biases on the parts of those reading it, and he is doing so:
without detailed evidence, and
instead of either (a) clarifying his remarks or (b) dropping the subject.
That’s green ink.
Edited for pronouns.
Edited for pronouns again, properly this time. Curse you, Picornaviridae Rhinovirus!
Someone disagreeing with other people and explaining why he thinks they are wrong is not “green ink”—unless that individual is behaving in a crazy fashion.
I don’t think anyone has any evidence that my behaviour is anything other than rational and sane in this case. At any rate, so far no such evidence has been presented AFAICS. So: I think “green ink” is a fairly clear mis-characterisation.
No, green ink covers a much wider span of writing than that. And honestly, no matter what disagreements you find yourself having with a group of people, and this would include circumstances where you were the only rationalist in a room full of crystal healers, you should never find yourself uttering the phrase “I am someone not in the thrall of the prevailing reality distortion field”.
I think that is just a difference of personalities.
If I am in a region where there’s a reality distortion field in action, I don’t necessarily avoid pointing that out for the sake of everyone’s feelings—or for some other reason.
That would let the participants continue in their trance—and that might not be good for them, or others they interact with.
You can point something out, but it is an act of petty sabotage to repeat the same statements over and over again with no apparent effect but to irritation of the public. Even if you are in fact right, and the other guys are lunatics.
I have nothing to say at the moment regarding your actual argumentation upthread—what I am criticizing is your reaction to the downvoting et seq. I don’t care what you call it: stop.
You are acting as if you are obviously correct. That is true far less often than you suppose. If you are not obviously correct, retaliating against the people who attacked you is counterproductive. Better is to expand your analysis or drop the subject altogether.
You choose to perpetuate the cycle. OK, then, I will drop out first. This thread has been dragged into the gutter—and I am not interested in following it down there. Bye.
Confused about pronouns even after your edit: who is “you”? My remarks aren’t being downvoted, so I assume “you” doesn’t mean me. And you used “he” to refer to Tim Tyler, so I assume “you” doesn’t mean him.
Yes I am mentally ill.
I had disclosed this on my own website.
I am not delusional.
For the disabled, carelessly throwing around psychiatric terms when you are not a practicing psychiatrist is foul and abusive.
I have very little formal education which is common among the mentally disabled.
I eagerly await your hierarchical models for the complexity of “AI”, and your elegant algorithms for its implementation.
I imagine your benefactors are also eagerly waiting.
Don’t disappoint them or they may pull their wasted funding.
I will continue hobbling along my “delusional” way, then after a while I will probably apply for the funding they are wasting on this place.
Of course the algorithm is the key, and that I will not be publishing.
Best of luck.
May the best crazy win.
Yes.
To me it seems like all arguments for the importance of friendly AI are based on the assumption that its moral evaluation function must be correct, or it will necessarily become evil or insane, due over optimization of some weird aspect.
However, with uncertainty in the system, as limited knowledge of the past, or as uncertainty in what the evaluation function is, optimization should take this into account, and make strategies to keep its options open. In the paperclip example, this would be avoiding making people into paperclips because it suspects that the paperclips might be for people.
Mathematically, an AI going evil insane corresponds to it seeking the most probable optimization, while doing multiple strategies corresponds to it integrating the probabilities over different outcomes.
I think the usual example assumes that the machine assigns a low probability to the hypothesis that paperclips are not the only valuable thing—because of how it was programmed.
Reaching, though: the dark ages were confined to Western Europe—and something like the Library of Alexandria couldn’t happen these days—there are too many libraries.
you think that there’s something more dangerous than the human race, who can’t quite decide whether global warming should be mitigated against, trying to build an AI, where you have to get the answer pretty close to perfect first time, whilst also preventing all other groups from rushing to beat you and building uFAI?
I’m not sure that is a proper sentence.
I do think that we could build something more dangerous to civilization than the human race is at that time—but that seems like a rather obvious thing to think—and the fact that it is possible does not necessarily mean that it is likely.
Key Noun phrase: the human race,..., trying to build an AI,
Then: {description of difficulty of said activity}
I’m not sure it’s proper either, but I’m sure you misparsed it.
Yay, that really helped!
Roko and I don’t see eye to eye on this issue. From my POV, we have had 50 years of unsuccessful attempts. That is not exactly “getting it right the first time”.
Google was not the first search engine, Microsoft was not the first OS maker—and Diffie–Hellman didn’t invent public key crypto.
Being first does not necessarily make players uncatchable—and there’s a selection process at work in the mean time, that weeds out certain classes of failures.
From my perspective, this is mainly a SIAI confusion. Because their funding is all oriented around the prospect of them saving the world from imminent danger, the execution of their mission apparently involves exaggerating the risks associated with that—which has the effect of stimulating funding from those who they convince that DOOM is imminent—and that the SIAI can help with averting in.
Humans will most likely get the machines they want—because people will build them to sell them—and because people won’t buy bad machines.
Tim, I think that what worries me is the “detailed reliable inheritance from human morals and meta-morals” bit. The worry that there will not be “detailed reliable inheritance from human morals and meta-morals” is robust to what specific way you think the future will go. Ems can break the inheritance. The first, second or fifteenth AGI system can break it. Intelligence enhancement gone wrong can break it. Any super-human “power” that doesn’t explicitly preserve it will break it.
All the examples you cite differ in the substantive dimension: the failure of attempt number 1 doesn’t preclude the success of attempt number two.
In the case of the future of humanity, the first failure to pass the physical representation of human morals and metamorals on to the next timeslice of the universe is game over.
The other thing to say is that there’s an important sense in which most modern creatures don’t value anything—except for their genetic heritage—which all living things necessarily value.
Contrast with a gold-atom maximiser. That values collections of pure gold atoms. It cares about something besides the survival of its genes (which obviously it also cares about—no genes, no gold). It strives to leave something of value behind.
Most modern organisms don’t leave anything behind—except for things that are inherited—genes and memes. Nothing that they expect to last for long, anyway. They keep dissipating energy gradients until everything is obliterated in high-entropy soup.
Those values are not very difficult to preserve—they are the default state.
If ecosystems cared about creating some sort of low-entropy state somewhere, then that property would take some effort to preserve (since it is vulnerable to invasion by creatures who use that low-entropy state as fuel). However, with the current situation, there aren’t really any values to preserve—except for those of the replicators concerned.
The idea has been called variously: goal system zero, god’s utility function, Shiva’s values.
Even the individual replicators aren’t really valued in themselves—except by themselves. There’s a parliament of genes, and any gene is expendable, on a majority vote. Genes are only potentially immortal. Over time, the representation of the original genes drops. Modern refactoring techniques will mean it will drop faster. There is not really a floor to the process—eventually, all may go.
I figure a fair amount of modern heritable information (such as morals) will not be lost. Civilization seems to be getting better at keeping and passing on records. You pretty-much have to hypothesize a breakdown of civilization for much of genuine value to be lost—an unprecedented and unlikely phenomenon.
However, I expect increasing amounts of it to be preserved mostly in history books and museums as time passes. Over time, that will probably include most DNA-based creatures—including humans.
Evolution is rather like a rope. Just as no strand in a rope goes from one end to the other, most genes don’t tend to do that either. That doesn’t mean the rope is weak, or that future creatures are not—partly—our descendants.
And how do museums lead to more paperclips?
Museums have some paperclips in them. You have to imagine future museums as dynamic things that recreate and help to visualise the past—as well as preserving artefacts.
If you were an intelligence only cared about the number of paperclips in the universe, you would not build a museum to the past, because you could make more paperclips with the resources needed to create such a museum.
This is not some clever, convoluted argument. This is the same as saying that if you make your computer execute
10: GOTO 20
20: GOTO 10
then it won’t at any point realize the program is “stupid” and stop looping. You could even give the computer another program which is capable of proving that the first one is an infinite loop, but it won’t care, because its goal is to execute the first program.
That’s a different question—and one which is poorly specified:
If insufficient look-ahead is used, such an agent won’t bother to remember its history—prefering instead the gratification of instant paperclips.
On the other hand, if you set the look-ahead further out, it will. That’s because most intelligent agents are motivated to remember the past—since only by remembering the past can they predict the future.
Understanding the history of their own evolution may well help them to understand the possible forms of aliens—which might well help them avoid being obliterated by alien races (along with all the paper clips they have made so far). Important stuff—and well worth building a few museums over.
Remebering the past is thus actually an proximate goal for a wide range of agents. If you want to argue paperclip-loving agents won’t build museums, you need to be much more specific about which paperclip-loving agents you are talking about—because some of them will.
Once you understand this you should be able to see what nonsense the “value is fragile” post is.
At this point, I’m only saying this to ensure you don’t take any new LWers with you in your perennial folly, but your post has anthropomorphic optimism written all over it.
This has nothing to do with anthropomorphism or optimism—it is a common drive for intelligent agents to make records of their pasts—so that they can predict the consequences of their actions in the future.
Once information is lost, it is gone for good. If information might be valuable in the future, a wide range of agents will want to preserve it—to help them attain their future goals. These points do not seem particularly complicated.
I hope at least that you now realise that your “loop” analogy was wrong. You can’t just argue that paperclipping agents will not have preserving the past in museums as a proximate goal—since their ultimate goal involves making paperclips. There is a clear mechanism by which preserving their past in museums might help them attain that goal in the long term.
A wide class of paperclipping agents who are not suffering from temporal myopia should attempt to conquer the universe before wasting precious time and resources with making any paperclips. Once the universe is securely in their hands—then they can get on with making paperclips. Otherwise they run a considerable risk of aliens—who have not been so distracted with useless trivia—eating them, and their paperclips. They will realise that they are in an alien race—and so they will run.
Did you make some huge transgression that I missed that is causing people to get together and downvote your comments?
Edit: My question has now been answered.
I haven’t downvoted, but I assume it’s because he’s conflating ‘sees the value in storing some kinds of information’ with ‘will build museums’. Museums don’t seem to be particularly efficient forms of data-storage, to me.
Future “museums” may not look exactly like current ones—and sure—some information will be preserved in “libraries”—which may not look exactly like current ones either—and in other ways.
‘Museum’ and ‘library’ both imply, to me at least, that the data is being made available to people who might be interested in it. In the case of a paperclipper, that seems rather unlikely—why would it keep us around, instead of turning the planet into an uninhabitable supercomputer that can more quickly consider complex paperclip-maximization strategies? The information about what we were like might still exist, but probably in the form of the paperclipper’s ‘personal memory’ - and more likely than not, it’d be tagged as ‘exploitable weaknesses of squishy things’ rather than ‘good patterns to reproduce’, which isn’t very useful to us, to say the least.
I see. We have different connotations of the word, then. For me, a museum is just a place where objects of historical interest are stored.
When I talked about humans being “preserved mostly in history books and museums”—I was intending to conjour up an institution somewhat like the Jurassic park theme park. Or perhaps—looking further out—something like The Matrix. Not quite like the museum of natural history as it is today—but more like what it will turn into.
Regarding the utility of existence in a museum—it may be quite a bit better than not existing at all.
Regarding the reason for keeping objects of historical around—that is for much the same reason as we do today—to learn from them, and to preserve them for future generations to study. They may have better tools for analysing things with in the future. If the objects of study are destroyed, future tools will not be able to access them.
Not really, just lots of little ones involving the misuse of almost valid ideas. They get distracting.
That’s pretty vague. Care to point to something specific?
The direct ancestors are perhaps not the most illustrative examples but they will do. (I downvoted them on their perceived merit completely independently of the name.)
A pathetic example, IMHO. Those were perfectly reasonable comments attempting to dispel a poster’s inaccurate beliefs about the phenomenon in question.
Feel free to provide a better one.
I disagree. That was what you were trying to do. You aren’t a troll, you are just quite bad at thinking so your posts often get downvoted. This reduces the likelyhood that you successfully propagate positions that are unfounded.
Clippy museums. Right.
Yet another vague accusation that is not worth replying to.
I’m getting bored with this pointless flamewar. I can see that the mere breath of dissent causes the community to rise up in arms to nuke the dissenter. Great fun for you folk, I am sure—but I can’t see any good reason for me to play along with your childish games.
It’s really not. Nothing good can come of this exchange, least of all to you.
People ask questions. People get answers. You included.
No, you’re actually just wrong and absurdly so. Clippy doesn’t need you for his museum.
It isn’t wise for me to admit it but yes, there is a certain amount of satisfaction to be derived from direct social competition. I’m human, I’m male.
I agree (without, obviously, accepting the label). You are better off sticking to your position and finding ways to have your desired influence that avoid unwanted social penalties.
Upvoted for honesty. It’s far better to be aware of it than not to be.
Anyhow, I think you don’t really need to add anything more at this point; the thread looks properly wrapped up to me.
You got voted down because you were rational. You went over some peoples heads.
These are popularity points, not rationality points.
That is something we worry about from time to time, but in this case I think the downvotes are justified. Tim Tyler has been repeating a particular form of techno-optimism for quite a while, which is fine; it’s good to have contrarians around.
However, in the current thread, I don’t think he’s taking the critique seriously enough. It’s been pointed out that he’s essentially searching for reasons that even a Paperclipper would preserve everything of value to us, rather than just putting himself in Clippy’s place and really asking for the most efficient way to maximize paperclips. (In particular, preserving the fine details of a civilization, let alone actual minds from it, is really too wasteful if your goal is to be prepared for a wide array of possible alien species.)
I feel (and apparently, so do others) that he’s just replying with more arguments of the same kind as the ones we generally criticize, rather than finding other types of arguments or providing a case why anthropomorphic optimism doesn’t apply here.
In any case, thanks for the laugh line:
My analysis of Tim Tyler in this thread isn’t very positive, but his replies seem quite clear to me; I’m frustrated on the meta-level rather than the object-level.
I don’t think that a paperclip maximiser would “preserve everything of value to us” in the first place. What I actually said at the beginning was:
Not everything. Things are constantly being lost.
What I said here was:
We do, in fact, have detailed information about how much our own civilisation is prepared to spend on preserving its own history. We preserve many things which are millions of years old—and which take up far more resources than a human. For example, see how this museum dinosaur dwarfs the humans in the foreground. We have many such exhibits—and we are still a planet-bound civilisation. Our descendants seem likely to have access to much greater resources—and so may devote a larger quantity of absolute resources to museums.
So: that’s the basis of my estimate. What is the basis of your estimate?
I agree with your criticism, but I doubt that good will come of replying to a comment like the one you’re replying to here, I’m afraid.
Fair enough; I should have replied to Tim directly, but couldn’t pass up the laugh-line bit.
The real dichotomy here is “maximising evaluation function” versus “maximising probability of positive evaluation function”
In paperclip making, or better, the game of Othello/Reversi, there are choices like this:
80% chance of winning 60-0, versus 90% chance of winning 33-31.
The first maximises the winning, and is similar to a paperclip maker consuming the entire universe. The second maximises the probability of succeeding, and is similar to a paperclip maker avoiding being annihilated by aliens or other unknown forces.
Mathematically, the first is similar to finding the shortest program in Kolmogorov Complexity, while the second is similar to integrating over programs.
So, friendly AI is surely of the second kind, while insane AI is of the first kind.
I guess you down-voters of me felt quite rational when doing so.
And this is precisely the reason I seldom post here, and only read a few posters that I know are rational from their own work on the net, not from what they write here:
There are too many fake rationalists here. The absence of any real arguments either way to my article above, is evidence of this.
My Othello/Reversi example above was easy to understand, and a very central problem in AI systems, so it should be of interest to real rationalists interested in AI, but there is only negative reaction instead, from people I guess have not even made a decent game playing AI, but nevertheless have strong opinions on how they must be.
So, for getting intelligent rational arguments on AI, this community is useless, as opposed to Yudkowsky, Schmidhuber, Hansen, Tyler, etc. which has shown on their own sites that they have something to contribute.
To get real results in AI and rationality, I do my own math and science.
Your Othello Reversi example is fundamentally flawed, but it may not seem like it unless you realize that at LW the tradition is to say that utility is linear in paperclips to Clippy. That may be our fault, but there’s your explanation. “Winning 60-0”, to us using our jargon, is equivalent to one paperclip, not 60. And “winning 33-31″ is also equivalent to one paperclip, not 33. (or they’re both equivalent to x paperclips, whatever)
So when I read your example, I read it as “80% chance of 1 paperclip, or 90% chance of 1 paperclip”.
I’m sure it’s very irritating to have your statement miscommunicated because of a jargon difference (paperclip = utility rather than f(paperclip) = utility)! I encourage you to post anyway, and begin with the assumption that we misunderstand you rather than the assumption that we are “fake rationalists”, but realize that in the current environment (unfortunately or not, but there it is) the burden of communication is on the poster.
While most of this of this seems sensible, I don’t understand how your last sentence follows. I have heard similar strategies suggested to reduce the probability of paperclipping, but it seems like if we actually succeed in producing a true friendly AI, the quantity it tries to maximize (expected winning, P(winning), or something else) will depend on how we evaluate outcomes.
This made some sense to me, at least to the point where I’d expect an intelligent refutation from disagreers, and seems posted in good faith. What am I missing about the voting system? Or about this post.
Your use of “get together” brings to mind some sort of Less Wrong cabal who gathered to make a decision. This is of course the opposite of the truth, which is that each downvote is the result of someone reading the thread and deciding to downvote the comment. They’re not necessarily uncorrelated, but “get together” is completely the wrong way to think about how these downvotes occur.
Actually, that’s what I was meaning to evoke. I read his recent comments, and while I didn’t agree with all of them, didn’t find them to be in bad faith. I found it odd that so many of them would be at −3, and wondered if I missed something.
In seriousness, why would you deliberately evoke a hypothesis that you know is wildly unrealistic? Surely whatever the real reasons for the downvoting pattern are, they are relevant to your enquiry?
Perhaps “cabal who gathered to make a decision [to downvote]” is an overly ominous image.
However, we’ve seen cases where every one of someone’s comments has been downvoted in a short span of time, which is clearly not the typical reason for a downvoting.
That’s the kind of thing I was asking about.
It is possible the first downvote tends to attract further downvotes (by priming, for example), but an equally parsimonious explanation is that there are several people refreshing the comments page at a time and a subset of them dislike the content independently.
But you can still be very confident that actual collusion wasn’t involved, so you shouldn’t be talking as if it might have been.
EDIT: as always I’m keen to know why the downvote—thanks! My current theory is that they come across as hostile, which they weren’t meant to, but I’d value better data than my guesses.
One hypothesis is that people can’t offer counter-arguments—but they don’t like the conclusions—because they are contrary to the received wisdom. That creates cognitive dissonance in them—and they have to find an outlet.
Really, never write comments proffering self-aggrandising explanations of why your comments are being badly received. You are way too smart and thoughtful to go all green ink on us like this.
Hah! I had to look up http://en.wikipedia.org/wiki/Green_ink
I like my comment—and think it shows my sense of humour. If you were among those who were not amused, then sorry! ;-)
I do usually keep off karma-related sub-threads. They are mostly noise to me. However, here, I was asked a direct question.
Anyway, if people here who disagree with me can’t be bothered to argue, I don’t see how will they ever learn anything ;-) <--
It may amuse you to know, I am told by my lawyer partners that a really astonishing proportion of crazy people who write to you really do use green ink. It seems astonishing to imagine that there really could be a correlation between sanity and favoured ink colour.
Yikes, did anyone else notice the amount of green on this site?
I was just noticing that myself. Withdrawing my request to change the color scheme.
We all write in black, though. :P
Mostly.
I should be astonished, but I’m not. A statistically unlikely proportion of the autistics I know have a strong preference for the color purple (and I don’t think I know any NTs with that preference), so the idea that color preference is a function of neurotype doesn’t seem too odd to me.
As an Asperger’s person who loves purple (and dislikes green) this thread is quite amusing.
This is the first place I have ever seen anyone say that green ink writers use actual green ink. Added to RW!
Incidentally, I hope you don’t mean the “self-aggrandising” / “green ink” comments literally!
Disagreeing with majorities is often a bad sign. Delusional individuals may create “green ink” explanations of why others are foolish enough to disagree with them. However, critics may also find themselves disagreeing with majorities—for example when in the company of the associates of those being criticised. That is fairly often my role here. I am someone not in the thrall of the prevailing reality distortion field. Under those circumstances disagreements do not have the same significance.
The indicated sections are green ink—claims which are easy to make regardless of the rectitude of your opinion, and which therefore are made by fools with higher-than-normal frequency.
I recommend you check with http://en.wikipedia.org/wiki/Green_ink
Arguing that fools make statement X with greater-than-average frequency is a rather feeble argument that someone making statement X is a fool. I am not sure why you are even bothering to present it.
Well, the first bold section is a true, general and relevant statement.
I won’t say what my estimate of a person’s rationality would be, given only the information that they had written the second bold section somewhere on the internet; but it wouldn’t be 100% crank, either.
That doesn’t mean the ink isn’t green. In this particular case, he is persistently claiming that his remarks are being attacked due to various sorts of biases on the parts of those reading it, and he is doing so:
without detailed evidence, and
instead of either (a) clarifying his remarks or (b) dropping the subject.
That’s green ink.
Edited for pronouns.
Edited for pronouns again, properly this time. Curse you, Picornaviridae Rhinovirus!
I think http://en.wikipedia.org/wiki/Green_ink makes it pretty clear that green ink is barely-coherent rambling coming from nutcases.
Someone disagreeing with other people and explaining why he thinks they are wrong is not “green ink”—unless that individual is behaving in a crazy fashion.
I don’t think anyone has any evidence that my behaviour is anything other than rational and sane in this case. At any rate, so far no such evidence has been presented AFAICS. So: I think “green ink” is a fairly clear mis-characterisation.
No, green ink covers a much wider span of writing than that. And honestly, no matter what disagreements you find yourself having with a group of people, and this would include circumstances where you were the only rationalist in a room full of crystal healers, you should never find yourself uttering the phrase “I am someone not in the thrall of the prevailing reality distortion field”.
Um—why not?
I think that is just a difference of personalities.
If I am in a region where there’s a reality distortion field in action, I don’t necessarily avoid pointing that out for the sake of everyone’s feelings—or for some other reason.
That would let the participants continue in their trance—and that might not be good for them, or others they interact with.
You can point something out, but it is an act of petty sabotage to repeat the same statements over and over again with no apparent effect but to irritation of the public. Even if you are in fact right, and the other guys are lunatics.
Hi, Vladimir! I don’t know what you are talking about, why you are bothering to say it—and nor do I much care.
I have nothing to say at the moment regarding your actual argumentation upthread—what I am criticizing is your reaction to the downvoting et seq. I don’t care what you call it: stop.
What was wrong with that?
Someone asked me why I was being downvoted.
I gave them my best hypothesis.
You want me to lie? You think my hypothesis was inaccurate? What exactly is the problem you have?
On the other hand, if you genuinely want me to stop defending my actions, it would help if people first stop attacking them—perhaps starting with you.
You are acting as if you are obviously correct. That is true far less often than you suppose. If you are not obviously correct, retaliating against the people who attacked you is counterproductive. Better is to expand your analysis or drop the subject altogether.
You choose to perpetuate the cycle. OK, then, I will drop out first. This thread has been dragged into the gutter—and I am not interested in following it down there. Bye.
Confused about pronouns even after your edit: who is “you”? My remarks aren’t being downvoted, so I assume “you” doesn’t mean me. And you used “he” to refer to Tim Tyler, so I assume “you” doesn’t mean him.
I apologize, I r dum.
Yes I am mentally ill. I had disclosed this on my own website. I am not delusional. For the disabled, carelessly throwing around psychiatric terms when you are not a practicing psychiatrist is foul and abusive. I have very little formal education which is common among the mentally disabled. I eagerly await your hierarchical models for the complexity of “AI”, and your elegant algorithms for its implementation. I imagine your benefactors are also eagerly waiting. Don’t disappoint them or they may pull their wasted funding. I will continue hobbling along my “delusional” way, then after a while I will probably apply for the funding they are wasting on this place. Of course the algorithm is the key, and that I will not be publishing. Best of luck. May the best crazy win.
Yes. To me it seems like all arguments for the importance of friendly AI are based on the assumption that its moral evaluation function must be correct, or it will necessarily become evil or insane, due over optimization of some weird aspect.
However, with uncertainty in the system, as limited knowledge of the past, or as uncertainty in what the evaluation function is, optimization should take this into account, and make strategies to keep its options open. In the paperclip example, this would be avoiding making people into paperclips because it suspects that the paperclips might be for people.
Mathematically, an AI going evil insane corresponds to it seeking the most probable optimization, while doing multiple strategies corresponds to it integrating the probabilities over different outcomes.
I think the usual example assumes that the machine assigns a low probability to the hypothesis that paperclips are not the only valuable thing—because of how it was programmed.
Possible precedents: the Library of Alexandria and the Dark Ages.
Reaching, though: the dark ages were confined to Western Europe—and something like the Library of Alexandria couldn’t happen these days—there are too many libraries.