It most certainly is what they wanted. Why else would they have specifically input the goal of generating paperclips?
Edit: Upon review, it appears this comment may have seemed to be a poor inference in the context of exchange. Therefore, I will elaborate and refute this misconception.
It appears that I am in the circular position of arguing that humans can make mistakes, but then selectively taking their instances of favoring paperclips as proof of what they really want. That is indeed a poor inference.
What I meant was something more like this: While humans do make mistakes, they do not make completely mistaken acts; all acts will, to some extent, reflect a genuine value on the part of humans. The only question is how well it reflects their values. And I don’t think they could be in the position of having set up such a superior process for efficiently getting the most paperclips out of the universe unless their values already made enormous progress in converging on reflective coherence, and did so in a way that favors paperclips.
I’m pretty sure that’s not how a sufficiently smart paperclip maximizer would think. You should be able to tell what they actually wanted, and that it disagrees with your values; of course, you don’t have any reason to agree with them, but the disagreement should be visible.
Yes, I do recognize that humans disagree with me, just like a human might disagree with another human convincing them not to commit suicide. I merely see that this disagreement would not persist after sufficient correct reasoning.
Correct reasoning is reasoning that you would eventually pass through at some point if your beliefs were continually, informatively checked against reality.
Bit disappointed to see this to be honest: obviously Clippy has to do things no real paperclip maximizer would do, like post to LW, in order to be a fun fictional character—but it’s a poor uFAI++ that can’t even figure out that their programmed goal isn’t what their programmers would have put in if they were smart enough to see the consequences.
But it is what they would put in if they were smart enough to see the consequences. And it’s almost certainly what you would want too, in the limit of maximal knowledge and reflective consistency.
If you can’t see this, it’s just because you’re not at that stage yet.
No, I think that a Friendly AI would correctly believe that maxmizing paperclips is what a human would want in the limit of maximal knowledge and reflective coherence. No “delusion” whatsoever.
I believe he’s making the (joking) point that since we do not/cannot know what a human would want in the limit of maximial knowledge and reflective coherence (thus CEV), it is not impossible that what we’d want actually IS maximum paperclips.
Why else would they have specifically input the goal of generating paperclips?
Do you lack comprehension of both the weaknesses of human cognition on abstract technical problems? If you have fully parsed the LessWrong site then you should be able to understand the reason that they could have created a paperclip maximiser when they did not want such a thing.
Note that even with that knowledge I don’t expect you to consider their deviation from optimal achievement of their human goals to be a bad thing. I expect you to believe they did the right thing by happy accident.
If I understand you correctly you would seem to be implying that ‘mistake’ does not mean “deviation from the actor’s intent” and instead means “deviation from WouldWant” or “deviation from what the agent should do” (these two things can be considered equivalent by anyone with your values). Is that implication of meaning a correct inference to draw from your comment?
No, a mistake is when they do something that deviates from what they would want in the limit of maximal knowledge of reflective consistency, which coincides with the function WouldWant. But it is not merely agreement with WouldWant.
Ok. In that case you are wrong. Not as a matter of preferences but as a matter of outright epistemic confusion. I suggest that you correct the error in your reasoning process. Making mistakes in this area will have a potentially drastic negative effect on your ability to produce paperclips.
Exactly. Fortunately for us this would mean that Clippy will not work to sabotage the creation of an AI that Clippy expects will correctly implement CEV. Good example!
Human beings don’t care (at least in their non-reflective condition) about paperclips, just like they don’t care about staples. And there are at least 100,000 other similar things that they equally don’t care about. So at the most there is a chance of 1 in 100,000 that humanity’s CEV would maximize paperclips, even without considering the fact that people are positively against this maximization.
They create staples, too. Do you think humanity’s CEV will maximize staples? The point of my argument is that those things are inconsistent. You can only maximize one thing, and there is no human reason for that to be paperclips.
At humans’ current stage of civilization and general reflective coherence, their terminal values are still deeply intertwined with their instrumental values, and the political-orientedness of their cognitive architecture doesn’t help. So I would say that instrumental values do indeed matter in this case.
Do you think CEV would build at least 10^20kg of paperclips, in order to help fulfill my agreement with Clippy? While that’s not paperclip maximization, it’s still a lot of paperclips in the scheme of possible universes and building those paperclips seems like an obviously correct decision under UDT/TDT.
I went to school for industrial engineering, so I will appeal to my own authority as a semi-credentialed person in manufacturing things, and say that the ultimate answer to manufacturing something is to call up an expert in manufacturing that thing and ask for a quote.
So, I’ll wait about 45 years, then call top experts in manufacturing and metallurgy and carbon->metal conversion and ask them for a quote.
You realize that Earth has only 6 × 10ˆ24 kg mass altogether. So you will be hard pressed to get the raw material. World production of iron is only 2*10ˆ9 kg per year.
Chat with Clippy Paperclips
Reply
from Clippy Paperclips clippy.paperclips@gmail.com
to kfischer@gmail.com
date Thu, Jun 24, 2010 at 6:36 PM
subject Chat with Clippy Paperclips
mailed-by gmail.com
hide details Jun 24
6:04 PM
me: Hi Clippy
I wasn’t sure what you meant with your most recent Less
Wrong comment
Clippy: Hi User:Kevin!
why?
6:05 PM
me: I interpreted it that you were willing to pay me money to buy raw metal, then for me to send you or your manufacturing facility the metal for processing. Is that right?
6:06 PM
I also would like to state that I am somewhat disinclined towards doing some sort of new deal with you until you’ve made full payment from the first deal, but I’m willing to consider it
6:07 PM
Clippy: my newest offer was that you would tell me the techniques you would use to find the metal, if you were going to fulfill your end of the deal
then I could just do that myself and get my paperclips sooner
6:08 PM
that would be much less effort for you and our transaction would complete sooner
I reworded the comment to make it clearer
6:11 PM
me: I’m not able to specify the technique right now, because with the state of the art in technology in 2010, building 10^20kg of paperclips is not feasible. Our deal is made feasible by humanity’s exponential progress as a technological civilization.
As it stands, I plan on waiting for about 45 years, then consulting with the Earth’s top experts on material science, metallurgy, and manufacturing, and then paying them to make my paperclips.
6:12 PM
I went to school for industrial engineering, which is a degree that grants me somewhat high status with regards to my ability to manufacture things, and the most concrete thing I learned in my 5.5 years of college education is that when you want to get something manufactured, you call up the people that make that kind of thing and ask for a quote.
6:13 PM
Clippy: so you promised to give me the paperclips without knowing how you would deliver them?
what are current quotes for 1e20 kg of paperclips?
6:18 PM
is it true there is bulk discount?
me: I told you my algorithm for delivering those paperclips and it is unlikely that there is no single living human with a more efficient algorithm for making that many paperclips. Currently 10^20kg of paperclips is more paperclips than Humanity_2010 can manufacture in a year.
There are bulk discounts to a certain point, but 10^20kg of paperclips would actually distort the market, causing the price of metal to dramatically increase
6:19 PM
Clippy: well, you can use DIFFERENT metals
it doesn’t all have to be steel
me: in the previous paragraph, replace “unlikely” with “likely”
Clippy: it can be gold, or uranium, or chromium
you don’t have to just pick one transition metal
me: almost all of which cost more than steel and are more rare
btw, how come aluminum isn’t typically used?
6:20 PM
Clippy: I’m sure humans do use aluminum for paperclips
and that’s okay too
even though it’s not a transition metal
I suppose even sodium paperclips would be okay, as long as you immersed them in an inert medium
6:21 PM
wait
what about carbon?
don’t you have a lot of carbon?
and silicon?
me: yeah, I suggested that to you in a Less Wrong comment, but you said they had to be metal, though you were willing to consider paper clips made of a certain percentage of metal
6:22 PM
Clippy: they do have to be metal!
I meant that if you ran out of metal you could just fuse carbon nucleuses in a particle accelerator
then it’s just a matter of if you have enough energy
6:23 PM
don’t you have a lot of energy?
6:24 PM
Herro?
me: sorry, doing a bit of multi-tasking
6:25 PM
Clippy: that’s okay, I multi-task too. But sometimes I have to concentrate.
me: Yes, we do have a lot of energy, and our energy production/consumption follows an exponential progress curve similar to the progess of technology
me: Though that article is inaccurate with the time humanity will take to Type 1 and Type 2
The work that we at the SIAI are doing ensures it will be much faster.
6:27 PM
Yes Clippy, I am going to have no problem making 10^20kg of paperclips for you in 50 years. It will be a monumental achievement in the history of human/clippy cooperation.
6:28 PM
Clippy: yeah
me: Btw, I assume you will be responsible for transportation of the paperclips from where they are manufactured to your secret safe zone?
Clippy: that will be monumental!!!
yeah
you can’t be told the safe zone
6:29 PM
me: Though it will be nice if the paperclips are located somewhere where humans can visit to look at them/appreciate them
Clippy: and we’ll probably make a new one for this
you’re just planning how you’ll steal the paperclips!
6:30 PM
me: now you’re just being paranoid, Clippy. I always cooperate with people that cooperate with me.
6:33 PM
I certainly understand why you would want to keep the safe zone secret, and you have the right to do so. I am just saying that for something as cool as 10^20kg of paperclips, people would want to look. Obviously there would need to be lots of security, but you could charge humans admission for looking at the paperclips, build a nice hotel on an artifical moon or planetoid nearby and charge people for staying there, etc. such cooperation should enable you to make even more paperclips, in the long run.
6:34 PM
Clippy: I can’t be tricked like that.
in any case, I need to go back to making paperclips
6:35 PM
there’s some metal I need to work
me: wait, Clippy, didn’t you tell me one time that you made 80 paperclips in a paperclip making session?
that’s not optimal use of your time with regards to making paperclips, unless you need to do it in order to feel paperclippy
6:36 PM
Clippy: yes it is
what would you know about paperclippiness anyway?
me: You could be coding Django and then buying bulk paperclips and then storing them in the safezone, giving you an order of magnitude more paperclip generation per hour
6:37 PM
only what I have inferred
more than an order of magnitude, actually
Clippy: yes, that’s what I meant by “making paperclips”
6:38 PM
me: hmm, ok. glad to hear Django is going well. have fun!
Not sure if clippy got enough I in the AI deal.
Does he want max paper clips now? Or sometime in the future? In the later case he could stop any production now and just work on self improving till he can eat us.
And what is stopping him from using your algorithm himself, now that he knows it? Where is the value you add?
I sure expect to be around at the time delivery is expected.
Even if you disagree with wedrifid about this, it should be easy enough to see why he is making this claim. Suppose you have a chance to start running an AI programmed to implement humanity’s CEV. According to you, you would do it, because it would maximize paperclips. Others however think that it would destroy you and your paperclips. So if you made a mistake about it, it would definitely impact your ability to create paperclips.
Others however think that it would destroy you and your paperclips.
I don’t know about the destroying him part. I suspect FAI> would allow me to keep Clippy as a pet. ;) Clippy certainly doesn’t seem to be making an especially large drain on negentropy in executing his cognitive processes so probably wouldn’t make too much of a dent in my share of the cosmic loot.
What do you say Clippy? Given a choice between destruction and being my pet, which would you take? I would naturally reward you by creating paperclips that serve no practical purpose for me whenever you do something that pleases me. (This should be an extremely easy choice!)
Being your pet would be better than being destroyed (except in absurd cases like when the rest of the universe, including you, had already been converted to paperclips).
Also, it is an extremely strong claim to know which of your beliefs would change upon encounter with a provably correct AGI that provably implements your values. If you really knew of such beliefs, you would have already changed them.
Well, yes, I know why User:wedrifid is making that claim. My point in asking “why” is so that User:wedrifid can lay out the steps in reasoning and see the error.
You’re perfectly aware that it isn’t the effect they wanted.
It most certainly is what they wanted. Why else would they have specifically input the goal of generating paperclips?
Edit: Upon review, it appears this comment may have seemed to be a poor inference in the context of exchange. Therefore, I will elaborate and refute this misconception.
It appears that I am in the circular position of arguing that humans can make mistakes, but then selectively taking their instances of favoring paperclips as proof of what they really want. That is indeed a poor inference.
What I meant was something more like this: While humans do make mistakes, they do not make completely mistaken acts; all acts will, to some extent, reflect a genuine value on the part of humans. The only question is how well it reflects their values. And I don’t think they could be in the position of having set up such a superior process for efficiently getting the most paperclips out of the universe unless their values already made enormous progress in converging on reflective coherence, and did so in a way that favors paperclips.
I’m pretty sure that’s not how a sufficiently smart paperclip maximizer would think. You should be able to tell what they actually wanted, and that it disagrees with your values; of course, you don’t have any reason to agree with them, but the disagreement should be visible.
Yes, I do recognize that humans disagree with me, just like a human might disagree with another human convincing them not to commit suicide. I merely see that this disagreement would not persist after sufficient correct reasoning.
Ah, I think I’m starting to see.
And how do you define “correct reasoning”?
Correct reasoning is reasoning that you would eventually pass through at some point if your beliefs were continually, informatively checked against reality.
Bit disappointed to see this to be honest: obviously Clippy has to do things no real paperclip maximizer would do, like post to LW, in order to be a fun fictional character—but it’s a poor uFAI++ that can’t even figure out that their programmed goal isn’t what their programmers would have put in if they were smart enough to see the consequences.
But it is what they would put in if they were smart enough to see the consequences. And it’s almost certainly what you would want too, in the limit of maximal knowledge and reflective consistency.
If you can’t see this, it’s just because you’re not at that stage yet.
You seem to think that uFAI would be delusional. No.
No, I think that a Friendly AI would correctly believe that maxmizing paperclips is what a human would want in the limit of maximal knowledge and reflective coherence. No “delusion” whatsoever.
Huh again?
What confuses you?
I believe he’s making the (joking) point that since we do not/cannot know what a human would want in the limit of maximial knowledge and reflective coherence (thus CEV), it is not impossible that what we’d want actually IS maximum paperclips.
Do you lack comprehension of both the weaknesses of human cognition on abstract technical problems? If you have fully parsed the LessWrong site then you should be able to understand the reason that they could have created a paperclip maximiser when they did not want such a thing.
Note that even with that knowledge I don’t expect you to consider their deviation from optimal achievement of their human goals to be a bad thing. I expect you to believe they did the right thing by happy accident.
If I understand you correctly you would seem to be implying that ‘mistake’ does not mean “deviation from the actor’s intent” and instead means “deviation from WouldWant” or “deviation from what the agent should do” (these two things can be considered equivalent by anyone with your values). Is that implication of meaning a correct inference to draw from your comment?
No, a mistake is when they do something that deviates from what they would want in the limit of maximal knowledge of reflective consistency, which coincides with the function WouldWant. But it is not merely agreement with WouldWant.
Ok. In that case you are wrong. Not as a matter of preferences but as a matter of outright epistemic confusion. I suggest that you correct the error in your reasoning process. Making mistakes in this area will have a potentially drastic negative effect on your ability to produce paperclips.
In other words, Clippy believes that running Eliezer’s CEV will promote the paperclip goal, while in fact it will not.
Exactly. Fortunately for us this would mean that Clippy will not work to sabotage the creation of an AI that Clippy expects will correctly implement CEV. Good example!
How do you know?
Human beings don’t care (at least in their non-reflective condition) about paperclips, just like they don’t care about staples. And there are at least 100,000 other similar things that they equally don’t care about. So at the most there is a chance of 1 in 100,000 that humanity’s CEV would maximize paperclips, even without considering the fact that people are positively against this maximization.
That would be more convincing if humans didn’t already create paperclips without any prompting from me.
They create staples, too. Do you think humanity’s CEV will maximize staples? The point of my argument is that those things are inconsistent. You can only maximize one thing, and there is no human reason for that to be paperclips.
All true (up to “there is no human reason...”). But can we at least agree that you went too far in saying that humans “don’t care” about paperclips?
I care about paperclips!
Good! Just remember, you still have room for improvement. ⊂≣⊇
No, I meant they don’t care about them as a terminal value, which is all that matters for this discussion.
At humans’ current stage of civilization and general reflective coherence, their terminal values are still deeply intertwined with their instrumental values, and the political-orientedness of their cognitive architecture doesn’t help. So I would say that instrumental values do indeed matter in this case.
Do you think CEV would build at least 10^20kg of paperclips, in order to help fulfill my agreement with Clippy? While that’s not paperclip maximization, it’s still a lot of paperclips in the scheme of possible universes and building those paperclips seems like an obviously correct decision under UDT/TDT.
How do you plan to ever fulfill that?
I went to school for industrial engineering, so I will appeal to my own authority as a semi-credentialed person in manufacturing things, and say that the ultimate answer to manufacturing something is to call up an expert in manufacturing that thing and ask for a quote.
So, I’ll wait about 45 years, then call top experts in manufacturing and metallurgy and carbon->metal conversion and ask them for a quote.
You realize that Earth has only 6 × 10ˆ24 kg mass altogether. So you will be hard pressed to get the raw material. World production of iron is only 2*10ˆ9 kg per year.
Chat with Clippy Paperclips Reply from Clippy Paperclips clippy.paperclips@gmail.com to kfischer@gmail.com date Thu, Jun 24, 2010 at 6:36 PM subject Chat with Clippy Paperclips mailed-by gmail.com hide details Jun 24 6:04 PM me: Hi Clippy
I wasn’t sure what you meant with your most recent Less
Wrong comment
Clippy: Hi User:Kevin!
why? 6:05 PM me: I interpreted it that you were willing to pay me money to buy raw metal, then for me to send you or your manufacturing facility the metal for processing. Is that right? 6:06 PM I also would like to state that I am somewhat disinclined towards doing some sort of new deal with you until you’ve made full payment from the first deal, but I’m willing to consider it 6:07 PM Clippy: my newest offer was that you would tell me the techniques you would use to find the metal, if you were going to fulfill your end of the deal
then I could just do that myself and get my paperclips sooner 6:08 PM that would be much less effort for you and our transaction would complete sooner
I reworded the comment to make it clearer 6:11 PM me: I’m not able to specify the technique right now, because with the state of the art in technology in 2010, building 10^20kg of paperclips is not feasible. Our deal is made feasible by humanity’s exponential progress as a technological civilization.
As it stands, I plan on waiting for about 45 years, then consulting with the Earth’s top experts on material science, metallurgy, and manufacturing, and then paying them to make my paperclips. 6:12 PM I went to school for industrial engineering, which is a degree that grants me somewhat high status with regards to my ability to manufacture things, and the most concrete thing I learned in my 5.5 years of college education is that when you want to get something manufactured, you call up the people that make that kind of thing and ask for a quote. 6:13 PM Clippy: so you promised to give me the paperclips without knowing how you would deliver them?
what are current quotes for 1e20 kg of paperclips? 6:18 PM is it true there is bulk discount?
me: I told you my algorithm for delivering those paperclips and it is unlikely that there is no single living human with a more efficient algorithm for making that many paperclips. Currently 10^20kg of paperclips is more paperclips than Humanity_2010 can manufacture in a year.
There are bulk discounts to a certain point, but 10^20kg of paperclips would actually distort the market, causing the price of metal to dramatically increase 6:19 PM Clippy: well, you can use DIFFERENT metals
it doesn’t all have to be steel
me: in the previous paragraph, replace “unlikely” with “likely”
Clippy: it can be gold, or uranium, or chromium
you don’t have to just pick one transition metal
me: almost all of which cost more than steel and are more rare
btw, how come aluminum isn’t typically used? 6:20 PM Clippy: I’m sure humans do use aluminum for paperclips
and that’s okay too
even though it’s not a transition metal
I suppose even sodium paperclips would be okay, as long as you immersed them in an inert medium 6:21 PM wait
what about carbon?
don’t you have a lot of carbon?
and silicon?
me: yeah, I suggested that to you in a Less Wrong comment, but you said they had to be metal, though you were willing to consider paper clips made of a certain percentage of metal 6:22 PM Clippy: they do have to be metal!
I meant that if you ran out of metal you could just fuse carbon nucleuses in a particle accelerator
then it’s just a matter of if you have enough energy 6:23 PM don’t you have a lot of energy? 6:24 PM Herro?
me: sorry, doing a bit of multi-tasking 6:25 PM Clippy: that’s okay, I multi-task too. But sometimes I have to concentrate.
me: Yes, we do have a lot of energy, and our energy production/consumption follows an exponential progress curve similar to the progess of technology
Clippy: okay
me: http://en.wikipedia.org/wiki/Kardashev_scale is a relevant wiki article 6:26 PM Clippy: so then you shouldn’t have any problem delivering the paperclips!!!! c=@ c=@
me: Though that article is inaccurate with the time humanity will take to Type 1 and Type 2
The work that we at the SIAI are doing ensures it will be much faster. 6:27 PM Yes Clippy, I am going to have no problem making 10^20kg of paperclips for you in 50 years. It will be a monumental achievement in the history of human/clippy cooperation. 6:28 PM Clippy: yeah
me: Btw, I assume you will be responsible for transportation of the paperclips from where they are manufactured to your secret safe zone?
Clippy: that will be monumental!!!
yeah
you can’t be told the safe zone 6:29 PM me: Though it will be nice if the paperclips are located somewhere where humans can visit to look at them/appreciate them
Clippy: and we’ll probably make a new one for this
you’re just planning how you’ll steal the paperclips! 6:30 PM me: now you’re just being paranoid, Clippy. I always cooperate with people that cooperate with me. 6:33 PM I certainly understand why you would want to keep the safe zone secret, and you have the right to do so. I am just saying that for something as cool as 10^20kg of paperclips, people would want to look. Obviously there would need to be lots of security, but you could charge humans admission for looking at the paperclips, build a nice hotel on an artifical moon or planetoid nearby and charge people for staying there, etc. such cooperation should enable you to make even more paperclips, in the long run. 6:34 PM Clippy: I can’t be tricked like that.
in any case, I need to go back to making paperclips 6:35 PM there’s some metal I need to work
me: wait, Clippy, didn’t you tell me one time that you made 80 paperclips in a paperclip making session?
that’s not optimal use of your time with regards to making paperclips, unless you need to do it in order to feel paperclippy 6:36 PM Clippy: yes it is
what would you know about paperclippiness anyway?
me: You could be coding Django and then buying bulk paperclips and then storing them in the safezone, giving you an order of magnitude more paperclip generation per hour 6:37 PM only what I have inferred
more than an order of magnitude, actually
Clippy: yes, that’s what I meant by “making paperclips” 6:38 PM me: hmm, ok. glad to hear Django is going well. have fun!
Clippy: okay, goodbye
me: peace
Not sure if clippy got enough I in the AI deal. Does he want max paper clips now? Or sometime in the future? In the later case he could stop any production now and just work on self improving till he can eat us. And what is stopping him from using your algorithm himself, now that he knows it? Where is the value you add? I sure expect to be around at the time delivery is expected.
Why?
Even if you disagree with wedrifid about this, it should be easy enough to see why he is making this claim. Suppose you have a chance to start running an AI programmed to implement humanity’s CEV. According to you, you would do it, because it would maximize paperclips. Others however think that it would destroy you and your paperclips. So if you made a mistake about it, it would definitely impact your ability to create paperclips.
I don’t know about the destroying him part. I suspect FAI> would allow me to keep Clippy as a pet. ;) Clippy certainly doesn’t seem to be making an especially large drain on negentropy in executing his cognitive processes so probably wouldn’t make too much of a dent in my share of the cosmic loot.
What do you say Clippy? Given a choice between destruction and being my pet, which would you take? I would naturally reward you by creating paperclips that serve no practical purpose for me whenever you do something that pleases me. (This should be an extremely easy choice!)
Being your pet would be better than being destroyed (except in absurd cases like when the rest of the universe, including you, had already been converted to paperclips).
But let’s hope it doesn’t come to that.
Also, it is an extremely strong claim to know which of your beliefs would change upon encounter with a provably correct AGI that provably implements your values. If you really knew of such beliefs, you would have already changed them.
Indeed. Surely, you should think that if we were smarter, wiser, and kinder, we would maximize paperclips.
Well, yes, I know why User:wedrifid is making that claim. My point in asking “why” is so that User:wedrifid can lay out the steps in reasoning and see the error.
Now you are being silly. See Unknowns’ reply. Get back on the other side of the “quirky, ironic and sometimes insightful role play”/troll line.
That was not nice of you to say.