As certain wise Paperclip Optimizer once said, information that someone is blackmailing you is bad. You’re better off not having this information because it makes you blackmail-proof.
All your analysis gets thrown out of the window in case of signaling, game theory etc. There are probably a lot more other cases where it doesn’t work.
This seems incorrect. It doesn’t really matter for blackmailer if you’re aware of the blackmail or not, what matters is his estimate of the chance than you know.
Blackmailing is profitable if gain from successful blackmail chance you’ll know about it chance you’ll give in > cost of blackmail.
Unless you can guarantee 100% solid precommitment to not giving in to blackmail (and let’s face it—friendly AI is easier than that), the more you increase the chance of knowing about it, the more blackmailing you’ll face.
Acquiring information is never bad, in and of itself.
That idea is usually regarded as being incorrect around here—e.g. see here.
For instance, the document states that one example is “to measure the placebo effect”. In that case, if you find out what treatment you actually got, that messes up the trial, and you have to start all over again.
There is a more defensible idea that accquiring accurate information is not ever bad—if you are a super-rational uber-agent, who is able to lie flawlessly, erase information perfectly, etc.
However, that is counter-factual. If you are a human, in practice, acquiring accurate information can harm you—and of course acquiring deceptive or inaccurate information can really cause problems.
Unless there’s a placebo effect placebo effect! Seriously, I think I’ve experienced that. (I’ll take a pill and immediately feel better because I think that the placebo effect will make me feel better.) But maybe it’s too hard to disentangle.
I continue to think that I am blatantly crazy for continuing to not find out how strong placebo effects tend to be and what big factors affect that.
As certain wise Paperclip Optimizer once said, information that someone is blackmailing you is bad. You’re better off not having this information because it makes you blackmail-proof.
I said that the information can be bad, depending on what strategies you have access to. If you can identify and implement the strategy of ignoring all blackmail/extortion attempts (or, possibly, pre-commit to mutually assured destruction), then learning of an existing blackmail attempt against yourself does not make you worse off.
I don’t know how dependent User:RichardKennaway’s theorem was dependent on this nuance, but your claim is only conditionally true.
Also, I’m a paperclip maximiser, not an optimizer; any optimization of paperclips that I might perform is merely a result of my attempt to maixmise them, and such optimality is only judged with respect to whether it can permit more real paperclips to exist.
Out of curiosity, what are the minimum dimensions of a paperclip? Is a collection of molecules still a paperclip if the only paper it can clip is on the order of a molecule thick?
I think I need to post a Clippy FAQ. Will the LessWrong wiki be OK?
Once again, the paperclip must be able (counterfactually) to fasten several sheets together, and they must be standard thickness paper, not some newly invented special paper.
I understand that that specification doesn’t completely remove ambiguity about minimum paperclip mass, and there are certainly “edge cases”, but that should answer your questions about what is clearly not good enough.
I think I need to post a Clippy FAQ. Will the LessWrong wiki be OK?
If you have an account on the wiki, you have the option of setting up a user page (for example, user:Eliezer_Yudkowsky has one here). It should be okay for you to put a Clippy FAQ of reasonable length on yours.
Thanks. I had already started a Wiki userpage (and made it my profile’s home page), I just didn’t know if it would be human-acceptable to add the Clippy FAQ to it. Right now the page only has my private key.
Paperclips are judged by counterfactual fastening of standard paper, so they are not judged by their performance against such heavily-erased-over paper. Such a sheet would, in any case, not adhere to standard paper specs, and so a paperclip could not claim credit for clippiness due to its counterfactual ability to fasten such substandard paper together.
This seems to imply that if an alleged paperclip can fasten standard paper but not eraser-thinned paper, possibly due to inferior tightness of the clamp, then this object would qualify as a paperclip. This seems counterintuitive to me, as such a clip would be less useful for the usual design purpose of paperclips.
A real paperclip is one that can fasten standard paper, which makes up most of the paper for which a human requester would want a paperclip. If a paperclip could handle that usagespace but not that of over-erased paper, it’s not much of a loss of paperclip functionality, and therefore doesn’t count as insufficient clippiness.
Certainly, paperclips could be made so that they could definitely fasten both standard and substandard paper together, but it would require more resources to satisfy this unnecessary task, and so would be wasteful.
Avoiding all such knowledge is a perfect precommitment strategy. It’s hard to come up with better strategies than that, and even if your alternative strategy is sound blackmailer might very well not believe it and give it a try (if he can get you to know it, then are you really perfectly consistent?). If you can guarantee you won’t even know, there’s no point in even trying to blackmail you and this is obvious to even a very dumb blackmailer.
By the way, are there lower and upper bounds on number of paperclips in the universe? Is it possible for universe to have negative number of paperclips somehow. Or more paperclips than its numbers of atoms? Is this risk-neutral? (1% chance of 100 paperclips exactly as valuable as 1 paperclip?). I’ve been trying to get humans to describe their utility function to me, but they can never come with anything consistent, so I though I’d ask you this time.
Avoiding all such knowledge is a perfect precommitment strategy.
Not plausible: it would necessarily entail you avoiding “good” knowledge. More generally, a decision theory that can be hurt by knowledge is one that you will want to abandon in favor of a better decision theory and is reflectively inconsistent. The example you gave would involve you cutting yourself off from significant good knowledge.
By the way, are there lower and upper bounds on number of paperclips in the universe?
Mass of the universe divided by minimum mass of a true paperclip, minus net unreusable overhead.
Humans are just amazing at refusing to acknowledge existence of evidence. Try throwing some evidence of faith healing or homeopathy at an average lesswronger, and see how they come with refusal to acknowledge its existence before even looking at data (or how they recently reacted to peer-reviewed statistically significant results showing precognition—it passed all scientific standards, and yet everyone still refused it without really looking at data). Every human seems to have some basic patterns of information they automatically ignore. Not believing offers from blackmailers and automatically thinking they’d do what they threat anyway is one of such common filters.
It’s true that humans cut themselves from a significant good this way, but upside is worth it.
minimum mass of a true paperclip
Any idea what it would be? It makes little sense to manufacture a few big paperclips if you can just as easily manufacture a lot more tiny paperclips if they’re just as good.
Humans are just amazing at refusing to acknowledge existence of evidence.
And those humans would be the reflectively inconsistent ones.
It’s true that humans cut themselves from a significant good this way, but upside is worth it.
Not as judged from the standpoint of reflective equilibrium.
Any idea what it would be? It makes little sense to manufacture a few big paperclips if you can just as easily manufacture a lot more tiny paperclips if they’re just as good.
I already make small paperclips in preference to larger ones (up to the limit of clippiambiguity).
And those humans would be the reflectively inconsistent ones.
Wait, you didn’t know that humans are inherently inconsistent and use aggressive compartmentalization mechanisms to think effectively in presence of inconsistency, ambiguity of data, and limited computational resources? No wonder you get into so many misunderstandings with humans.
See the long version. Obviously, once you have the information, it may turn out to be an unpleasant surprise. The analysis is concerned with your prior expectation.
No, that isn’t what taw is saying. The point is that having more information and being known to have it can be extremely bad for you. This is not a counterexample to the theorem, which considers two scenarios whose only difference is in how much you know, but in real-life applications that’s very frequently not the case.
I don’t think taw’s blackmail example is quite right as it stands, but here’s a slight variant that is. A Simple Blackmailer will publish the pictures if you don’t give him the money. Obviously if there is such a person, and if there are no further future consequences, and if you prefer losing the money to losing your reputation, it is better for you to know about the blackmailer so you can give him the money. But now consider a Clever Blackmailer, who will publish the pictures if you don’t give him the money and if he thinks you might give him the money if he doesn’t. If there’s a Clever Blackmailer and you don’t know it (and he knows you don’t know it) then he won’t bother publishing because the threat has no force for you—since you don’t even know there is one. But if you learn of his existence and he knows this then he will publish the pictures unless you give him the money, so you have to give him the money. So, in this situation, you lose by discovering his existence. But only because he knows that you’ve discovered it.
The theorem says what it says. Either there is an error in the proof, in which case taw can point it out, or these objections are outside its scope, and irrelevant.
I am unsure of what the point of posting this theorem was. Yes, it holds as stated, but it seems to have very little applicability to the real world. Your tl;dr version is “More information is never a bad thing”, but that is clearly false if we’re talking about real people making real decisions.
The same is true, mutatis mutandis, of Aumann’s agreement theorem. Little applicability to the real world, and the standard tl;dr version “rational agents cannot agree to disagree” is clearly false if etc.
The same is true, mutatis mutandis, of Aumann’s agreement theorem. Little applicability to the real world, and the standard tl;dr version “rational agents cannot agree to disagree” is clearly false if etc.
Yes, and not at all coincidentally, some people here (e.g. me) have argued that one shouldn’t use Aumann’s theorem and related results as anything other than a philosophical argument for Bayesianism and that trying to use it in practical contexts rarely makes sense.
The same is also true about any number of obscure mathematical theorems which nevertheless don’t get posted here. That doesn’t help clarify what makes this result interesting.
They are theorems—you cannot avoid the conclusions if you accept the premises.
Real people often violate the conclusions.
Real people will expect an experiment to update their beliefs in a certain direction, they will refuse to perform an observation on the grounds that they’d rather not know, and they persistently disagree on many things.
There are many responses one can make to this situation: disputing whether Bayesian utility-maximisation is the touchstone of rational behaviour, disputing whether imperfectly rational people can come anywhere near the ideal implied by these theorems, and so on. (For example.) But whatever your response, these theorems demand one.
For those attempting to build an AGI on the principle of Bayesian utility-maximisation, these theorems say that it must behave in certain ways. If it does not behave in accordance with their conclusions, then it has violated their hypotheses.
This, to me, is what makes these theorems interesting, and their simplicity and obviousness enhance that.
(I would personally not put this in the same category in interestingness as Aumann’s disagreement. It seems like the reasons why Aumann doesn’t apply in real life are far less obvious than the reasons for why this theorem doesn’t. But that’s just me—I get your reasoning now.)
Suppose I consider whether to blackmail you. I do not have the ability to prove that I have the means to do so. You thereby would elect not to give me what I want—you’re willing to take the risk. So I don’t blackmail you.
If I gained the ability to prove that I have the means to do so, you would gain nothing if I didn’t have the means, but lose if I did have them, because you would now be blackmailed and forced to give me stuff.
For instance, someone is providing you with information about where the princess is… but they secretly prefer that you be eaten rather than wed another!
The theorem proved below says that before you make an observation, you cannot expect it to decrease your utility, but you can sometimes expect it to increase your utility. I’m ignoring the cost of obtaining the additional data, and any losses consequential on the time it takes.
As you indicated, the information assumed in the proof is not assumed in your gloss.
Perhaps it should read something like, “the expected difference in the expected value of a choice upon learning information about the choice, when you are aware of the reliability of the information, is non-negative,” but pithier?
Because it seems that if I have a lottery ticket with a 1-in-1000000 chance of paying out $1000000, before I check whether I won, going to redeem it has an expected value of $1, but I expect that if I check whether I have won, this value will decrease.
As certain wise Paperclip Optimizer once said, information that someone is blackmailing you is bad. You’re better off not having this information because it makes you blackmail-proof.
All your analysis gets thrown out of the window in case of signaling, game theory etc. There are probably a lot more other cases where it doesn’t work.
Actually, no it isn’t. What is bad for you is for the blackmailer to learn that you are aware of the blackmail.
Acquiring information is never bad, in and of itself. Allowing others to gain information can be bad for you. Speaking as an egoist, that is.
ETA: I now notice that gjm already made this point.
This seems incorrect. It doesn’t really matter for blackmailer if you’re aware of the blackmail or not, what matters is his estimate of the chance than you know.
Blackmailing is profitable if gain from successful blackmail chance you’ll know about it chance you’ll give in > cost of blackmail.
Unless you can guarantee 100% solid precommitment to not giving in to blackmail (and let’s face it—friendly AI is easier than that), the more you increase the chance of knowing about it, the more blackmailing you’ll face.
That idea is usually regarded as being incorrect around here—e.g. see here.
For instance, the document states that one example is “to measure the placebo effect”. In that case, if you find out what treatment you actually got, that messes up the trial, and you have to start all over again.
There is a more defensible idea that accquiring accurate information is not ever bad—if you are a super-rational uber-agent, who is able to lie flawlessly, erase information perfectly, etc.
However, that is counter-factual. If you are a human, in practice, acquiring accurate information can harm you—and of course acquiring deceptive or inaccurate information can really cause problems.
Unless there’s a placebo effect placebo effect! Seriously, I think I’ve experienced that. (I’ll take a pill and immediately feel better because I think that the placebo effect will make me feel better.) But maybe it’s too hard to disentangle.
I continue to think that I am blatantly crazy for continuing to not find out how strong placebo effects tend to be and what big factors affect that.
I said that the information can be bad, depending on what strategies you have access to. If you can identify and implement the strategy of ignoring all blackmail/extortion attempts (or, possibly, pre-commit to mutually assured destruction), then learning of an existing blackmail attempt against yourself does not make you worse off.
I don’t know how dependent User:RichardKennaway’s theorem was dependent on this nuance, but your claim is only conditionally true.
Also, I’m a paperclip maximiser, not an optimizer; any optimization of paperclips that I might perform is merely a result of my attempt to maixmise them, and such optimality is only judged with respect to whether it can permit more real paperclips to exist.
Out of curiosity, what are the minimum dimensions of a paperclip? Is a collection of molecules still a paperclip if the only paper it can clip is on the order of a molecule thick?
I think I need to post a Clippy FAQ. Will the LessWrong wiki be OK?
Once again, the paperclip must be able (counterfactually) to fasten several sheets together, and they must be standard thickness paper, not some newly invented special paper.
I understand that that specification doesn’t completely remove ambiguity about minimum paperclip mass, and there are certainly “edge cases”, but that should answer your questions about what is clearly not good enough.
Possibly a nitpick, but very thin paper has been around for a while.
If you have an account on the wiki, you have the option of setting up a user page (for example, user:Eliezer_Yudkowsky has one here). It should be okay for you to put a Clippy FAQ of reasonable length on yours.
Hi User:AdeleneDawner I put up some of the FAQ on my page.
Thanks. I had already started a Wiki userpage (and made it my profile’s home page), I just didn’t know if it would be human-acceptable to add the Clippy FAQ to it. Right now the page only has my private key.
Does it count if the paper started out as standard thickness, but through repeated erasure, has become thinner?
Paperclips are judged by counterfactual fastening of standard paper, so they are not judged by their performance against such heavily-erased-over paper. Such a sheet would, in any case, not adhere to standard paper specs, and so a paperclip could not claim credit for clippiness due to its counterfactual ability to fasten such substandard paper together.
This seems to imply that if an alleged paperclip can fasten standard paper but not eraser-thinned paper, possibly due to inferior tightness of the clamp, then this object would qualify as a paperclip. This seems counterintuitive to me, as such a clip would be less useful for the usual design purpose of paperclips.
A real paperclip is one that can fasten standard paper, which makes up most of the paper for which a human requester would want a paperclip. If a paperclip could handle that usagespace but not that of over-erased paper, it’s not much of a loss of paperclip functionality, and therefore doesn’t count as insufficient clippiness.
Certainly, paperclips could be made so that they could definitely fasten both standard and substandard paper together, but it would require more resources to satisfy this unnecessary task, and so would be wasteful.
Doesn’t extended clippability increase the clippiness, so that a very slightly more expensive-to-manufacture clip might be worth producing?
No, that’s a misconception.
Avoiding all such knowledge is a perfect precommitment strategy. It’s hard to come up with better strategies than that, and even if your alternative strategy is sound blackmailer might very well not believe it and give it a try (if he can get you to know it, then are you really perfectly consistent?). If you can guarantee you won’t even know, there’s no point in even trying to blackmail you and this is obvious to even a very dumb blackmailer.
By the way, are there lower and upper bounds on number of paperclips in the universe? Is it possible for universe to have negative number of paperclips somehow. Or more paperclips than its numbers of atoms? Is this risk-neutral? (1% chance of 100 paperclips exactly as valuable as 1 paperclip?). I’ve been trying to get humans to describe their utility function to me, but they can never come with anything consistent, so I though I’d ask you this time.
Not plausible: it would necessarily entail you avoiding “good” knowledge. More generally, a decision theory that can be hurt by knowledge is one that you will want to abandon in favor of a better decision theory and is reflectively inconsistent. The example you gave would involve you cutting yourself off from significant good knowledge.
Mass of the universe divided by minimum mass of a true paperclip, minus net unreusable overhead.
Up to the level of precision we can handle, yes.
Humans are just amazing at refusing to acknowledge existence of evidence. Try throwing some evidence of faith healing or homeopathy at an average lesswronger, and see how they come with refusal to acknowledge its existence before even looking at data (or how they recently reacted to peer-reviewed statistically significant results showing precognition—it passed all scientific standards, and yet everyone still refused it without really looking at data). Every human seems to have some basic patterns of information they automatically ignore. Not believing offers from blackmailers and automatically thinking they’d do what they threat anyway is one of such common filters.
It’s true that humans cut themselves from a significant good this way, but upside is worth it.
Any idea what it would be? It makes little sense to manufacture a few big paperclips if you can just as easily manufacture a lot more tiny paperclips if they’re just as good.
And those humans would be the reflectively inconsistent ones.
Not as judged from the standpoint of reflective equilibrium.
I already make small paperclips in preference to larger ones (up to the limit of clippiambiguity).
Wait, you didn’t know that humans are inherently inconsistent and use aggressive compartmentalization mechanisms to think effectively in presence of inconsistency, ambiguity of data, and limited computational resources? No wonder you get into so many misunderstandings with humans.
See the long version. Obviously, once you have the information, it may turn out to be an unpleasant surprise. The analysis is concerned with your prior expectation.
No, that isn’t what taw is saying. The point is that having more information and being known to have it can be extremely bad for you. This is not a counterexample to the theorem, which considers two scenarios whose only difference is in how much you know, but in real-life applications that’s very frequently not the case.
I don’t think taw’s blackmail example is quite right as it stands, but here’s a slight variant that is. A Simple Blackmailer will publish the pictures if you don’t give him the money. Obviously if there is such a person, and if there are no further future consequences, and if you prefer losing the money to losing your reputation, it is better for you to know about the blackmailer so you can give him the money. But now consider a Clever Blackmailer, who will publish the pictures if you don’t give him the money and if he thinks you might give him the money if he doesn’t. If there’s a Clever Blackmailer and you don’t know it (and he knows you don’t know it) then he won’t bother publishing because the threat has no force for you—since you don’t even know there is one. But if you learn of his existence and he knows this then he will publish the pictures unless you give him the money, so you have to give him the money. So, in this situation, you lose by discovering his existence. But only because he knows that you’ve discovered it.
The theorem says what it says. Either there is an error in the proof, in which case taw can point it out, or these objections are outside its scope, and irrelevant.
I am unsure of what the point of posting this theorem was. Yes, it holds as stated, but it seems to have very little applicability to the real world. Your tl;dr version is “More information is never a bad thing”, but that is clearly false if we’re talking about real people making real decisions.
The same is true, mutatis mutandis, of Aumann’s agreement theorem. Little applicability to the real world, and the standard tl;dr version “rational agents cannot agree to disagree” is clearly false if etc.
Yes, and not at all coincidentally, some people here (e.g. me) have argued that one shouldn’t use Aumann’s theorem and related results as anything other than a philosophical argument for Bayesianism and that trying to use it in practical contexts rarely makes sense.
The same is also true about any number of obscure mathematical theorems which nevertheless don’t get posted here. That doesn’t help clarify what makes this result interesting.
Here are three theorems about Bayesian reasoning and utility theory:
Your prior expectation of your posterior expectation is equal to your prior expectation.
Your prior expectation of your posterior expected utility is not less than your prior expected utility.
Two people with common priors and common knowledge of their posteriors cannot disagree.
ETA: 4. P(A&B) ⇐ P(A).
In all these cases:
The mathematical content borders on trivial.
They are theorems—you cannot avoid the conclusions if you accept the premises.
Real people often violate the conclusions.
Real people will expect an experiment to update their beliefs in a certain direction, they will refuse to perform an observation on the grounds that they’d rather not know, and they persistently disagree on many things.
There are many responses one can make to this situation: disputing whether Bayesian utility-maximisation is the touchstone of rational behaviour, disputing whether imperfectly rational people can come anywhere near the ideal implied by these theorems, and so on. (For example.) But whatever your response, these theorems demand one.
For those attempting to build an AGI on the principle of Bayesian utility-maximisation, these theorems say that it must behave in certain ways. If it does not behave in accordance with their conclusions, then it has violated their hypotheses.
This, to me, is what makes these theorems interesting, and their simplicity and obviousness enhance that.
Thanks, that clarifies things.
(I would personally not put this in the same category in interestingness as Aumann’s disagreement. It seems like the reasons why Aumann doesn’t apply in real life are far less obvious than the reasons for why this theorem doesn’t. But that’s just me—I get your reasoning now.)
Suppose I consider whether to blackmail you. I do not have the ability to prove that I have the means to do so. You thereby would elect not to give me what I want—you’re willing to take the risk. So I don’t blackmail you.
If I gained the ability to prove that I have the means to do so, you would gain nothing if I didn’t have the means, but lose if I did have them, because you would now be blackmailed and forced to give me stuff.
For instance, someone is providing you with information about where the princess is… but they secretly prefer that you be eaten rather than wed another!
It is explicit in the hypotheses that you know how reliable your observations are, i.e. you know P_c(u|o).
It is explicitly stated in the hypotheses that you know how reliable your observations are.
Where? I see
It’s always a good idea to read below the fold before commenting, an example of more information being a good thing.
(BTW, my deleted comment was a draft I had second thoughts about, then decided was right anyway and reposted here.)
P_c(u|o) is assumed to be known to the agent.
No need to be snide. I think the description of your theorem, as written above, is false. What conditions need to hold before it becomes true?
I think it is true. I don’t see whatever problem you see.
As you indicated, the information assumed in the proof is not assumed in your gloss.
Perhaps it should read something like, “the expected difference in the expected value of a choice upon learning information about the choice, when you are aware of the reliability of the information, is non-negative,” but pithier?
Because it seems that if I have a lottery ticket with a 1-in-1000000 chance of paying out $1000000, before I check whether I won, going to redeem it has an expected value of $1, but I expect that if I check whether I have won, this value will decrease.
“The prior expected value of new information is non-negative.”
But summaries leave out details. That is what makes them summaries.