I think you’re correct in raising the general issue of what hypothetical problems it makes sense to consider, but your application to Newcomb’s does not go very far.
Personally, I can think of LOTS of reasons to doubt that Newcomb’s problem is even theoretically possible to set.
You didn’t give any, though, and Newcomb’s problem does not require an infallible Omega, only a fairly reliable one. The empirical barrier to believing in Omega is assumed away by another hypothesis: that you are sure that Omega is honest and reliable.
Personally, I think I can reliably predict that Eliezer would one-box against Omega, based on his public writings. I’m not sure if that implies that he would one-box against me, even if he agrees that he would one-box against Omega and that my prediction is based on good evidence that he would.
I’m pretty sure Eliezer would one-box against Omega any time box B contained more money than box A. Against you or me, I’m pretty sure he would one box with the original 1000000:1000 problem (that’s kind of the obvious answer), but not sure if it were a 1200:1000 problem.
A further thing to note: If Eliezer models other people as either significantly overestimating or significantly understimating the probability he’ll one-box against them, both possibilities increase the probability he’ll actually two-box against them.
So it all depends on Eliezer’s model of other people’s model of Eliezer’s model of their model. Insert The Princess Bride reference. :-)
Or at least your model of Eliezer models other people modeling his model of them. He may go one level deeper and model other people’s model of his model of other people’s model of his model of them, or (more likely) not bother and just use general heuristics. Because modeling breaks down around one or two layers of recursion most of the time.
Now we are getting somewhere good! Certainty rarely shows up in predictions, especially about the future. Your decision theory may be timeless, but don’t confuse the map with the territory, the universe may not be timeless.
Unless you are assigning a numerical, non-zero, non-unity probability to Omega’s accuracy, you do not know when to one-box and when to two-box with arbitrary amounts of money in the boxes. And unless your FAI is a chump, it is considering LOTS of details in estimating Omega’s accuracy, no doubt including considerations of how much the FAI’s own finiteness of knowledge and computation fails to constrain the possibility that Omega is tricking it.
A NASA engineer had been telling Feynman that the liquid rocket motor had a zero probability of exploding on takeoff. Feynman convinced him that this was not an engineering answer. The NASA engineer then smiled and told Feynman the probability of the liquid rocket motor exploding on take off was “epsilon.” Feynman replied (and I paraphrase from memory) “Good! Now we are getting somewhere! Now all you have to tell me is what your estimate for the value of epsilon is, and how you arrived at that number.”
Any calculation of your estimate of Omega’s responsibility which does not include gigantic terms for the evaluation of the probability that Omega is tricking you in a way you haven’t figure out yet is likely to fail. I base that on the prevalence and importance of con games in the best natural experiment on intelligence we have: humans.
If Eliezer knows that your prediction is based on good evidence that he would one-box, then that screens off the dependence between your prediction and his decision, so he should two-box.
Surely the same applies to Omega. By hypothesis, Eliezer knows that Omega is reliable, and since Eliezer does not believe in magic, he deduces that Omega’s prediction is based on good evidence, even if Omega doesn’t say anything about the evidence.
My only reason for being unsure that Eliezer would one-box against me is that there may be some reflexivity issue I haven’t thought of, but I don’t think this one works.
One issue is that I’m not going around making these offers to everyone, but the only role that that plays in the original problem is to establish Omega’s reliability without Newcomb having to explain how Omega does it. But I don’t think it matters where the confidence in Omega’s reliability comes from, as long as it is there.
If you know that Omega came to a conclusion about you based on things you wrote on the Internet, and you know that the things you wrote imply you will one-box, then you are free to two-box.
Edit: basically the thing you have to ask is, if you know where Omega’s model of you comes from, is that model like you to a sufficient extent that whatever you decide to do, the model will also do?
Ah, but the thing you DON’T know is that Omega isn’t cheating. Cheating LOOKS like magic but isn’t. Implicit in my point, certainly part of my thinking, is that unless you understand deeply and for sure HOW the trick is done, you can expect the trick will be done on you. So unless you can think of a million dollar upside to not getting the million dollars, you should let yourself be the mark of the conman Omega since your role seems to include getting a million dollars for whatever reasons Omega has to do that.
You should only two box if you understand Omega’s trick so well that you are sure you can break it, i.e. that you will get the million dollars anyway. And the value of breaking Omega’s trick is that the world doesn’t need more successful con men.
Considering the likelihood of being confronted by a fake Omega rather than a real one, it would seem a matter of great lack of foresight to not want to address this problem in coding your FAI.
Unless he figures you’re not an idiot and you already know that, in which case it’s better for him to have a rule that says “always one-box on Newcomb-like problems whenever the payoff for doing so exceeds n times the payoff for failed two-boxing” where n is a number (probably between 1.1 and 100) that represents the payment differences. Obviously, if he’s playing against something with no ability to predict his actions (e.g. a brick) he’s going to two-box no matter what. But a human with theory of mind is definitely not a brick and can predict his action with far better than random accuracy.
Personally, I think I can reliably predict that Eliezer would one-box against Omega, based on his public writings. I’m not sure if that implies that he would one-box against me,
And since any FAI Eliezer codes is (nearly) infinitely more likely to be presented Newcomb’s boxes by one such as you, or Penn and Teller, or Madoff than by Omega or his ilk, this would seem to be a more important question than the Newcomb’s problem with Omega.
Really the main point of my post is “Omega is (nearly) impossible therefore problems presuming Omega are (nearly) useless”. But the discussion has come mostly to my Newcomb’s example making explicit its lack of dependence on an Omega. But here in this comment you do point out that the “magical” aspect of Omega MAY influence the coding choice made. I think this supports my claim that even Newcomb’s problem, which COULD be stated without an Omega, may have a different answer than when stated with an Omega. That it is important when coding an FAI to consider just how much evidence it should require that it has an Omega it is dealing with before it concludes that it does. In the long run, my concern is that an FAI coded to accept an Omega will be susceptible to accepting people deliberately faking Omega, which are in our universe (nearly) infinitely more present than true Omegas.
Omega problems are not posed for the purpose of being prepared to deal with Omega should you, or an FAI, ever meet him. They are idealised test problems, thought experiments, for probing the strengths and weaknesses of formalised decision theories, especially regarding issues of self-reference and agents modelling themselves and each other. Some of these problems may turn out to be ill-posed, but you have to look at each such problem to decide whether it makes sense or not.
I think you’re correct in raising the general issue of what hypothetical problems it makes sense to consider, but your application to Newcomb’s does not go very far.
You didn’t give any, though, and Newcomb’s problem does not require an infallible Omega, only a fairly reliable one. The empirical barrier to believing in Omega is assumed away by another hypothesis: that you are sure that Omega is honest and reliable.
Personally, I think I can reliably predict that Eliezer would one-box against Omega, based on his public writings. I’m not sure if that implies that he would one-box against me, even if he agrees that he would one-box against Omega and that my prediction is based on good evidence that he would.
I’m pretty sure Eliezer would one-box against Omega any time box B contained more money than box A. Against you or me, I’m pretty sure he would one box with the original 1000000:1000 problem (that’s kind of the obvious answer), but not sure if it were a 1200:1000 problem.
A further thing to note: If Eliezer models other people as either significantly overestimating or significantly understimating the probability he’ll one-box against them, both possibilities increase the probability he’ll actually two-box against them.
So it all depends on Eliezer’s model of other people’s model of Eliezer’s model of their model. Insert The Princess Bride reference. :-)
Or at least your model of Eliezer models other people modeling his model of them. He may go one level deeper and model other people’s model of his model of other people’s model of his model of them, or (more likely) not bother and just use general heuristics. Because modeling breaks down around one or two layers of recursion most of the time.
Now we are getting somewhere good! Certainty rarely shows up in predictions, especially about the future. Your decision theory may be timeless, but don’t confuse the map with the territory, the universe may not be timeless.
Unless you are assigning a numerical, non-zero, non-unity probability to Omega’s accuracy, you do not know when to one-box and when to two-box with arbitrary amounts of money in the boxes. And unless your FAI is a chump, it is considering LOTS of details in estimating Omega’s accuracy, no doubt including considerations of how much the FAI’s own finiteness of knowledge and computation fails to constrain the possibility that Omega is tricking it.
A NASA engineer had been telling Feynman that the liquid rocket motor had a zero probability of exploding on takeoff. Feynman convinced him that this was not an engineering answer. The NASA engineer then smiled and told Feynman the probability of the liquid rocket motor exploding on take off was “epsilon.” Feynman replied (and I paraphrase from memory) “Good! Now we are getting somewhere! Now all you have to tell me is what your estimate for the value of epsilon is, and how you arrived at that number.”
Any calculation of your estimate of Omega’s responsibility which does not include gigantic terms for the evaluation of the probability that Omega is tricking you in a way you haven’t figure out yet is likely to fail. I base that on the prevalence and importance of con games in the best natural experiment on intelligence we have: humans.
If Eliezer knows that your prediction is based on good evidence that he would one-box, then that screens off the dependence between your prediction and his decision, so he should two-box.
Surely the same applies to Omega. By hypothesis, Eliezer knows that Omega is reliable, and since Eliezer does not believe in magic, he deduces that Omega’s prediction is based on good evidence, even if Omega doesn’t say anything about the evidence.
My only reason for being unsure that Eliezer would one-box against me is that there may be some reflexivity issue I haven’t thought of, but I don’t think this one works.
One issue is that I’m not going around making these offers to everyone, but the only role that that plays in the original problem is to establish Omega’s reliability without Newcomb having to explain how Omega does it. But I don’t think it matters where the confidence in Omega’s reliability comes from, as long as it is there.
If you know that Omega came to a conclusion about you based on things you wrote on the Internet, and you know that the things you wrote imply you will one-box, then you are free to two-box.
Edit: basically the thing you have to ask is, if you know where Omega’s model of you comes from, is that model like you to a sufficient extent that whatever you decide to do, the model will also do?
Ah, but the thing you DON’T know is that Omega isn’t cheating. Cheating LOOKS like magic but isn’t. Implicit in my point, certainly part of my thinking, is that unless you understand deeply and for sure HOW the trick is done, you can expect the trick will be done on you. So unless you can think of a million dollar upside to not getting the million dollars, you should let yourself be the mark of the conman Omega since your role seems to include getting a million dollars for whatever reasons Omega has to do that.
You should only two box if you understand Omega’s trick so well that you are sure you can break it, i.e. that you will get the million dollars anyway. And the value of breaking Omega’s trick is that the world doesn’t need more successful con men.
Considering the likelihood of being confronted by a fake Omega rather than a real one, it would seem a matter of great lack of foresight to not want to address this problem in coding your FAI.
Unless he figures you’re not an idiot and you already know that, in which case it’s better for him to have a rule that says “always one-box on Newcomb-like problems whenever the payoff for doing so exceeds n times the payoff for failed two-boxing” where n is a number (probably between 1.1 and 100) that represents the payment differences. Obviously, if he’s playing against something with no ability to predict his actions (e.g. a brick) he’s going to two-box no matter what. But a human with theory of mind is definitely not a brick and can predict his action with far better than random accuracy.
And since any FAI Eliezer codes is (nearly) infinitely more likely to be presented Newcomb’s boxes by one such as you, or Penn and Teller, or Madoff than by Omega or his ilk, this would seem to be a more important question than the Newcomb’s problem with Omega.
Really the main point of my post is “Omega is (nearly) impossible therefore problems presuming Omega are (nearly) useless”. But the discussion has come mostly to my Newcomb’s example making explicit its lack of dependence on an Omega. But here in this comment you do point out that the “magical” aspect of Omega MAY influence the coding choice made. I think this supports my claim that even Newcomb’s problem, which COULD be stated without an Omega, may have a different answer than when stated with an Omega. That it is important when coding an FAI to consider just how much evidence it should require that it has an Omega it is dealing with before it concludes that it does. In the long run, my concern is that an FAI coded to accept an Omega will be susceptible to accepting people deliberately faking Omega, which are in our universe (nearly) infinitely more present than true Omegas.
Omega problems are not posed for the purpose of being prepared to deal with Omega should you, or an FAI, ever meet him. They are idealised test problems, thought experiments, for probing the strengths and weaknesses of formalised decision theories, especially regarding issues of self-reference and agents modelling themselves and each other. Some of these problems may turn out to be ill-posed, but you have to look at each such problem to decide whether it makes sense or not.