“Before you entered the room, I ran a simulation of this problem as presented to an agent running TDT. ….”
This needs some serious mathematics underneath it. Omega is supposed to run a simulation of how an agent of a certain sort handled a certain problem, the result of that simulation being a part of the problem itself. I don’t think it’s possible to tell, just from these English words, that there is a solution to this fixed-point formulation. And TDT itself hasn’t been formalised, although I assume there are people (Eliezer? Marcello? Wei Dai?) working on that.
Cf. the construction of Gödel sentences: you can’t just assume that a proof-system can talk about itself, you have to explicitly construct a way for it to talk about itself and show precisely what “talking about itself” means, before you can do all the cool stuff about undecidable sentences, Löb’s theorem, and so on.
This seems well-specified to me: Since the agent is not told its own output in advance, it is possible to run the “simulation” and the “real version” in finite time. If you hand me a computer program that is the agent, I will hand you a computer program that is Omega and the environment.
if (being_simulated()) pick_box_b();
else pick_both_boxes();
being_simulated() is an interesting function; it would presumably consist of things like comparing star maps across as broad a range of EM frequencies as possible with historical knowledge, the positions of planetary bodies, Earthly satellites, quasar signatures, and inputs of other individuals and networks. It could also be as simple as “if (get_current_average_cosmic_ray_bit_flips() != historical_bit_flips_average) return true;” Since my brain tries to play Omega with me every night (well, on the nights I get good sleep) I think we all have some practical experience with being_simulated(). While our brains are nowhere near as capable as Omega they are fairly good at suspending our own disbelief during dreams . What generally breaks disbelief is trying to behave rationally, including observing/testing the environment which in reality would be the outside world but in a dream is simulated. Ultimately Omega is forced to simulate a very large subset of the Universe in order to cause being_simulated() to return false, or to alter the program itself so that being_simulated vacuously returns true.
To prevent the latter attack the program should execute a fully homomorphically encrypted sub-function and base all its decisions on the outputs of that function. In other words its true decision theory should be hidden even from the hardware running it. This pushes Omega’s capabilities into the cryptographic realm and the ability to solve arbitrarily hard cryptography problems. Once this is done, Omega becomes an oracle for solving any hard problem, e.g. by running this function as an input to the final decision theory:
if (strange_paradoxical_decision_is_required()) {
bank_secret = factor_omegas_bank_account_public_key();
return sign_and_cash_check_for_one_billion_dollars(bank_secret);
}
This restricts Omega to putting a time limit (or computation limit) on its simulations to avoid being suckered by its own computing power. Presumably Omega also puts a time limit on the real time a being can spend thinking before choosing which boxes it wants, but presumably that time is long enough for some fairly serious philosophizing and likely more than enough time for a reliable being_simulated() call.
Another approach is to consider how much information an intelligent being can gather about its environment. A human would have trouble determining whether two boxes are empty or full, but additional sensors could detect the presence of money in the boxes fairly reliably. What matters for Omega is making sure that a being cannot determine the contents of the boxes before picking them. From the perspective of a rational being this is equivalent to the boxes being filled with cash after making a decision. If Omega has the capability to obscure the contents of boxes then Omega certainly has the ability to obscure the placement of money into the boxes as they are chosen (just a glorified magic trick). Given that interpretation, CDT will one-box.
EDIT: I apologize for the formatting, I am not very good at escaping/formatting apparently.
In general it’s difficult, because by assumption Omega has the computational power to simulate more or less anything (including an environment matching the world as you remember it; this might be like the real world, or you might have spent your whole life so far as a sim). And the usual environment for these problems is a sealed room, so that you can’t look at the stars etc.
This needs some serious mathematics underneath it. Omega is supposed to run a simulation of how an agent of a certain sort handled a certain problem, the result of that simulation being a part of the problem itself. I don’t think it’s possible to tell, just from these English words, that there is a solution to this fixed-point formulation. And TDT itself hasn’t been formalised, although I assume there are people (Eliezer? Marcello? Wei Dai?) working on that.
Cf. the construction of Gödel sentences: you can’t just assume that a proof-system can talk about itself, you have to explicitly construct a way for it to talk about itself and show precisely what “talking about itself” means, before you can do all the cool stuff about undecidable sentences, Löb’s theorem, and so on.
This seems well-specified to me: Since the agent is not told its own output in advance, it is possible to run the “simulation” and the “real version” in finite time. If you hand me a computer program that is the agent, I will hand you a computer program that is Omega and the environment.
Here is a useful program:
if (being_simulated()) pick_box_b(); else pick_both_boxes();
being_simulated() is an interesting function; it would presumably consist of things like comparing star maps across as broad a range of EM frequencies as possible with historical knowledge, the positions of planetary bodies, Earthly satellites, quasar signatures, and inputs of other individuals and networks. It could also be as simple as “if (get_current_average_cosmic_ray_bit_flips() != historical_bit_flips_average) return true;” Since my brain tries to play Omega with me every night (well, on the nights I get good sleep) I think we all have some practical experience with being_simulated(). While our brains are nowhere near as capable as Omega they are fairly good at suspending our own disbelief during dreams . What generally breaks disbelief is trying to behave rationally, including observing/testing the environment which in reality would be the outside world but in a dream is simulated. Ultimately Omega is forced to simulate a very large subset of the Universe in order to cause being_simulated() to return false, or to alter the program itself so that being_simulated vacuously returns true.
To prevent the latter attack the program should execute a fully homomorphically encrypted sub-function and base all its decisions on the outputs of that function. In other words its true decision theory should be hidden even from the hardware running it. This pushes Omega’s capabilities into the cryptographic realm and the ability to solve arbitrarily hard cryptography problems. Once this is done, Omega becomes an oracle for solving any hard problem, e.g. by running this function as an input to the final decision theory:
if (strange_paradoxical_decision_is_required()) { bank_secret = factor_omegas_bank_account_public_key(); return sign_and_cash_check_for_one_billion_dollars(bank_secret); }
This restricts Omega to putting a time limit (or computation limit) on its simulations to avoid being suckered by its own computing power. Presumably Omega also puts a time limit on the real time a being can spend thinking before choosing which boxes it wants, but presumably that time is long enough for some fairly serious philosophizing and likely more than enough time for a reliable being_simulated() call.
Another approach is to consider how much information an intelligent being can gather about its environment. A human would have trouble determining whether two boxes are empty or full, but additional sensors could detect the presence of money in the boxes fairly reliably. What matters for Omega is making sure that a being cannot determine the contents of the boxes before picking them. From the perspective of a rational being this is equivalent to the boxes being filled with cash after making a decision. If Omega has the capability to obscure the contents of boxes then Omega certainly has the ability to obscure the placement of money into the boxes as they are chosen (just a glorified magic trick). Given that interpretation, CDT will one-box.
EDIT: I apologize for the formatting, I am not very good at escaping/formatting apparently.
This strategy is discussed in the follow-up article.
In general it’s difficult, because by assumption Omega has the computational power to simulate more or less anything (including an environment matching the world as you remember it; this might be like the real world, or you might have spent your whole life so far as a sim). And the usual environment for these problems is a sealed room, so that you can’t look at the stars etc.
But TDT already has this problem—TDT is all about finding a fixed point decision.