Very very wrong. The world program P (or what it does, anyway) is the only thing that’s actually controlled in this control problem statement (more generally, a list of programs, which could equivalently be represented by one program parametrized by an integer).
Edit: I misinterpreted the way Tyrrell used “P”, correction here.
These considerations lead to the following design for the decision algorithm S. S is coded with a vector of programs that it cares about, and a utility function on vectors of the form that defines its preferences on how those programs should run. When it receives an input X, it looks inside the programs P1, P2, P3, …, and uses its “mathematical intuition” to form a probability distribution P_Y over the set of vectors for each choice of output string Y. Finally, it outputs a string Y* that maximizes the expected utility Sum P_Y() U(). (This specifically assumes that expected utility maximization is the right way to deal with mathematical uncertainty. Consider it a temporary placeholder until that problem is solved. Also, I’m describing the algorithm as a brute force search for simplicity. In reality, you’d probably want it to do something cleverer to find the optimal Y* more quickly.)
If I am reading him correctly, he uses the letter “P” in two different ways. In one use, he writes Pi, where i is an integer, to denote a program. In the other use, he writes P_Y, where Y is an output vector, to denote a probability distribution.
Okay, the characterization of P_Y seems right. For my reaction I blame the prior.
Returning to the original argument,
the agent always cares about all possible worlds according to how probable those worlds seemed to the agent’s builders when they wrote the agent’s source code.
P_Y is not a description of probabilities of possible worlds conceived by agent’s builder, it’s something produced by “mathematical intuition module” for a given output Y (or, strategy Y if you incorporate the later patch to UDT).
P_Y is not a description of probabilities of possible worlds conceived by agent’s builder, it’s something produced by “mathematical intuition module” for a given output Y (or, strategy Y if you incorporate the later patch to UDT).
You are right here. Like you, I misremembered Wei Dai’s notation. See my last (I hope) edit to that comment.
I would appreciate it if you edited your comment where you say that I was “very very wrong” to say that P isn’t controlled by the agent’s decisions.
It’s easier to have a linear discussion, rather than trying to patch everything by reediting it from the start (just saying, you are doing this for the third time to that poor top-level comment). You’ve got something wrong, then I’ve got something wrong, the errors were corrected as the discussion developed, moving on. The history doesn’t need to be corrected. (I insert corrections to comments this way, without breaking the sequence.)
Very very wrong. The world program P (or what it does, anyway) is the only thing that’s actually controlled in this control problem statement (more generally, a list of programs, which could equivalently be represented by one program parametrized by an integer).
Edit: I misinterpreted the way Tyrrell used “P”, correction here.
Here is the relevant portion of Wei Dai’s post:
If I am reading him correctly, he uses the letter “P” in two different ways. In one use, he writes Pi, where i is an integer, to denote a program. In the other use, he writes P_Y, where Y is an output vector, to denote a probability distribution.
I was referring to the second use.
Okay, the characterization of P_Y seems right. For my reaction I blame the prior.
Returning to the original argument,
P_Y is not a description of probabilities of possible worlds conceived by agent’s builder, it’s something produced by “mathematical intuition module” for a given output Y (or, strategy Y if you incorporate the later patch to UDT).
You are right here. Like you, I misremembered Wei Dai’s notation. See my last (I hope) edit to that comment.
I would appreciate it if you edited your comment where you say that I was “very very wrong” to say that P isn’t controlled by the agent’s decisions.
It’s easier to have a linear discussion, rather than trying to patch everything by reediting it from the start (just saying, you are doing this for the third time to that poor top-level comment). You’ve got something wrong, then I’ve got something wrong, the errors were corrected as the discussion developed, moving on. The history doesn’t need to be corrected. (I insert corrections to comments this way, without breaking the sequence.)
Thank you for the edit.