I agree it’s an easy point, but what I really want to drive home is that this easy point actually disproves the intuitively appealing conjecture: “A sufficiently rational agent in a fair environment should estimate the value of each option and choose the option with the highest value.”
There are lots of intuitive objections to this conjecture which people often raise, but most of them don’t seem to hold up. So when I realized that this one actually does hold up, I wanted to flesh it out in much more detail.
“A sufficiently rational agent in a fair environment should estimate the value of each option and choose the option with the highest value.”
I agree with you in the case of humans, but not in the case of sufficiently rational agents. It is a theorem that the expected value of X-Y is positive if and only if the expected value of X is greater than the expected value of Y. To rephrase using symbols:
This depends on what consistency conditions you get to impose on your agent. I agree that for probability distributions E[X-Y] = E[X] - E[Y].
Any computable agent, no matter how rational, isn’t going to have its beliefs closed under all of the obvious consistency conditions, otherwise it would assign P(T) = 1 for each theorem T. This isn’t just a quirk of human irrationality.
Maybe we should specify a subset of the consistency conditions which is achievable, and then we can say that expected utility maximization is optimal if you satisfy those consistency conditions. This is what I have been doing when thinking about these issues, but it doesn’t seem straightforward nor standard.
Can you find a toy model in which an agent choosing between options X, Y, Z estimates the three differences X—Y, Y—Z, Z—X and finds that they don’t add up to zero?
The variable diffs get set to a list of the three estimates. Adding them up we get −0.424. What has gone wrong?
X, Y, and Z are all 1⁄2. But they are tricky to measure. (defun x () (random 1.0)) is modelling the idea that when we estimate X we get a random variable uniform from 0.0 to 1.0.
(defun x-y () (- (x) (y)))
is modelling the idea that we estimate X and estimate Y and subtract. (And don’t remember our estimate)
(defun y-z () (- (y) (z)))
is modelling the idea that we start from scratch, estimating Y (again) and then Z before finally subtracting.
Since the Y in X-Y is our first go at estimating Y and the Y in Y-Z is our second go at estimating Y, they have different random errors and don’t cancel like they should.
tl;dr: Sometimes you can get a better estimate of the difference between two things than you can of either of them separately.
I agree it’s an easy point, but what I really want to drive home is that this easy point actually disproves the intuitively appealing conjecture: “A sufficiently rational agent in a fair environment should estimate the value of each option and choose the option with the highest value.”
There are lots of intuitive objections to this conjecture which people often raise, but most of them don’t seem to hold up. So when I realized that this one actually does hold up, I wanted to flesh it out in much more detail.
I agree with you in the case of humans, but not in the case of sufficiently rational agents. It is a theorem that the expected value of X-Y is positive if and only if the expected value of X is greater than the expected value of Y. To rephrase using symbols:
E(X-Y) > 0 iff E(X) > E(Y)
This depends on what consistency conditions you get to impose on your agent. I agree that for probability distributions E[X-Y] = E[X] - E[Y].
Any computable agent, no matter how rational, isn’t going to have its beliefs closed under all of the obvious consistency conditions, otherwise it would assign P(T) = 1 for each theorem T. This isn’t just a quirk of human irrationality.
Maybe we should specify a subset of the consistency conditions which is achievable, and then we can say that expected utility maximization is optimal if you satisfy those consistency conditions. This is what I have been doing when thinking about these issues, but it doesn’t seem straightforward nor standard.
Can you find a toy model in which an agent choosing between options X, Y, Z estimates the three differences X—Y, Y—Z, Z—X and finds that they don’t add up to zero?
I fear that I’ve missed your point, but here is my runnable toy model written in Common Lisp
The variable diffs get set to a list of the three estimates. Adding them up we get −0.424. What has gone wrong?
X, Y, and Z are all 1⁄2. But they are tricky to measure. (defun x () (random 1.0)) is modelling the idea that when we estimate X we get a random variable uniform from 0.0 to 1.0.
is modelling the idea that we estimate X and estimate Y and subtract. (And don’t remember our estimate)
is modelling the idea that we start from scratch, estimating Y (again) and then Z before finally subtracting.
Since the Y in X-Y is our first go at estimating Y and the Y in Y-Z is our second go at estimating Y, they have different random errors and don’t cancel like they should.