Stanovich’s paper on why humans are apparently worse at following the VNM axioms than some animals has some interesting things to say, although I don’t like the way it says them. I quit halfway through the paper out of frustration, but what I got out of the paper (which may not be what the paper itself was trying to say) is more or less the following: humans model the world at different levels of complexity at different times, and at each of those levels different considerations come into play for making decisions. An agent behaving in this way can appear to be behaving VNM-irrationally when really it is just trying to efficiently use cognitive resources by not modeling the world at the maximum level of complexity all the time. Non-human animals may model the world at more similar levels of complexity over time, so they behave more VNM-rationally even if they have less overall optimization power than humans.
A related consideration, which is more about the methodology of studies claiming to measure human irrationality, is that the problem you think a test subject is solving is not necessarily the problem they’re actually solving. I guess a well-known example is when you ask people to play the prisoner’s dilemma but in their heads they’re really playing the iterated prisoner’s dilemma.
And another point: an agent can have a utility function and still behave VNM-irrationally if computing the VNM-rational thing to do given its utility function takes too much time, so the agent computes some approximation of it. It’s a given in practical applications of Bayesian statistics that Bayesian inference is usually intractable, so it’s necessary to compute some approximation to it, e.g. using Monte Carlo methods. The human brain may be doing something similar (a possibility explored in Lieder-Griffiths-Goodman, for example).
(Which reminds me: we don’t talk anywhere near enough about computational complexity on LW for my tastes. What’s up with that? An agent can’t do anything right if it can’t compute what “right” means before the Sun explodes.)
(Which reminds me: we don’t talk anywhere near enough about computational complexity on LW for my tastes. What’s up with that? An agent can’t do anything right if it can’t compute what “right” means before the Sun explodes.)
I agree with this concern (and my professional life is primarily focused on heuristic optimization methods, where computational complexity is huge).
I suspect it doesn’t get talked about much here because of the emphasis on intelligence explosion, missing AI insights, provably friendly, normative rationality, and there not being much to say. (The following are not positions I necessarily endorse.) An arbitrarily powerful intelligence might not care much about computational complexity (though it’s obviously important if you still care about marginal benefit and marginal cost at that level of power). Until we understand what’s necessary for AGI, the engineering details separating polynomial, exponential, and totally intractable algorithms might not be very important. It’s really hard to prove how well heuristics do at optimization, let alone robustness. The Heuristics and Biases literature focuses on areas where it’s easy to show humans aren’t using the right math, rather than how best to think given the hardware you have, and some of that may be deeply embedded in the LW culture.
I think that there’s a strong interest in prescriptive rationality, though, and if you have something to say on that topic or computational complexity, I’m interested in hearing it.
A related consideration, which is more about the methodology of studies claiming to measure human irrationality, is that the problem you think a test subject is solving is not necessarily the problem they’re actually solving. I guess a well-known example is when you ask people to play the prisoner’s dilemma but in their heads they’re really playing the iterated prisoner’s dilemma.
Right, this is an important point that could use more discussion.
In closer inspection a lot of the “irrationalities” are either rational on a higher-level game, or to be expected given the inability of people to “feel” abstract facts that they are told.
That said, the inability to properly incorporate abstract information is quite a rationality problem.
(Which reminds me: we don’t talk anywhere near enough about computational complexity on LW for my tastes. What’s up with that? An agent can’t do anything right if it can’t compute what “right” means before the Sun explodes.)
I spent a large chunk of Sunday and Monday finally reading Death Note and came to appreciate how some people on LW can think that agents meticulously working out each other’s “I know that you know that I know” and then acting so as to interact with their simulations of each other, including their simulations of simulating each other, can seem a reasonable thing to aspire to. Even if actual politicians and so forth seem to do it by intuition, i.e., much more in hardware.
Have you ever played that thumb game where you stand around in a circle with some people and at each turn show 0, 1 or 2 thumbs? And each person takes turns calling out a guess for the total number of thumbs that will be shown? Playing that game gives a really strong sense of “Aha! I modeled you correctly because I knew that you knew that I knew …” but I never actually know if it’s real modeling or hindsight bias because of the way the game is played in real time. Maybe there’s a way to modify the rules to test that?
I once spent a very entertaining day with a friend wandering around art exhibits once, with both of us doing a lot of “OK, you really like that and that and that and you hate that and that” prediction and subsequent correction.
One thing that quickly became clear was that I could make decent guesses about her judgments long before I could articulate the general rules I was applying to do so, which gave me a really strong sense of having modeled her really well.
One thing that became clear much more slowly was that the general rules I was applying, once I became able to articulate them, were not nearly as complex as they seemed to be when I was simply engaging with them as these ineffable chunks of knowledge.
I concluded from this that that strong ineffable sense of complex modeling is no more evidence of complex modeling than the similar strong ineffable sense of “being on someone’s wavelength” is evidence of telepathy. It’s just the way my brain feels when it’s applying rules it can’t articulate to predict the behavior of complex systems.
This kind of explicit modelling is a recurring fictional trope. For example, Herbert uses it a lot in Dosadi Experiment to show off how totes cognitively advanced the Dosadi are.
And another point: an agent can have a utility function and still behave VNM-irrationally if computing the VNM-rational thing to do given its utility function takes too much time, so the agent computes some approximation of it. It’s a given in practical applications of Bayesian statistics that Bayesian inference is usually intractable, so it’s necessary to compute some approximation to it, e.g. using Monte Carlo methods. The human brain may be doing something similar (a possibility explored in Lieder-Griffiths-Goodman, for example).
Yes. “(Real bounded decision systems will take shortcuts for efficiency and may not achieve perfect rationality, like how real floating point arithmetic isn’t associative).”
(Which reminds me: we don’t talk anywhere near enough about computational complexity on LW for my tastes. What’s up with that? An agent can’t do anything right if it can’t compute what “right” means before the Sun explodes.)
On one hand, a lot of this is lacking a proper theory of logical uncertainty, which a lot of this is (I think).
On the other hand, the usual solution is to step up a level to choose best decision algorithm instead of trying to directly compute best decision. Then you can step up to not taking forever at this. I don’t know how to bottom this out.
Related: A properly built AI need not do any explicit utility maximizing at all; it could all be built implicitly into hardcoded algorithms, the same way most algorithms have implicit probability distributions. Of course, one of the easiest ways to maximize expected utility is to explicitly do so, but I would still expect most code in an optimized AI to be implicitly maximizing.
What you need to estimate for maximizing the utility is not utility but sign of the difference in expected utilities. “More accurate” estimation of utility on one side of the comparison can lead to less accurate estimation of the sign of the difference. Which is what Pascal muggers use.
The main implication is that actions based on comparison between most complete available estimations of utility do not maximize utility. It is similar to evaluating sums; when evaluating 1-1/2+1/3-1/4 and so on, the 1+1/3+1/5+1/7 is a more complete sum than 1 - you have processed more terms (and can pat yourself on the head for doing more arithmetics) , but less accurate. In practice one obtains highly biased “estimates” from someone putting a lot more effort into finding terms of the sign that benefits them the most, and sometimes, from some terms being easier to find.
In the above example, attempts to produce a most accurate estimate of the sum do a better job than attempts to produce most complete sum.
In general what you learn from applied mathematics is that plenty of methods that are in some abstract sense more distant from the perfect method have a result closer to the result of the perfect method.
E.g. the perfect method could evaluate every possible argument, sum all of them, and then decide. The approximate method can evaluate a least biased sample of the arguments, sum them, and then decide, whereas the method that tries to match the perfect method the most would sum all available arguments. If you could convince an agent that the latter is ‘most rational’ (which may be intuitively appealing because it does resemble the perfect method the most) and is what should be done, then in a complex subject where agent does not itself enumerate all arguments, you can feed arguments to that agent, biasing the sum, and extract profit of some kind.
humans model the world at different levels of complexity at different times, and at each of those levels different considerations come into play for making decisions. An agent behaving in this way can appear to be behaving VNM-irrationally when really it is just trying to efficiently use cognitive resources by not modeling the world at the maximum level of complexity all the time. Non-human animals may model the world at more similar levels of complexity over time, so they behave more VNM-rationally even if they have less overall optimization power than humans.
Notice the obvious implications to the ability of super-human AI’s to behave VNM-rationally.
Which are what? The AI that is managing some sort of upload society could trade it’s clock time for utility.
It’s no different from humans where you can either waste your time pondering if you’re being rational about how jumpy you are when you see a moving shadow that looks sort of like a sabre-toothed tiger, or you can figure out how to tie a rock to a stick; in the modern times, ponder what is a better deal at the store vs try to invent something and make a lot of money.
But the point is, it’s computing time costs utility, and so it can’t waste it on things that will not gain it enough utility.
If you consider 2x1x1 cube to have probability of 1⁄6 of landing on each side, you can still be VNM rational about that—then you won’t be dutch booked, you’ll lose money though because that cube is not a perfect die and you’ll accept losing bets. Real world is like that, it doesn’t give cookies for non-dutch-bookability, it gives cookies for correct predictions of what is actually going to happen.
Stanovich’s paper on why humans are apparently worse at following the VNM axioms than some animals has some interesting things to say, although I don’t like the way it says them. I quit halfway through the paper out of frustration, but what I got out of the paper (which may not be what the paper itself was trying to say) is more or less the following: humans model the world at different levels of complexity at different times, and at each of those levels different considerations come into play for making decisions. An agent behaving in this way can appear to be behaving VNM-irrationally when really it is just trying to efficiently use cognitive resources by not modeling the world at the maximum level of complexity all the time. Non-human animals may model the world at more similar levels of complexity over time, so they behave more VNM-rationally even if they have less overall optimization power than humans.
A related consideration, which is more about the methodology of studies claiming to measure human irrationality, is that the problem you think a test subject is solving is not necessarily the problem they’re actually solving. I guess a well-known example is when you ask people to play the prisoner’s dilemma but in their heads they’re really playing the iterated prisoner’s dilemma.
And another point: an agent can have a utility function and still behave VNM-irrationally if computing the VNM-rational thing to do given its utility function takes too much time, so the agent computes some approximation of it. It’s a given in practical applications of Bayesian statistics that Bayesian inference is usually intractable, so it’s necessary to compute some approximation to it, e.g. using Monte Carlo methods. The human brain may be doing something similar (a possibility explored in Lieder-Griffiths-Goodman, for example).
(Which reminds me: we don’t talk anywhere near enough about computational complexity on LW for my tastes. What’s up with that? An agent can’t do anything right if it can’t compute what “right” means before the Sun explodes.)
I agree with this concern (and my professional life is primarily focused on heuristic optimization methods, where computational complexity is huge).
I suspect it doesn’t get talked about much here because of the emphasis on intelligence explosion, missing AI insights, provably friendly, normative rationality, and there not being much to say. (The following are not positions I necessarily endorse.) An arbitrarily powerful intelligence might not care much about computational complexity (though it’s obviously important if you still care about marginal benefit and marginal cost at that level of power). Until we understand what’s necessary for AGI, the engineering details separating polynomial, exponential, and totally intractable algorithms might not be very important. It’s really hard to prove how well heuristics do at optimization, let alone robustness. The Heuristics and Biases literature focuses on areas where it’s easy to show humans aren’t using the right math, rather than how best to think given the hardware you have, and some of that may be deeply embedded in the LW culture.
I think that there’s a strong interest in prescriptive rationality, though, and if you have something to say on that topic or computational complexity, I’m interested in hearing it.
Right, this is an important point that could use more discussion.
In closer inspection a lot of the “irrationalities” are either rational on a higher-level game, or to be expected given the inability of people to “feel” abstract facts that they are told.
That said, the inability to properly incorporate abstract information is quite a rationality problem.
I’ve made this point quite a few times here and here
Depends, sometimes this is actually a decent way avoid believing every piece of abstract information one is presented with.
I spent a large chunk of Sunday and Monday finally reading Death Note and came to appreciate how some people on LW can think that agents meticulously working out each other’s “I know that you know that I know” and then acting so as to interact with their simulations of each other, including their simulations of simulating each other, can seem a reasonable thing to aspire to. Even if actual politicians and so forth seem to do it by intuition, i.e., much more in hardware.
Have you ever played that thumb game where you stand around in a circle with some people and at each turn show 0, 1 or 2 thumbs? And each person takes turns calling out a guess for the total number of thumbs that will be shown? Playing that game gives a really strong sense of “Aha! I modeled you correctly because I knew that you knew that I knew …” but I never actually know if it’s real modeling or hindsight bias because of the way the game is played in real time. Maybe there’s a way to modify the rules to test that?
I once spent a very entertaining day with a friend wandering around art exhibits once, with both of us doing a lot of “OK, you really like that and that and that and you hate that and that” prediction and subsequent correction.
One thing that quickly became clear was that I could make decent guesses about her judgments long before I could articulate the general rules I was applying to do so, which gave me a really strong sense of having modeled her really well.
One thing that became clear much more slowly was that the general rules I was applying, once I became able to articulate them, were not nearly as complex as they seemed to be when I was simply engaging with them as these ineffable chunks of knowledge.
I concluded from this that that strong ineffable sense of complex modeling is no more evidence of complex modeling than the similar strong ineffable sense of “being on someone’s wavelength” is evidence of telepathy. It’s just the way my brain feels when it’s applying rules it can’t articulate to predict the behavior of complex systems.
This kind of explicit modelling is a recurring fictional trope.
For example, Herbert uses it a lot in Dosadi Experiment to show off how totes cognitively advanced the Dosadi are.
Yes, but aspiring to it as an achievable thing very much strikes me as swallowing fictional evidence whole. (And, around LW, manga and anime.)
No argument; just citing prior fictional art. :-)
Yes. “(Real bounded decision systems will take shortcuts for efficiency and may not achieve perfect rationality, like how real floating point arithmetic isn’t associative).”
On one hand, a lot of this is lacking a proper theory of logical uncertainty, which a lot of this is (I think).
On the other hand, the usual solution is to step up a level to choose best decision algorithm instead of trying to directly compute best decision. Then you can step up to not taking forever at this. I don’t know how to bottom this out.
Related: A properly built AI need not do any explicit utility maximizing at all; it could all be built implicitly into hardcoded algorithms, the same way most algorithms have implicit probability distributions. Of course, one of the easiest ways to maximize expected utility is to explicitly do so, but I would still expect most code in an optimized AI to be implicitly maximizing.
What you need to estimate for maximizing the utility is not utility but sign of the difference in expected utilities. “More accurate” estimation of utility on one side of the comparison can lead to less accurate estimation of the sign of the difference. Which is what Pascal muggers use.
This is a very good point.
I wonder what the implications are...
The main implication is that actions based on comparison between most complete available estimations of utility do not maximize utility. It is similar to evaluating sums; when evaluating 1-1/2+1/3-1/4 and so on, the 1+1/3+1/5+1/7 is a more complete sum than 1 - you have processed more terms (and can pat yourself on the head for doing more arithmetics) , but less accurate. In practice one obtains highly biased “estimates” from someone putting a lot more effort into finding terms of the sign that benefits them the most, and sometimes, from some terms being easier to find.
Yes, that is a problem.
Are there other schemes that do a better job, though?
In the above example, attempts to produce a most accurate estimate of the sum do a better job than attempts to produce most complete sum.
In general what you learn from applied mathematics is that plenty of methods that are in some abstract sense more distant from the perfect method have a result closer to the result of the perfect method.
E.g. the perfect method could evaluate every possible argument, sum all of them, and then decide. The approximate method can evaluate a least biased sample of the arguments, sum them, and then decide, whereas the method that tries to match the perfect method the most would sum all available arguments. If you could convince an agent that the latter is ‘most rational’ (which may be intuitively appealing because it does resemble the perfect method the most) and is what should be done, then in a complex subject where agent does not itself enumerate all arguments, you can feed arguments to that agent, biasing the sum, and extract profit of some kind.
“Taken together the four experiments provide support for the Sampling Hypothesis, and the idea that there may be a rational explanation for the variability of children’s responses in domains like causal inference.”
That seems to be behind what I suspect is a paywall, except that the link I’d expect to solicit me for money is broken. Got a version that isn’t?
It’s going through a university proxy, so it’s just broken for you. Here’s the paper: http://dl.dropboxusercontent.com/u/85192141/2013-denison.pdf
Notice the obvious implications to the ability of super-human AI’s to behave VNM-rationally.
Which are what? The AI that is managing some sort of upload society could trade it’s clock time for utility.
It’s no different from humans where you can either waste your time pondering if you’re being rational about how jumpy you are when you see a moving shadow that looks sort of like a sabre-toothed tiger, or you can figure out how to tie a rock to a stick; in the modern times, ponder what is a better deal at the store vs try to invent something and make a lot of money.
It still has to deal with the external world.
But the point is, it’s computing time costs utility, and so it can’t waste it on things that will not gain it enough utility.
If you consider 2x1x1 cube to have probability of 1⁄6 of landing on each side, you can still be VNM rational about that—then you won’t be dutch booked, you’ll lose money though because that cube is not a perfect die and you’ll accept losing bets. Real world is like that, it doesn’t give cookies for non-dutch-bookability, it gives cookies for correct predictions of what is actually going to happen.