Let’s talk about why a VNM utility is useful in the first place. The first reason is prescriptive: if you don’t have a VNM utility function, you risk being mugged by wandering Bayesians (similar to Dutch Book arguments). The second is descriptive: humans definitely aren’t perfect VNM-rational agents, but it’s very often a useful approximation. These two use-cases give different answers regarding the role of completeness.
First use-case: avoiding losing one’s shirt to an unfriendly Bayesian, who I’ll call Dr Malicious. The risk here is that, if we don’t even have well-ordered preferences in some region of world-space, then Dr Malicious could push us into that region and then money-pump us. But this really only matters to the extent that someone might actually attempt to pull a Dr Malicious on us, and could feasibly push us into a region where we don’t have well-ordered preferences. No one can feasibly push us into a world of peach ice-cream, and if they could, they’d probably have easier ways to make money than money-pumping us.
Second use-case: prediction based on approximate-VNM. Just like the first use-case, completeness really only matters over regions of world-space likely to come up in the problem at hand. If someone has no implicit utility outside that region, it usually won’t matter for our predictions.
So to close: this is an instance of spherical cow in a vacuum. In general, the spherical-cow-vacuum assumption is useful right up until it isn’t. Use common sense, remember that the real world does not perfectly follow the math, but the math is still really useful. You can add in corrections if and when you need them.
if you don’t have a VNM utility function, you risk being mugged by wandering Bayesians
I don’t see why this is true. While “VNM utility function ⇒ safe from wandering Bayesians”, it’s not clear to me that “no VNM utility function ⇒ vulnerable to wandering Bayesians.” I think the vulnerability to wandering Bayesians comes from failing to satisfy Transitivity rather than failing to satisfy Completeness. I have not done the math on that.
But the general point, about approximation, I like. Utility functions in game theory (decision theory?) problems normally involve only a small space. I think completeness is an entirely safe assumption when talking about humans deciding which route to take to their destination, or what bets to make in a specified game. My question comes from the use of VNM utility in AI papers like this one: http://intelligence.org/files/FormalizingConvergentGoals.pdf, where agents have a utility function over possible states of the universe (with the restriction that the space is finite).
Is the assumption that an AGI reasoning about universe-states has a utility function an example of reasonable use, for you?
Your intuition about transitivity being the key requirement is a good intuition. Completeness is more of a model foundation; we need completeness in order to even have preferences which can be transitive in the first place. A failure of completeness would mean that there “aren’t preferences” in some region of world-space. In practice, that’s probably a failure of the model—if the real system is offered a choice, it’s going to do something, even if that something amounts to really weird implied preferences.
So when I talk about Dr Malicious pushing us into a region without ordered preferences, that’s what I’m talking about. Even if our model contains no preferences in some region, we’re still going to have some actual behavior in that region. Unless that behavior implies ordered preferences, it’s going to be exploitable.
As for AIs reasoning about universe-states...
First, remember that there’s no rule saying that the utility must depend on all of the state variables. I don’t care about the exact position of every molecule in my ice cream, and that’s fine. Your universe can be defined by an infinite-dimensional state vector, and your AI can be indifferent to all but the first five variables. That’s fine.
Other than that, the above comments on completeness still apply. Faced with a choice, the AI is going to do something. Unless its behavior implies ordered preferences, it’s going to be exploitable, at least when faced with those kinds of choices. And as long as that exploitability is there, Dr Malicious will have an incentive to push the AI into the region where completeness fails. But if the AI has ordered preferences in all scenarios, Dr Malicious won’t have any reason to develop peach-ice-cream-destroying nanobots, and we probably just won’t need to worry about it.
if you don’t have a VNM utility function, you risk being mugged by wandering Bayesians (similar to Dutch Book arguments)
Ok, but since you don’t actually risk being mugged by wandering Bayesians, this isn’t actually a problem, right? (And I don’t mean this in a trivial “I live in a nice neighborhood and the police would save me” sense, but in a deep, decision-theoretic sense.)
prediction based on approximate-VNM
Could you give some examples of cases where this has actually been used?
The approximation of VNM rationality is foundational to most of economics. The whole field is basically “hey, what happens if you stick together VNM agents with different utility functions, information and resource baskets?”. So pretty much any successful prediction of economics is an example of humans approximating VNM-rational behavior. This includes really basic things like “prices increase when supply is expected to decrease”. If people lacked (approximate) utility functions, then prices wouldn’t increase (we’d just trade things in circles). If people weren’t taking the expectation of that utility function, then the mere expectation of shortage wouldn’t increase prices.
This is the sort of thing you need VNM utility for: it’s the underlying reason for lots of simple, everyday things. People pursue goals, despite having imperfect information about their environment—that’s VNM utility at work. Yes, people violate the math in many corner cases, but this is remarkable precisely because people do approximate VNM pretty well most of the time. Violations of transitivity, for instance, require fairly unusual conditions.
As for the risk of mugging, there are situations where you will definitely be money-pumped for violating VNM—think Wall Street or Vegas. In those situations, it’s either really cheap to money-pump someone (Wall Street), or lots of people are violating VNM (Vegas). In most day-to-day life, it’s not worth the effort to go hunting for people with inconsistent preferences or poor probability skills. Even if you found someone, they’d catch onto your money-pumping pretty quick, at which point they’d update to better approximate VNM rationality. Since it’s not profitable, people don’t usually do it. But as Wall Street and Vegas suggest, if a supply of VNM irrationality can be exploited with reasonable payoff-to-effort, people will exploit it.
Let’s talk about why a VNM utility is useful in the first place. The first reason is prescriptive: if you don’t have a VNM utility function, you risk being mugged by wandering Bayesians (similar to Dutch Book arguments). The second is descriptive: humans definitely aren’t perfect VNM-rational agents, but it’s very often a useful approximation. These two use-cases give different answers regarding the role of completeness.
First use-case: avoiding losing one’s shirt to an unfriendly Bayesian, who I’ll call Dr Malicious. The risk here is that, if we don’t even have well-ordered preferences in some region of world-space, then Dr Malicious could push us into that region and then money-pump us. But this really only matters to the extent that someone might actually attempt to pull a Dr Malicious on us, and could feasibly push us into a region where we don’t have well-ordered preferences. No one can feasibly push us into a world of peach ice-cream, and if they could, they’d probably have easier ways to make money than money-pumping us.
Second use-case: prediction based on approximate-VNM. Just like the first use-case, completeness really only matters over regions of world-space likely to come up in the problem at hand. If someone has no implicit utility outside that region, it usually won’t matter for our predictions.
So to close: this is an instance of spherical cow in a vacuum. In general, the spherical-cow-vacuum assumption is useful right up until it isn’t. Use common sense, remember that the real world does not perfectly follow the math, but the math is still really useful. You can add in corrections if and when you need them.
I’m not sure about the first case:
I don’t see why this is true. While “VNM utility function ⇒ safe from wandering Bayesians”, it’s not clear to me that “no VNM utility function ⇒ vulnerable to wandering Bayesians.” I think the vulnerability to wandering Bayesians comes from failing to satisfy Transitivity rather than failing to satisfy Completeness. I have not done the math on that.
But the general point, about approximation, I like. Utility functions in game theory (decision theory?) problems normally involve only a small space. I think completeness is an entirely safe assumption when talking about humans deciding which route to take to their destination, or what bets to make in a specified game. My question comes from the use of VNM utility in AI papers like this one: http://intelligence.org/files/FormalizingConvergentGoals.pdf, where agents have a utility function over possible states of the universe (with the restriction that the space is finite).
Is the assumption that an AGI reasoning about universe-states has a utility function an example of reasonable use, for you?
Your intuition about transitivity being the key requirement is a good intuition. Completeness is more of a model foundation; we need completeness in order to even have preferences which can be transitive in the first place. A failure of completeness would mean that there “aren’t preferences” in some region of world-space. In practice, that’s probably a failure of the model—if the real system is offered a choice, it’s going to do something, even if that something amounts to really weird implied preferences.
So when I talk about Dr Malicious pushing us into a region without ordered preferences, that’s what I’m talking about. Even if our model contains no preferences in some region, we’re still going to have some actual behavior in that region. Unless that behavior implies ordered preferences, it’s going to be exploitable.
As for AIs reasoning about universe-states...
First, remember that there’s no rule saying that the utility must depend on all of the state variables. I don’t care about the exact position of every molecule in my ice cream, and that’s fine. Your universe can be defined by an infinite-dimensional state vector, and your AI can be indifferent to all but the first five variables. That’s fine.
Other than that, the above comments on completeness still apply. Faced with a choice, the AI is going to do something. Unless its behavior implies ordered preferences, it’s going to be exploitable, at least when faced with those kinds of choices. And as long as that exploitability is there, Dr Malicious will have an incentive to push the AI into the region where completeness fails. But if the AI has ordered preferences in all scenarios, Dr Malicious won’t have any reason to develop peach-ice-cream-destroying nanobots, and we probably just won’t need to worry about it.
Ok, but since you don’t actually risk being mugged by wandering Bayesians, this isn’t actually a problem, right? (And I don’t mean this in a trivial “I live in a nice neighborhood and the police would save me” sense, but in a deep, decision-theoretic sense.)
Could you give some examples of cases where this has actually been used?
The approximation of VNM rationality is foundational to most of economics. The whole field is basically “hey, what happens if you stick together VNM agents with different utility functions, information and resource baskets?”. So pretty much any successful prediction of economics is an example of humans approximating VNM-rational behavior. This includes really basic things like “prices increase when supply is expected to decrease”. If people lacked (approximate) utility functions, then prices wouldn’t increase (we’d just trade things in circles). If people weren’t taking the expectation of that utility function, then the mere expectation of shortage wouldn’t increase prices.
This is the sort of thing you need VNM utility for: it’s the underlying reason for lots of simple, everyday things. People pursue goals, despite having imperfect information about their environment—that’s VNM utility at work. Yes, people violate the math in many corner cases, but this is remarkable precisely because people do approximate VNM pretty well most of the time. Violations of transitivity, for instance, require fairly unusual conditions.
As for the risk of mugging, there are situations where you will definitely be money-pumped for violating VNM—think Wall Street or Vegas. In those situations, it’s either really cheap to money-pump someone (Wall Street), or lots of people are violating VNM (Vegas). In most day-to-day life, it’s not worth the effort to go hunting for people with inconsistent preferences or poor probability skills. Even if you found someone, they’d catch onto your money-pumping pretty quick, at which point they’d update to better approximate VNM rationality. Since it’s not profitable, people don’t usually do it. But as Wall Street and Vegas suggest, if a supply of VNM irrationality can be exploited with reasonable payoff-to-effort, people will exploit it.