Just because you think an outcome is a thousand times more desirable, doesn’t necessarily mean you would accept a 1,000:1 bet for it.
Correct. VNM utility is not necessarily linear with respect to the intuitive strength of the preference. Your utility function is defined based on what bets you would accept, rather than being a way of telling you what bets you should accept.
Like, 1,000 people dying seems like it should objectively be 1,000 worse than 1 person dying. Does that mean I should pay the mugger?
Nope; see above. You can define a notion of utility that corresponds to this kind of notion of importance, but this will not necessarily be the decision-theoretic notion of utility. For example, suppose an agent wants there to be many happy people, and thinks that the goodness of an outcome is proportional to the number of happy people, so it gives its utility function as U(there are n happy people) = n. And suppose it has the following way of assigning utilities to uncertain outcomes: It picks some strictly increasing continuous function f (which could be arctan, for instance), it calculates the expected value of f(n), and applies f^-1 to that to get the utility. Assuming f is nonlinear, this agent does not use the mean utility as the utility of a gamble, but it is still VNM-rational, and thus by the VNM theorem, there exists a utility function V (not the same as U), such that the agent acts as if it was maximizing the expected value of V; this utility function is given by V(there are n happy people) = f(n).
Correct. VNM utility is not necessarily linear with respect to the intuitive strength of the preference. Your utility function is defined based on what bets you would accept, rather than being a way of telling you what bets you should accept.
Nope; see above. You can define a notion of utility that corresponds to this kind of notion of importance, but this will not necessarily be the decision-theoretic notion of utility. For example, suppose an agent wants there to be many happy people, and thinks that the goodness of an outcome is proportional to the number of happy people, so it gives its utility function as U(there are n happy people) = n. And suppose it has the following way of assigning utilities to uncertain outcomes: It picks some strictly increasing continuous function f (which could be arctan, for instance), it calculates the expected value of f(n), and applies f^-1 to that to get the utility. Assuming f is nonlinear, this agent does not use the mean utility as the utility of a gamble, but it is still VNM-rational, and thus by the VNM theorem, there exists a utility function V (not the same as U), such that the agent acts as if it was maximizing the expected value of V; this utility function is given by V(there are n happy people) = f(n).