I think most examples of “arguments from expected utility maximisation” are going to look like what Rob wrote: not actually using expected utility maximization, but rather “having goals that you do a pretty good job at accomplishing”. This gets you things like “more is better” with respect to resources like negentropy and computation, it gets you the idea that it’s better to achieve the probability that your goal is achieved, it gets you some degree of resisting changes to goal content (altho I think contra Omohundro this can totally happen in bargaining scenarios), if “goal content” exists, and it gets you “better to not be diverted from your goal by meddling humans”.
Also: I don’t understand how one is supposed to get from “trained agents have a variety of contextual influences on decision making” to “trained agents are not expected utility maximizers”, without somehow rebutting the arguments people make for why utility maximizers are good—and once we refute these arguments, we don’t need talk of “shards” to refute them extra hard. Like, you can have different influences on your behaviour at different times that all add up to coherence. For example, one obvious influence that would plausibly be reinforced is “think about coherence so you don’t randomly give up resources”. Maybe this is supposed to make more sense if we use “expected utility maximizer” in a way that excludes “thing that is almost expected-utility-optimal” or “thing that switches between different regimes of expected utility maximization” but that strikes me as silly.
I think most examples of “arguments from expected utility maximisation” are going to look like what Rob wrote: not actually using expected utility maximization, but rather “having goals that you do a pretty good job at accomplishing”. This gets you things like “more is better” with respect to resources like negentropy and computation, it gets you the idea that it’s better to achieve the probability that your goal is achieved, it gets you some degree of resisting changes to goal content (altho I think contra Omohundro this can totally happen in bargaining scenarios), if “goal content” exists, and it gets you “better to not be diverted from your goal by meddling humans”.
Also: I don’t understand how one is supposed to get from “trained agents have a variety of contextual influences on decision making” to “trained agents are not expected utility maximizers”, without somehow rebutting the arguments people make for why utility maximizers are good—and once we refute these arguments, we don’t need talk of “shards” to refute them extra hard. Like, you can have different influences on your behaviour at different times that all add up to coherence. For example, one obvious influence that would plausibly be reinforced is “think about coherence so you don’t randomly give up resources”. Maybe this is supposed to make more sense if we use “expected utility maximizer” in a way that excludes “thing that is almost expected-utility-optimal” or “thing that switches between different regimes of expected utility maximization” but that strikes me as silly.