I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI.
Sorry if I misrepresented you, my intended meaning matches what you wrote. I was trying to replace “pure consequentialist” with its definition to make it obvious that it’s a ridiculously strong expectation that you’re saying Eliezer and others have.
Yes, assumptions about the domain of the utility function are needed in order to judge its behaviour as coherent or not. Rereading Coherent decisions imply consistent utilities, Eliezer is usually clear about the assumed domain of the utility function in each thought experiment. For example, he’s very clear here that you need the preferences as an assumption:
In the hospital thought experiment, he specifies the goal as an assumption:
In the pizza example, he doesn’t specify the domain, but it’s fairly obvious implicitly. In the fruit example, it’s also implicit but obvious.
There’s a few paragraphs at the end of the Allias paradox section about the (very non-consequentialist) goal of feeling certain during the decision-making process. I don’t get the impression from those paragraphs that Eliezer is saying that this preference is ruled out by any implicit assumption. In fact he explicitly says that this preference isn’t mathematically improper. It seems he’s saying this kind of preference cuts against coherence only if it’s getting in the way of more valuable decisions:
I think this quote in particular invalidates your statements.
There is a whole stack of assumptions[1] that Eliezer isn’t explicit about in that post. It’s intended to give a taste of the reasoning that gives us probability and expected utility, not the precise weakest set of assumptions required to make a coherence argument work.
I think one thing that is missing from that post are the reasons we usually do have prior knowledge of goals (among humans and for predicting advanced AI). Among humans we have good priors that heavily restrict the goal-space, plus introspection and stated preferences as additional data. For advanced AI, we can usually use usefulness (on some specified set of tasks) and generality (across a very wide range of potential obstacles) to narrow down the goal-domain. Only after this point, and with a couple of other assumptions, do we apply coherence arguments to show that it’s okay to use EUM and probability.
The reason I think this is worth talking about is that I was actively confused about exactly this topic in the year or two before I joined Vivek’s team. Re-reading the coherence and advanced agency cluster of Arbital posts (and a couple of comments from Nate) made me realise I had misinterpreted them. I must have thought they were intended to prove more than they do about AI risk. And this update flowed on to a few other things. Maybe partially because the next time I read Eliezer as saying something that seemed unreasonably strong I tried to steelman it and found a nearby reasonable meaning. And also because I had a clearer idea of the space of agents that are “allowed”, and this was useful for interpreting other arguments.
I’d be happy to call if that’s a more convenient way to talk, although it is nice to do this publicly. Also completely happy to stop talking about this if you aren’t interested, since I think your object-level beliefs about this ~match mine (“impure consequentialism” is expected of advanced AI).
E.g. I think we need a bunch of extra structure about self-modification to apply anything like a money pump argument to resolute/updateless agents. I think we need some non-trivial arguments and an assumption to make the VNM continuity money pump work. I remember there being some assumption that went into complete class that I thought was non-obvious, but I’ve forgotten exactly what it was. The post is very clear that it’s just giving a few tastes of the kind of reasoning needed to pin down utility and probability as a reasonable model of advanced agents.