That makes sense. But it isn’t what Eliezer says in that talk:
There’s a whole set of different ways we could look at agents, but as long as the agents are sufficiently advanced that we have pumped most of the qualitatively bad behavior out of them, they will behave as if they have coherent probability distributions and consistent utility functions.
Basically agree, and it’s nearly the same point I was trying to get at, though by less supposing utility functions are definitely the right thing. I’d leave open more possibility that we’re wrong about utility functions always being the best subclass of preference relations, but even if we’re wrong about that our solutions must at least work for utility functions, they being a smaller set of all possible ways something could decide.
That makes sense. But it isn’t what Eliezer says in that talk:
Do you disagree with him on that?
Basically agree, and it’s nearly the same point I was trying to get at, though by less supposing utility functions are definitely the right thing. I’d leave open more possibility that we’re wrong about utility functions always being the best subclass of preference relations, but even if we’re wrong about that our solutions must at least work for utility functions, they being a smaller set of all possible ways something could decide.