Okay, that makes sense, I agree with that. As I mentioned in the opinion, I definitely agree with this in the case of a superintelligence optimizing a utility function. Probably my mindset when writing that opinion was that it seems likely to me that AI systems that we actually deploy won’t look like a single agent optimizing for particular preferences, for reasons that I couldn’t really articulate. I still have this intuition, and I think I’m closer to being able to explain it now, but not in a comment.
Okay, that makes sense, I agree with that. As I mentioned in the opinion, I definitely agree with this in the case of a superintelligence optimizing a utility function. Probably my mindset when writing that opinion was that it seems likely to me that AI systems that we actually deploy won’t look like a single agent optimizing for particular preferences, for reasons that I couldn’t really articulate. I still have this intuition, and I think I’m closer to being able to explain it now, but not in a comment.