I suspect that there might be a crux that’s something like: are future AIs more naturally oriented toward something like consequentialist reasoning or shaped cognition:
I think this is closer to a restatement of your / Dario’s position, rather than a crux. My claim is that it doesn’t matter whether specific future AIs are “naturally” consequentialists or something else, or how many degrees of freedom there are to be or not be a consequential and still get stuff done. Without bringing AI into it at all, we can already know (I claim, but am not really expanding on here), that consequentialism itself is extremely powerful, natural, optimal, etc. and there are some very general and deep lessons that we can learn from this. “There might be a way to build an AI without all that” or even “In practice that won’t happen by default given current training methods, at least for a while” could be true, but it wouldn’t change my position.
But if something like this is where Dario is coming from, then I wouldn’t say that the problem is that he has missed a bit about how the world works. It’s that he has noticed that current AI looks like it’d be based on shaped cognition if extrapolated further,
OK, sure.
and that there hasn’t been a strong argument for why it couldn’t be kept that way relatively straightforwardly.
Right, this is closer to where I disagree. I think there is a strong argument about this that doesn’t have anything to do with “shaped cognition” or even AI in particular.
On the other hand, expertise research finds that trying to do consequentialist reasoning in most established domains is generally error-prone and a mark of novices, and experts have had their cognition shaped to just immediately see the right thing and execute it. And people are generally not very consequentialist about navigating their lives and just do whatever everyone else does, and often this is actually a better idea than trying to figure out everything in your life from first principles.
I would flag this as exactly the wrong kind of lesson / example to learn something interesting about consequentialism—failure and mediocrity are overdetermined; it’s just not that interesting that there are particular contrived examples where some humans fail at applying consequentialism. Some of the best places to look for the deeper lessons and intuitions about consequentialism are environments where there is a lot of cut-throat competition, possibility for outlier success and failure, not artificially constrained or bounded in time or resources, etc.
I think this is closer to a restatement of your / Dario’s position, rather than a crux. My claim is that it doesn’t matter whether specific future AIs are “naturally” consequentialists or something else, or how many degrees of freedom there are to be or not be a consequential and still get stuff done. Without bringing AI into it at all, we can already know (I claim, but am not really expanding on here), that consequentialism itself is extremely powerful, natural, optimal, etc. and there are some very general and deep lessons that we can learn from this. “There might be a way to build an AI without all that” or even “In practice that won’t happen by default given current training methods, at least for a while” could be true, but it wouldn’t change my position.
OK, sure.
Right, this is closer to where I disagree. I think there is a strong argument about this that doesn’t have anything to do with “shaped cognition” or even AI in particular.
I would flag this as exactly the wrong kind of lesson / example to learn something interesting about consequentialism—failure and mediocrity are overdetermined; it’s just not that interesting that there are particular contrived examples where some humans fail at applying consequentialism. Some of the best places to look for the deeper lessons and intuitions about consequentialism are environments where there is a lot of cut-throat competition, possibility for outlier success and failure, not artificially constrained or bounded in time or resources, etc.