evolution does not grow minds, it grows hyperparameters for minds.
Imo this is a nitpick that isn’t really relevant to the point of the analogy. Evolution is a good example of how selection for X doesn’t necessarily lead to a thing that wants (‘optimizes for’) X; and more broadly it’s a good example for how the results of an optimization process can be unexpected.
I want to distinguish two possible takes here:
The argument from direct implication: “Humans are misaligned wrt evolution, therefore AIs will be misaligned wrt their objectives”
Evolution as an intuition pump: “Thinking about evolution can be helpful for thinking about AI. In particular it can help you notice ways in which AI training is likely to produce AIs with goals you didn’t want”
It sounds like you’re arguing against (1). Fair enough, I too think (1) isn’t a great take in isolation. If the evolution analogy does not help you think more clearly about AI at all then I don’t think you should change your mind much on the strength of the analogy alone. But my best guess is that most people incl Nate mean (2).
> evolution does not grow minds, it grows hyperparameters for minds.
Imo this is a nitpick that isn’t really relevant to the point of the analogy. Evolution is a good example of how selection for X doesn’t necessarily lead to a thing that wants (‘optimizes for’) X; and more broadly it’s a good example for how the results of an optimization process can be unexpected.
I think it’s extremely relevant, if we want to ensure that we only analogize between processes which share enough causal structure to ensure that lessons from e.g. evolution actually carry over to e.g. AI training (due to those shared mechanisms). If the shared mechanisms aren’t there, then we’re playing reference class tennis because someone decided to call both processes “optimization processes.”
The argument I think is good (nr (2) in my previous comment) doesn’t go through reference classes at all. I don’t want to make an outside-view argument (eg “things we call optimization often produce misaligned results, therefore sgd is dangerous”). I like the evolution analogy because it makes salient some aspects of AI training that make misalignment more likely. Once those aspects are salient you can stop thinking about evolution and just think directly about AI.
Imo this is a nitpick that isn’t really relevant to the point of the analogy. Evolution is a good example of how selection for X doesn’t necessarily lead to a thing that wants (‘optimizes for’) X; and more broadly it’s a good example for how the results of an optimization process can be unexpected.
I want to distinguish two possible takes here:
The argument from direct implication: “Humans are misaligned wrt evolution, therefore AIs will be misaligned wrt their objectives”
Evolution as an intuition pump: “Thinking about evolution can be helpful for thinking about AI. In particular it can help you notice ways in which AI training is likely to produce AIs with goals you didn’t want”
It sounds like you’re arguing against (1). Fair enough, I too think (1) isn’t a great take in isolation. If the evolution analogy does not help you think more clearly about AI at all then I don’t think you should change your mind much on the strength of the analogy alone. But my best guess is that most people incl Nate mean (2).
I think it’s extremely relevant, if we want to ensure that we only analogize between processes which share enough causal structure to ensure that lessons from e.g. evolution actually carry over to e.g. AI training (due to those shared mechanisms). If the shared mechanisms aren’t there, then we’re playing reference class tennis because someone decided to call both processes “optimization processes.”
The argument I think is good (nr (2) in my previous comment) doesn’t go through reference classes at all. I don’t want to make an outside-view argument (eg “things we call optimization often produce misaligned results, therefore sgd is dangerous”). I like the evolution analogy because it makes salient some aspects of AI training that make misalignment more likely. Once those aspects are salient you can stop thinking about evolution and just think directly about AI.