I think it’s a good comparison, though I do think they’re importantly different. Evolution figured out how to make things that figure out how to figure stuff out. So you turn off evolution, and you still have an influx of new ability to figure stuff out, because you have a figure-stuff-out figure-outer. It’s harder to get the human to just figure stuff out without also figuring out more about how to figure stuff out, which is my point.
Tsvi appears to take the fact that you can stop gradient-descent without stopping the main operation of the NN to be evidence that the whole setup isn’t on a path to produce strong minds.
(I don’t see why it appears that I’m thinking that.) Specialized to NNs, what I’m saying is more like: If/when NNs make strong minds, it will be because the training—the explicit-for-us, distal ex quo—found an NN that has its own internal figure-stuff-out figure-outer, and then the figure-stuff-out figure-outer did a lot of figuring out how to figure stuff out, so the NN ended up with a lot of ability to figure stuff out; but a big chunk of the leading edge of that ability to figure stuff out came from the NN’s internal figure-stuff-out figure-outer, not “from the training”; so you can’t turn off the NN’s figure-stuff-out figure-outer just by pausing training. I’m not saying that the setup can’t find an NN-internal figure-stuff-out figure-outer (though I would be surprised if that happens with the exact architectures I’m aware of currently existing).
I think it’s a good comparison, though I do think they’re importantly different. Evolution figured out how to make things that figure out how to figure stuff out. So you turn off evolution, and you still have an influx of new ability to figure stuff out, because you have a figure-stuff-out figure-outer. It’s harder to get the human to just figure stuff out without also figuring out more about how to figure stuff out, which is my point.
(I don’t see why it appears that I’m thinking that.) Specialized to NNs, what I’m saying is more like: If/when NNs make strong minds, it will be because the training—the explicit-for-us, distal ex quo—found an NN that has its own internal figure-stuff-out figure-outer, and then the figure-stuff-out figure-outer did a lot of figuring out how to figure stuff out, so the NN ended up with a lot of ability to figure stuff out; but a big chunk of the leading edge of that ability to figure stuff out came from the NN’s internal figure-stuff-out figure-outer, not “from the training”; so you can’t turn off the NN’s figure-stuff-out figure-outer just by pausing training. I’m not saying that the setup can’t find an NN-internal figure-stuff-out figure-outer (though I would be surprised if that happens with the exact architectures I’m aware of currently existing).