I always feel like this question is missing something, because it forgets the biggest cost for evolution: deploying the algorithms in the real world and seeing what effects they have.
That is, even if you had all the compute necessary to simulate human brains for all of evolutionary time, you would also need to figure out the effects of the brains on the real world in order to know their usefulness in practice. Doing this directly requires compute on the order of magnitude of the real world, though obviously in practice a lot of this can be optimized away.
Interesting, my first reaction was that evolution doesn’t need to “figure out” the extended phenotype (= “effects on the real world”) It just blindly deploys its algorithms, and natural selection does the optimization.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work. You’re right!
I should add: I think it is the hardest-to-compute aspects of this that are the most important to the evolution of general intelligence. With a “reasonable” compute budget, you could set up a gauntlet of small tasks that challenge your skills in various ways. However, this could probably be Goodharted relatively “easily”. But the real-world isn’t some closed short-term system; it also tests you against effects that take years or even decades to become relevant, just as hard as it tests you for effects that immediately become relevant. And that is something I think we will have a hard time with optimizing our AIs against.
No, I’m saying that AI might be much worse than us at long-term planning, because evolution has selected us on the basis of very long chains of causal effects, whereas we can’t train AIs on the basis of such long chains of causal effects.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work.
I always feel like this question is missing something, because it forgets the biggest cost for evolution: deploying the algorithms in the real world and seeing what effects they have.
That is, even if you had all the compute necessary to simulate human brains for all of evolutionary time, you would also need to figure out the effects of the brains on the real world in order to know their usefulness in practice. Doing this directly requires compute on the order of magnitude of the real world, though obviously in practice a lot of this can be optimized away.
Interesting, my first reaction was that evolution doesn’t need to “figure out” the extended phenotype (= “effects on the real world”) It just blindly deploys its algorithms, and natural selection does the optimization.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work. You’re right!
I should add: I think it is the hardest-to-compute aspects of this that are the most important to the evolution of general intelligence. With a “reasonable” compute budget, you could set up a gauntlet of small tasks that challenge your skills in various ways. However, this could probably be Goodharted relatively “easily”. But the real-world isn’t some closed short-term system; it also tests you against effects that take years or even decades to become relevant, just as hard as it tests you for effects that immediately become relevant. And that is something I think we will have a hard time with optimizing our AIs against.
You’re saying AI will be much better than us at long-term planning?
It’s hard to train for tasks where the reward is only known after a long time (e.g. how would you train for climate prediction?)
No, I’m saying that AI might be much worse than us at long-term planning, because evolution has selected us on the basis of very long chains of causal effects, whereas we can’t train AIs on the basis of such long chains of causal effects.
Yep.