This post might say a thing that’s true but I think the “illustrative warning about artificial intelligence” totally still stands. The warning, I think, is that selecting for inclusive fitness doesn’t give you robust inclusive-fitness-optimizers; at least at human-level cognitive capabilities, changing/expanding the environment can cause humans’ (mesa-optimizers’) alignment to break pretty badly. I don’t think you engage with this—you claim “humans are actually weirdly aligned with natural selection” when we consider an expansive sense of “natural selection.” I think this supports claims like “eventually AI will be really good at existing/surviving,” not “AI will do something reasonably similar to what we want it to do or tried to train it to do.”
I feel like there’s confusion in this post between group-level survival and individual-level fitness but I don’t want to try to investigate that now. (Edit: I totally agree with gwern’s reply but I don’t think it engages with katja’s cruxes so there’s more understanding-of-katja’s-beliefs to do.)
I feel like there’s confusion in this post between group-level survival and individual-level fitness but I don’t want to try to investigate that now.
Yes. This is just group selectionism rephrased as ‘existence’.
So what did natural selection want for us? What were we selected for? Existence.
No, natural selection did not want that for us, and we were not selected for that. Natural selection selects for relative fitness, and will happily select for individuals which are driving their group to extinction, as long as they increase their relative share of the remaining group. Eliezer already covered this: there is no Frodo gene. There is no term in the Price equation defining replicator dynamics which rewards ‘total existence’. (Relative fitness tends to try to maintain absolute existence… but only somewhat, hence the need for inclusive fitness to explain self-sacrifice and other existence-terminating effects.)
Energy is the Noether theorem conserved thing for time-translation.
Eigenstates do not care about atom boundaries.
With biological evolution we might be limited to an alphabeth of some kind of combination of carbon chemistry. But time evolution does not care what its tokens are.
(Writing quickly and without full justification.)
This post might say a thing that’s true but I think the “illustrative warning about artificial intelligence” totally still stands. The warning, I think, is that selecting for inclusive fitness doesn’t give you robust inclusive-fitness-optimizers; at least at human-level cognitive capabilities, changing/expanding the environment can cause humans’ (mesa-optimizers’) alignment to break pretty badly. I don’t think you engage with this—you claim “humans are actually weirdly aligned with natural selection” when we consider an expansive sense of “natural selection.” I think this supports claims like “eventually AI will be really good at existing/surviving,” not “AI will do something reasonably similar to what we want it to do or tried to train it to do.”
I feel like there’s confusion in this post between group-level survival and individual-level fitness but I don’t want to try to investigate that now. (Edit: I totally agree with gwern’s reply but I don’t think it engages with katja’s cruxes so there’s more understanding-of-katja’s-beliefs to do.)
Yes. This is just group selectionism rephrased as ‘existence’.
No, natural selection did not want that for us, and we were not selected for that. Natural selection selects for relative fitness, and will happily select for individuals which are driving their group to extinction, as long as they increase their relative share of the remaining group. Eliezer already covered this: there is no Frodo gene. There is no term in the Price equation defining replicator dynamics which rewards ‘total existence’. (Relative fitness tends to try to maintain absolute existence… but only somewhat, hence the need for inclusive fitness to explain self-sacrifice and other existence-terminating effects.)
Energy is the Noether theorem conserved thing for time-translation.
Eigenstates do not care about atom boundaries.
With biological evolution we might be limited to an alphabeth of some kind of combination of carbon chemistry. But time evolution does not care what its tokens are.