Selective optimization finding predictive optimizers (with a different objective) is the main idea of “Risks from Learned Optimisation” and indeed they have Section 2: Conditions for mesa-optimization
I mean, “selection pressure creates artefacts with learning/predictive capabiltiies” is also just how evolution works. It’s selective optimisation creating predictive optimisers all the way down: Even cells and nematodes have learning capabilities. What we humans see as our unique intelligence can be considered belonging to a long and storied genre of pathfinding and future-simulating behaviours, only now carried out in higher and higher dimensional action spaces—and at each step selection is used to grow and develop these capabilties. Given that, it seems reasonable to say that if evolution can be described as a coherent phenomenon at all, it will be a phenomenon that acts on intelligent goal-driven systems. (This comment comes from some notes I wrote down while reading the Moloch essay)
Selective optimization finding predictive optimizers (with a different objective) is the main idea of “Risks from Learned Optimisation” and indeed they have Section 2: Conditions for mesa-optimization
I mean, “selection pressure creates artefacts with learning/predictive capabiltiies” is also just how evolution works. It’s selective optimisation creating predictive optimisers all the way down: Even cells and nematodes have learning capabilities. What we humans see as our unique intelligence can be considered belonging to a long and storied genre of pathfinding and future-simulating behaviours, only now carried out in higher and higher dimensional action spaces—and at each step selection is used to grow and develop these capabilties. Given that, it seems reasonable to say that if evolution can be described as a coherent phenomenon at all, it will be a phenomenon that acts on intelligent goal-driven systems. (This comment comes from some notes I wrote down while reading the Moloch essay)