I think the really interesting interaction between these two frames is when selection pressures lead to predictive capacities. When does this happen? A first guess might be: when the training (selecting) environment is so complicated, and there is so much local variance that the selective loop finds its easiest to instill a predictive agent and let that take care of the local adaptation.
A lot of stuff works like this: you can have generic chess/math heuristics but you need to be able to do local calculations to not fall flat on your face; evolution more or less works like this in mammals and obviously humans, maybe much more; presumably LLMs work like this; our central nervous system/mind works like this wrt individual cells in the body.
Are there other factors that mediate how a selective process can give rise to local predictive agents? What consequences does this transition have? Cancer/parasites/fraud are three instances of one example, what else?
Selective optimization finding predictive optimizers (with a different objective) is the main idea of “Risks from Learned Optimisation” and indeed they have Section 2: Conditions for mesa-optimization
I mean, “selection pressure creates artefacts with learning/predictive capabiltiies” is also just how evolution works. It’s selective optimisation creating predictive optimisers all the way down: Even cells and nematodes have learning capabilities. What we humans see as our unique intelligence can be considered belonging to a long and storied genre of pathfinding and future-simulating behaviours, only now carried out in higher and higher dimensional action spaces—and at each step selection is used to grow and develop these capabilties. Given that, it seems reasonable to say that if evolution can be described as a coherent phenomenon at all, it will be a phenomenon that acts on intelligent goal-driven systems. (This comment comes from some notes I wrote down while reading the Moloch essay)
I think the really interesting interaction between these two frames is when selection pressures lead to predictive capacities. When does this happen? A first guess might be: when the training (selecting) environment is so complicated, and there is so much local variance that the selective loop finds its easiest to instill a predictive agent and let that take care of the local adaptation.
A lot of stuff works like this: you can have generic chess/math heuristics but you need to be able to do local calculations to not fall flat on your face; evolution more or less works like this in mammals and obviously humans, maybe much more; presumably LLMs work like this; our central nervous system/mind works like this wrt individual cells in the body.
Are there other factors that mediate how a selective process can give rise to local predictive agents? What consequences does this transition have? Cancer/parasites/fraud are three instances of one example, what else?
Selective optimization finding predictive optimizers (with a different objective) is the main idea of “Risks from Learned Optimisation” and indeed they have Section 2: Conditions for mesa-optimization
I mean, “selection pressure creates artefacts with learning/predictive capabiltiies” is also just how evolution works. It’s selective optimisation creating predictive optimisers all the way down: Even cells and nematodes have learning capabilities. What we humans see as our unique intelligence can be considered belonging to a long and storied genre of pathfinding and future-simulating behaviours, only now carried out in higher and higher dimensional action spaces—and at each step selection is used to grow and develop these capabilties. Given that, it seems reasonable to say that if evolution can be described as a coherent phenomenon at all, it will be a phenomenon that acts on intelligent goal-driven systems. (This comment comes from some notes I wrote down while reading the Moloch essay)