systems that have a tendency to evolve towards a narrow target configuration set when started from any point within a broader basin of attraction, and continue to do so despite perturbations.
When determining whether a system “optimizes” in practice, the heavy lifting is done by the degree to which the set of states that the system evolves toward—the suspected “target set”—feels like it forms a natural class to the observer.
The issue here is that what the observer considers “a natural class” is informed by the data-distribution that the observer has previously been exposed to.
It’s worse, even, in a certain way, than that: the existence of optimizing systems organized around a certain idea of “natural class” feeds back into more observers observing data that is distributed according to this idea of “natural class”, leading to more optimizing systems being built around that idea of “natural class”, and so on.
Once a certain idea of “natural class” gains a foothold somewhere, observers will make real changes in the world that further suggest this particular idea of “natural class” to others, and this forms a feedback loop.
When determining whether a system “optimizes” in practice, the heavy lifting is done by the degree to which the set of states that the system evolves toward—the suspected “target set”—feels like it forms a natural class to the observer.
The issue here is that what the observer considers “a natural class” is informed by the data-distribution that the observer has previously been exposed to.
It’s worse, even, in a certain way, than that: the existence of optimizing systems organized around a certain idea of “natural class” feeds back into more observers observing data that is distributed according to this idea of “natural class”, leading to more optimizing systems being built around that idea of “natural class”, and so on.
Once a certain idea of “natural class” gains a foothold somewhere, observers will make real changes in the world that further suggest this particular idea of “natural class” to others, and this forms a feedback loop.