I’m surprised nobody has yet replied that the two examples are both products of significant optimizers with relevant optimization targets, and that the naive definition seems to work with one modification:
A system is downstream from an optimizer of some objective function to the extent that that objective function attains much higher values than would be attained if the system didn’t exist, or were doing some other random thing.
I’m surprised nobody has yet replied that the two examples are both products of significant optimizers with relevant optimization targets.
Yes, this seems pretty important and relevant.
That being said, I think that that definition suggests that natural selection and/or the earth’s crust are downstream from an optimiser of the number of Holiday Inns, or that my liver is downstream from an optimiser from my income, both of which aren’t right.
Probably it’s important to relate ‘natural subgoals’ to some ideal definition—which offers some hope, since ‘subgoal’ is really a computational notion, so maybe investigation along these lines would offer a more computational characterisation of optimisation.
[EDIT: I made this comment longer and more contentful]
Okay, so another necessary condition for being downstream from an optimizer is being causally downstream. I’m sure there are other conditions, but the claim still feels like an important addition to the conversation.
I’m surprised nobody has yet replied that the two examples are both products of significant optimizers with relevant optimization targets, and that the naive definition seems to work with one modification:
A system is downstream from an optimizer of some objective function to the extent that that objective function attains much higher values than would be attained if the system didn’t exist, or were doing some other random thing.
Yes, this seems pretty important and relevant.
That being said, I think that that definition suggests that natural selection and/or the earth’s crust are downstream from an optimiser of the number of Holiday Inns, or that my liver is downstream from an optimiser from my income, both of which aren’t right.
Probably it’s important to relate ‘natural subgoals’ to some ideal definition—which offers some hope, since ‘subgoal’ is really a computational notion, so maybe investigation along these lines would offer a more computational characterisation of optimisation.
[EDIT: I made this comment longer and more contentful]
Okay, so another necessary condition for being downstream from an optimizer is being causally downstream. I’m sure there are other conditions, but the claim still feels like an important addition to the conversation.