[EDIT: 2019-11-09: The argument I made here seems incorrect; see here (H/T Abram for showing me that my reasoning on this was wrong).]
If there’s a learning-theoretic setup which incentivizes the development of “full agency” (whatever that even means, really!) I don’t know what it is yet.
Consider evolutionary algorithms. It seems that (theoretically) they tend to yield non-myopic models by default given sufficiently long runtime. For example, a network parameter value that causes behavior that minimizes loss in future training inferences might be more likely to end up in the final model than one that causes behavior that minimizes loss in the current inference at a great cost for the loss in future ones.
That’s an interesting point, but I’m very skeptical that this effect is great enough to really hold in the long run. Toward the end, the parameter value could mutate away. And how would you apply evolutionary algorithms to really non-myopic settings, like reinforcement learning where you can’t create any good episode boundaries (for example, you have a robot interacting with an environment “live”, no resets, and you want to learn on-line)?
Toward the end, the parameter value could mutate away.
I agree that it’s possible to get myopic models in the population after arbitrarily long runtime due to mutations. It seems less likely the more bits that need to change—in any model in the current population—to get a model that in completely myopic.
From a safety perspective, if the prospect of some learning algorithm yielding a non-myopic model is concerning, the prospect of it creating non-myopic models along the way is plausibly also concerning.
And how would you apply evolutionary algorithms to really non-myopic settings, like reinforcement learning where you can’t create any good episode boundaries (for example, you have a robot interacting with an environment “live”, no resets, and you want to learn on-line)?
In this example, if we train an environment model on the data collected so far (ignoring the “you want to learn on-line” part), evolutionary algorithms might be an alternative to regular deep learning. More realistically, some actors would probably invest a lot of resources in developing top predictive models for stock prices etc., and evolutionary algorithms might be one of the approaches being experimented with.
Also, people might experiment with evolutionary algorithms as an alternative to RL, for environments that can be simulated, as OpenAI did; they wrote (2017): “Our work suggests that neuroevolution approaches can be competitive with reinforcement learning methods on modern agent-environment benchmarks, while offering significant benefits related to code complexity and ease of scaling to large-scale distributed settings.”.
I agree that it’s possible to get myopic models in the population after arbitrarily long runtime due to mutations. It seems less likely the more bits that need to change—in any model in the current population—to get a model that in completely myopic.
Yeah, I agree there’s something to think about here. The reason I responded as I did was because it seems more tractable to think about asymptotic results, ie, what happens if you run an algorithm to convergence. But ultimately we also need to think about what an algorithm does with realistic resources.
From a safety perspective, if the prospect of some learning algorithm yielding a non-myopic model is concerning, the prospect of it creating non-myopic models along the way is plausibly also concerning.
I’m not assuming partial agency is uniformly good or bad from a safety perspective; in some cases one, in some cases the other. If I want an algorithm to reliably produce full agency, I can’t use algorithms which produce some full-agency effects in the short term but which have a tendency to wash them out in the long term, unless I have a very clear understanding of when I’m in the short term.
In this example, if we train an environment model on the data collected so far (ignoring the “you want to learn on-line” part),
Ignoring the on-line part ignores too much, I think.
Also, people might experiment with evolutionary algorithms as an alternative to RL, for environments that can be simulated,
Similarly, assuming environments that can be simulated seems to assume too much. A simulatable environment gives us the ability to perfectly optimize from outside of that environment, allowing full optimization and hence full agency. So it’s not solving the problem I was talking about.
I think this is an example of selection/control confusion. Evolution is a selection algorithm, and can only be applied indirectly to control problems.
[EDIT: 2019-11-09: The argument I made here seems incorrect; see here (H/T Abram for showing me that my reasoning on this was wrong).]
Consider evolutionary algorithms. It seems that (theoretically) they tend to yield non-myopic models by default given sufficiently long runtime. For example, a network parameter value that causes behavior that minimizes loss in future training inferences might be more likely to end up in the final model than one that causes behavior that minimizes loss in the current inference at a great cost for the loss in future ones.
That’s an interesting point, but I’m very skeptical that this effect is great enough to really hold in the long run. Toward the end, the parameter value could mutate away. And how would you apply evolutionary algorithms to really non-myopic settings, like reinforcement learning where you can’t create any good episode boundaries (for example, you have a robot interacting with an environment “live”, no resets, and you want to learn on-line)?
I agree that it’s possible to get myopic models in the population after arbitrarily long runtime due to mutations. It seems less likely the more bits that need to change—in any model in the current population—to get a model that in completely myopic.
From a safety perspective, if the prospect of some learning algorithm yielding a non-myopic model is concerning, the prospect of it creating non-myopic models along the way is plausibly also concerning.
In this example, if we train an environment model on the data collected so far (ignoring the “you want to learn on-line” part), evolutionary algorithms might be an alternative to regular deep learning. More realistically, some actors would probably invest a lot of resources in developing top predictive models for stock prices etc., and evolutionary algorithms might be one of the approaches being experimented with.
Also, people might experiment with evolutionary algorithms as an alternative to RL, for environments that can be simulated, as OpenAI did; they wrote (2017): “Our work suggests that neuroevolution approaches can be competitive with reinforcement learning methods on modern agent-environment benchmarks, while offering significant benefits related to code complexity and ease of scaling to large-scale distributed settings.”.
Yeah, I agree there’s something to think about here. The reason I responded as I did was because it seems more tractable to think about asymptotic results, ie, what happens if you run an algorithm to convergence. But ultimately we also need to think about what an algorithm does with realistic resources.
I’m not assuming partial agency is uniformly good or bad from a safety perspective; in some cases one, in some cases the other. If I want an algorithm to reliably produce full agency, I can’t use algorithms which produce some full-agency effects in the short term but which have a tendency to wash them out in the long term, unless I have a very clear understanding of when I’m in the short term.
Ignoring the on-line part ignores too much, I think.
Similarly, assuming environments that can be simulated seems to assume too much. A simulatable environment gives us the ability to perfectly optimize from outside of that environment, allowing full optimization and hence full agency. So it’s not solving the problem I was talking about.
I think this is an example of selection/control confusion. Evolution is a selection algorithm, and can only be applied indirectly to control problems.