Optimizing for the outcome metric alone on some training distribution, without any insight into the process producing that outcome, runs the risk that the system won’t behave as desired when out-of-distribution. This is probably a serious concern to the system maintainers, even ignoring (largely externalized) X-risks.
I understand Ivan’s first point. My main concern is that we don’t have the right processes laid out for these models to follow. In the end, we want these models to determine their own process of doing things (if we don’t find a way to emulate human brain processes into machines) and establishing a clear-cut process for tasks could limit the model’s creativity. We would have to have a perfect model of how each of these NN tasks should be carried out.
However, the idea of combining the two is interested. As research suggests, backprop and a global update function doesn’t exist in the brain (although large sections of the brain can carry out orchestrated tasks amazingly). There must be a combination of local updates to these synaptic weights (aligned with specific process-based tasks) which follow some global loss function in the brain. It’d be interesting to get more thoughts on this.
Optimizing for the outcome metric alone on some training distribution, without any insight into the process producing that outcome, runs the risk that the system won’t behave as desired when out-of-distribution. This is probably a serious concern to the system maintainers, even ignoring (largely externalized) X-risks.
I understand Ivan’s first point. My main concern is that we don’t have the right processes laid out for these models to follow. In the end, we want these models to determine their own process of doing things (if we don’t find a way to emulate human brain processes into machines) and establishing a clear-cut process for tasks could limit the model’s creativity. We would have to have a perfect model of how each of these NN tasks should be carried out.
However, the idea of combining the two is interested. As research suggests, backprop and a global update function doesn’t exist in the brain (although large sections of the brain can carry out orchestrated tasks amazingly). There must be a combination of local updates to these synaptic weights (aligned with specific process-based tasks) which follow some global loss function in the brain. It’d be interesting to get more thoughts on this.