One of the earliest records of a hierarchical organization comes from the Bible (Exodus 18). Basically, Moses starts out completely “in touch with reality,” judging all disputes among the Israelites from minor to severe, from dawn until dusk. His father in law, Jethro, notices that he is getting burnt out, so he gives him some advice on dividing up the load:
You will surely wear yourself out, as well as these people who are with you, because the task is too heavy for you. You cannot do it alone, by yourself. Now listen to my voice—I will give you advice.… You should seek out capable men out of all the people—men who fear God, men of truth, who hate bribery. Appoint them to be rulers over thousands, hundreds, fifties and tens. Let them judge the people all the time. Then let every major case be brought to you, but every minor case they can judge for themselves. Make it easier for yourself, as they bear the burden with you.
It seems that in a system like this, all levels of the managerial (judicial) hierarchy stay in touch with reality. The only difference between management levels is that deeper levels require deeper wisdom and greater competence at assessing decisions at the “widget level” (or at least greater willingness to accept responsibility for bad decisions). I wonder if a similar strategy could help mitigate some of the failures you pointed out.
Relatedly, in deep learning, ResNets use linear skip connections to expose otherwise deeply hidden layers to the gradient signal (and to the input features) more directly. It tends to make training more stable and faster to converge while still taking advantage of the computational power of a hierarchical model. Obviously, this won’t prevent Goodharting in an RL system, but I would say that it does help keep models more internally cooperative.
One of the earliest records of a hierarchical organization comes from the Bible (Exodus 18). Basically, Moses starts out completely “in touch with reality,” judging all disputes among the Israelites from minor to severe, from dawn until dusk. His father in law, Jethro, notices that he is getting burnt out, so he gives him some advice on dividing up the load:
It seems that in a system like this, all levels of the managerial (judicial) hierarchy stay in touch with reality. The only difference between management levels is that deeper levels require deeper wisdom and greater competence at assessing decisions at the “widget level” (or at least greater willingness to accept responsibility for bad decisions). I wonder if a similar strategy could help mitigate some of the failures you pointed out.
Relatedly, in deep learning, ResNets use linear skip connections to expose otherwise deeply hidden layers to the gradient signal (and to the input features) more directly. It tends to make training more stable and faster to converge while still taking advantage of the computational power of a hierarchical model. Obviously, this won’t prevent Goodharting in an RL system, but I would say that it does help keep models more internally cooperative.
Interesting (although I do think “judge” is a fairly different job than “manager”, so I’d expect fairly different dynamics.