Appreciate this post! I had seen the good regulator theorem referenced every now and then, but wasn’t sure what exactly the relevant claims were, and wouldn’t have known how to go through the original proof myself. This is helpful.
(E.g. the result was cited by Frith & Metzinger as part of their argument that, as an agent seeks to avoid being punished by society, this constitutes an attempt to regulate society’s behavior; and for the regulation be successful, the agent needs to internalize a model of the society’s preferences, which once internalized becomes something like a subagent which then regulates the agent in turn and causes behaviors such as self-punishment. It sounds like the math of the theorem isn’t very strongly relevant for that particular argument, though some form of the overall argument still sounds plausible to me regardless.)
Appreciate this post! I had seen the good regulator theorem referenced every now and then, but wasn’t sure what exactly the relevant claims were, and wouldn’t have known how to go through the original proof myself. This is helpful.
(E.g. the result was cited by Frith & Metzinger as part of their argument that, as an agent seeks to avoid being punished by society, this constitutes an attempt to regulate society’s behavior; and for the regulation be successful, the agent needs to internalize a model of the society’s preferences, which once internalized becomes something like a subagent which then regulates the agent in turn and causes behaviors such as self-punishment. It sounds like the math of the theorem isn’t very strongly relevant for that particular argument, though some form of the overall argument still sounds plausible to me regardless.)