I believe the traditional term is principal-agent problem. There’s a lot of econ research on the topic. Not sure if FAI has any insights at all to offer here.
Here is my thought on how this might relate to FAI.
If we build FAI, or a FAI modifies itself, “friendliness” will be
determined within a specific context. Outside of that context
friendliness can’t be guaranteed (or might be meaningless).
For example an industrial robot may be considered safe when used
within a cage designed to keep people out. But it might be very
dangerous if used unprotected in an area people may pass through.
To expand the contexts that an industrial robot is safe in, we might
have independent inspectors check the installation, or we might add
proximity sensors to the robot.
For a safer government we might create different branches of
government that can cross check each other.
Biological systems have genetic regulatory networks that can optimize
an organism for survival in the face of a changing environment.
What these approaches have in common is a distribution of cross
checking controls.
I think that advancing our knowledge on how to build these types of
regulatory networks is essential to creating practical FAI, and could
also help improve other types of systems.
I believe the traditional term is principal-agent problem. There’s a lot of econ research on the topic. Not sure if FAI has any insights at all to offer here.
Thanks for pointing out the prior art.
Here is my thought on how this might relate to FAI.
If we build FAI, or a FAI modifies itself, “friendliness” will be determined within a specific context. Outside of that context friendliness can’t be guaranteed (or might be meaningless).
For example an industrial robot may be considered safe when used within a cage designed to keep people out. But it might be very dangerous if used unprotected in an area people may pass through.
To expand the contexts that an industrial robot is safe in, we might have independent inspectors check the installation, or we might add proximity sensors to the robot.
For a safer government we might create different branches of government that can cross check each other.
Biological systems have genetic regulatory networks that can optimize an organism for survival in the face of a changing environment.
What these approaches have in common is a distribution of cross checking controls.
I think that advancing our knowledge on how to build these types of regulatory networks is essential to creating practical FAI, and could also help improve other types of systems.