Great idea to write a story where the humans take the role of the AI. This allows to view the goals from the perspective of the AI and motivated via empthy with the AI search for ways at the edge of the constraint.
It makes clear that whatever the actual constraint are. If they limit the AI in any way than the AI will work at the very edge of it. Any limit that was conceived as a soft limit (e.g. maximize some aspect of well being) will become a hard limit (well being exactly as much as is balanced with the other rules).
Great idea to write a story where the humans take the role of the AI. This allows to view the goals from the perspective of the AI and motivated via empthy with the AI search for ways at the edge of the constraint.
It makes clear that whatever the actual constraint are. If they limit the AI in any way than the AI will work at the very edge of it. Any limit that was conceived as a soft limit (e.g. maximize some aspect of well being) will become a hard limit (well being exactly as much as is balanced with the other rules).