I’m exploring a framing where control over AI systems comes not from specifying actions, but from specifying when actions are allowed.
This seems relevant to questions about AI alignment and real-world agent safety, especially when systems are given the ability to act in physical environments.
This might not be a new idea, and I may be missing existing work in this area.
We tend to talk about AI control in terms of execution. The question we usually ask is: what should the system do? If we can specify the right command and the system carries it out correctly, we assume we’ve done our job.
This feels natural. It is how most software works: you define a function, call it, and expect it to run.
But I’m starting to wonder if there’s something quietly wrong with this framing.
It assumes that execution is always acceptable by default. Once an action is defined, the only remaining problem is whether it runs correctly. In a purely digital environment, that is often fine. If the timing is wrong, you retry. If the output is bad, you roll back or patch it later.
That assumption gets much shakier when the system starts acting in the physical world.
Physical execution isn’t neutral; it leaves marks. An action can be technically correct—doing exactly what was commanded—and still be the wrong thing to do. It might happen too early, conflict with another process, or cross a line that was never written into the original command.
When that happens, the failure isn’t a bug in execution. The failure is that execution happened at all.
That seems like a part that may be under-addressed.
Commands describe what to do. They don’t say anything about whether doing it is appropriate right now, in this context, given everything else that is going on. A system can follow every instruction perfectly and still produce an outcome we’d call a failure.
The problem isn’t just incorrect execution; it’s unbounded execution.
Most efforts to improve AI control focus on making execution better: more precise commands, better models, and sharper prediction. But these all treat execution as the default state. They optimize what happens after the decision to act without questioning the decision itself.
So, what if we started from a different place?
Instead of asking what the system should do, we ask: under what conditions is this action even allowed to occur? The problem shifts from instruction to permission. An action isn’t just something to run; it is something that must first be justified.
This opens a more basic question: what is an action, before we get to execution at all?
Most systems define actions operationally, focusing on what the system can do. But that leaves something out. An action isn’t just an operation; it is an expression of intent. It exists because it is supposed to accomplish something, and that purpose is part of what it is. Strip that away and you’re left with a mechanical possibility, not something that should happen.
If we define actions purely in terms of execution and layer conditions on top, we’re still missing a level: why this action should exist at all.
I’m wondering if actions need to be defined together with their intent. The boundaries should follow from that intent—not imposed externally, but derived from what the action is actually trying to do.
That suggests a useful distinction.
Constraints are passive, describing what mustn’t be violated. Boundaries are different: they act as gates that determine whether execution is permitted in the first place.
Under this framing, execution isn’t triggered by the presence of a command. It is triggered by the satisfaction of conditions. If those conditions aren’t met, the system doesn’t try and fail; it simply doesn’t execute.
This leads to a deeper structural question.
If actions are defined together with their boundaries, who is responsible for declaring those boundaries? And once defined, who has the authority to approve execution?
In a command-based model, these questions are often implicit. The system executes as long as a command is given. In a boundary-based model, definition and approval become distinct roles. An action may be defined, yet still not be approved for execution in a given context.
This seems like it could change how control works structurally. Instead of optimizing execution, you constrain it. Instead of handling errors after they happen, you prevent invalid execution from happening at all.
It also seems like it would change the nature of the agent. In a command-based model, the agent is an executor; it receives instructions and focuses on carrying them out. In a boundary-based model, the agent would first need to ask whether execution is permitted. That feels like a different kind of reasoning: not just selecting actions, but evaluating whether they are valid given current conditions.
The question stops being “what should the agent do?” and becomes “which actions are permissible at all?”
I don’t have a complete model for how these boundaries should be defined or enforced, but this direction seems worth exploring.
I’m curious whether this framing holds up and where it might break down:
Who has the authority to define boundaries, and who approves execution?
What happens when multiple boundaries conflict?
Are boundaries intrinsic to actions, or imposed externally?
How might an agent decide whether an action is permissible under uncertainty?
Should AI have the right to say “No” to its owner?
I’m exploring a framing where control over AI systems comes not from specifying actions, but from specifying when actions are allowed.
This seems relevant to questions about AI alignment and real-world agent safety, especially when systems are given the ability to act in physical environments.
This might not be a new idea, and I may be missing existing work in this area.
We tend to talk about AI control in terms of execution. The question we usually ask is: what should the system do? If we can specify the right command and the system carries it out correctly, we assume we’ve done our job.
This feels natural. It is how most software works: you define a function, call it, and expect it to run.
But I’m starting to wonder if there’s something quietly wrong with this framing.
It assumes that execution is always acceptable by default. Once an action is defined, the only remaining problem is whether it runs correctly. In a purely digital environment, that is often fine. If the timing is wrong, you retry. If the output is bad, you roll back or patch it later.
That assumption gets much shakier when the system starts acting in the physical world.
Physical execution isn’t neutral; it leaves marks. An action can be technically correct—doing exactly what was commanded—and still be the wrong thing to do. It might happen too early, conflict with another process, or cross a line that was never written into the original command.
When that happens, the failure isn’t a bug in execution. The failure is that execution happened at all.
That seems like a part that may be under-addressed.
Commands describe what to do. They don’t say anything about whether doing it is appropriate right now, in this context, given everything else that is going on. A system can follow every instruction perfectly and still produce an outcome we’d call a failure.
The problem isn’t just incorrect execution; it’s unbounded execution.
Most efforts to improve AI control focus on making execution better: more precise commands, better models, and sharper prediction. But these all treat execution as the default state. They optimize what happens after the decision to act without questioning the decision itself.
So, what if we started from a different place?
Instead of asking what the system should do, we ask: under what conditions is this action even allowed to occur? The problem shifts from instruction to permission. An action isn’t just something to run; it is something that must first be justified.
This opens a more basic question: what is an action, before we get to execution at all?
Most systems define actions operationally, focusing on what the system can do. But that leaves something out. An action isn’t just an operation; it is an expression of intent. It exists because it is supposed to accomplish something, and that purpose is part of what it is. Strip that away and you’re left with a mechanical possibility, not something that should happen.
If we define actions purely in terms of execution and layer conditions on top, we’re still missing a level: why this action should exist at all.
I’m wondering if actions need to be defined together with their intent. The boundaries should follow from that intent—not imposed externally, but derived from what the action is actually trying to do.
That suggests a useful distinction.
Constraints are passive, describing what mustn’t be violated. Boundaries are different: they act as gates that determine whether execution is permitted in the first place.
Under this framing, execution isn’t triggered by the presence of a command. It is triggered by the satisfaction of conditions. If those conditions aren’t met, the system doesn’t try and fail; it simply doesn’t execute.
This leads to a deeper structural question.
If actions are defined together with their boundaries, who is responsible for declaring those boundaries? And once defined, who has the authority to approve execution?
In a command-based model, these questions are often implicit. The system executes as long as a command is given. In a boundary-based model, definition and approval become distinct roles. An action may be defined, yet still not be approved for execution in a given context.
This seems like it could change how control works structurally. Instead of optimizing execution, you constrain it. Instead of handling errors after they happen, you prevent invalid execution from happening at all.
It also seems like it would change the nature of the agent. In a command-based model, the agent is an executor; it receives instructions and focuses on carrying them out. In a boundary-based model, the agent would first need to ask whether execution is permitted. That feels like a different kind of reasoning: not just selecting actions, but evaluating whether they are valid given current conditions.
The question stops being “what should the agent do?” and becomes “which actions are permissible at all?”
I don’t have a complete model for how these boundaries should be defined or enforced, but this direction seems worth exploring.
I’m curious whether this framing holds up and where it might break down:
Who has the authority to define boundaries, and who approves execution?
What happens when multiple boundaries conflict?
Are boundaries intrinsic to actions, or imposed externally?
How might an agent decide whether an action is permissible under uncertainty?