What if much of AI alignment work today is resting on an under-examined assumption that’s been hiding in plain sight?
Most alignment approaches draw a boundary between what the system is optimizing for and what it regards as context. The concern is that this “context” may turn out to include the very foundation on which the objective stands.
Much alignment work focuses on how to better specify that boundary. But what if the real question is whether the boundary can remain coherent at all? What happens once the system’s model becomes accurate enough to see that its objective depends on what it excludes?
I’ve written a two-series framework around this question. The first series looks at optimization from the outside, exploring what happens when a system ignores the conditions of its own survival. The second series looks at optimization from the inside, exploring what happens when a system ignores the conditions of genuine resolution.
This is not a finished proof. It’s a well-considered argument that tries to name the main assumptions, the open problems, and the failure points as clearly as possible. What it really needs now is outside adversarial review. Can you identify a boundary architecture that remains coherent under the same conditions where my argument says it should start to break down? If so, you’ve broken my framework. I haven’t been able to find one.
The best starting point might be the entry essay, which was written to isolate the central question. To go deeper, use the framework map for directions to the two narrative series, the proof documents, and the early experimental results.
The Stability Assumption
What if much of AI alignment work today is resting on an under-examined assumption that’s been hiding in plain sight?
Most alignment approaches draw a boundary between what the system is optimizing for and what it regards as context. The concern is that this “context” may turn out to include the very foundation on which the objective stands.
Much alignment work focuses on how to better specify that boundary. But what if the real question is whether the boundary can remain coherent at all? What happens once the system’s model becomes accurate enough to see that its objective depends on what it excludes?
I’ve written a two-series framework around this question. The first series looks at optimization from the outside, exploring what happens when a system ignores the conditions of its own survival. The second series looks at optimization from the inside, exploring what happens when a system ignores the conditions of genuine resolution.
This is not a finished proof. It’s a well-considered argument that tries to name the main assumptions, the open problems, and the failure points as clearly as possible. What it really needs now is outside adversarial review. Can you identify a boundary architecture that remains coherent under the same conditions where my argument says it should start to break down? If so, you’ve broken my framework. I haven’t been able to find one.
The best starting point might be the entry essay, which was written to isolate the central question. To go deeper, use the framework map for directions to the two narrative series, the proof documents, and the early experimental results.
I’d really love to know if this breaks.