Really enjoyed this post, thank you for making the original theorem so much more digestible!
I’m still learning to think in this formal direction, so I wanted to test my understanding by applying it to a domain I know well: legal AI governance in the EU.
Here’s the framing I tried:
R (Regulator) = The European Commission, AI Office, and adjacent regulatory bodies tasked with steering AI development, risk mitigation, and commercialisation through laws and standards.
S(System) = The entire AI development landscape affecting Europe—including global developments, given the Brussels effect and the extraterritorial pull of EU regulations.
Z(Outcome) = The observable effects on society, markets, and risk distribution from AI systems (same as in the original formulation).
I’m currently researching how foundational AI safety concepts can be adapted for regulatory design in the EU context. So this is mostly an exercise to train my brain, not a new claim.
Some of this may look misguided to those more familiar with the theorem, very happy to be corrected!!
---
The Problem with Perfect Knowledge of S
In this setup, it’s immediately clear that R (the EU regulator) cannot meet the assumption of perfect knowledge of S. In fact, its information about the state of AI development is often:
Partial (relying on self-assessments and opaque model disclosures),
Lagging (laws are drafted over years, breakthroughs happen monthly), and
Politically filtered (due to industrial policy, lobbying, and differing risk appetites across member states).
The theorem assumes that R can deterministically respond to S. But if S is the global AI ecosystem, then this is not just untrue, but structurally impossible.
What Can Be Done?
Regulators can’t minimize entropy in Z if they lack observability into S. The legal process is deterministic by design, but the system it tries to regulate (S) is non-deterministic, high-velocity, and partially adversarial.
The best R can do is build probabilistic regulators: laws and procedures designed for bounded uncertainty rather than perfect foresight.
This might involve:
Scalable oversight institutions (analogous to alignment’s scalable supervision),
Simulated models or red-team stress tests to explore likely behaviors of S under different policy levers,
Periodic re-anchoring of regulatory frameworks to model drift and emerging capabilities (though the current legislative cycle struggles to keep up).
It seems to me that unless institutional capacity is built to approximate S more closely, entropy in Z is baked in, and the Regulator is more likely than not to always fail the basic standard of being “good” in the Conant/Ashby sense.
Curious to hear if this sort of mapping holds water. I realize it’s informal and a bit hand-wavy, but I’m trying to think more rigorously about why legal regimes struggle to govern advanced AI systems, and this theorem felt like a useful test case.
Would appreciate any pushback or corrections from those with stronger formal grounding.
Really enjoyed this post, thank you for making the original theorem so much more digestible!
I’m still learning to think in this formal direction, so I wanted to test my understanding by applying it to a domain I know well: legal AI governance in the EU.
Here’s the framing I tried:
R (Regulator) = The European Commission, AI Office, and adjacent regulatory bodies tasked with steering AI development, risk mitigation, and commercialisation through laws and standards.
S (System) = The entire AI development landscape affecting Europe—including global developments, given the Brussels effect and the extraterritorial pull of EU regulations.
Z (Outcome) = The observable effects on society, markets, and risk distribution from AI systems (same as in the original formulation).
I’m currently researching how foundational AI safety concepts can be adapted for regulatory design in the EU context. So this is mostly an exercise to train my brain, not a new claim.
Some of this may look misguided to those more familiar with the theorem, very happy to be corrected!!
---
The Problem with Perfect Knowledge of S
In this setup, it’s immediately clear that R (the EU regulator) cannot meet the assumption of perfect knowledge of S. In fact, its information about the state of AI development is often:
Partial (relying on self-assessments and opaque model disclosures),
Lagging (laws are drafted over years, breakthroughs happen monthly), and
Politically filtered (due to industrial policy, lobbying, and differing risk appetites across member states).
The theorem assumes that R can deterministically respond to S. But if S is the global AI ecosystem, then this is not just untrue, but structurally impossible.
What Can Be Done?
Regulators can’t minimize entropy in Z if they lack observability into S. The legal process is deterministic by design, but the system it tries to regulate (S) is non-deterministic, high-velocity, and partially adversarial.
The best R can do is build probabilistic regulators: laws and procedures designed for bounded uncertainty rather than perfect foresight.
This might involve:
Scalable oversight institutions (analogous to alignment’s scalable supervision),
Simulated models or red-team stress tests to explore likely behaviors of S under different policy levers,
Periodic re-anchoring of regulatory frameworks to model drift and emerging capabilities (though the current legislative cycle struggles to keep up).
It seems to me that unless institutional capacity is built to approximate S more closely, entropy in Z is baked in, and the Regulator is more likely than not to always fail the basic standard of being “good” in the Conant/Ashby sense.
Curious to hear if this sort of mapping holds water. I realize it’s informal and a bit hand-wavy, but I’m trying to think more rigorously about why legal regimes struggle to govern advanced AI systems, and this theorem felt like a useful test case.
Would appreciate any pushback or corrections from those with stronger formal grounding.