Co-founder @ Gladstone AI.
Contact: edouard@gladstone.ai
Website: eharr.is
Edouard Harris
Karma: 597
Well said. The argument in this post is the exact reason Jer and I thought the Apollo founders (Marius and Lee) were so intriguing when we met them in early 2023. They, almost alone, were able to state a compact hypothesis that underpinned a line of attack that had a shot at directly solving alignment. That was something we’d only rarely if ever seen done before.
The way I’d phrase the Apollos’ hypothesis would be: “Deception is fundamentally more computationally expensive to execute that honesty, because a deceptive entity needs to keep at least two sets of mental books (the truth and the lie(s)) while an honest entity only needs to keep one.” If that hypothesis is true, and if we can confirm that it’s true with high confidence, then we may indeed be able to build deception detectors that are 100% reliable across a relevant set of scenarios. That in turn would be a sufficient condition to train for honesty in a way that scales to superhuman intelligence—either across a defined subset of scenarios, or in full generality.
Now of course each of those subproblems (confirm that property of deception is true; use it to build a perfect deception detector; implement honesty training signal; …) is itself very hard, and the underlying hypothesis could turn out to be false. But you can see that this is at least a more interesting problem decomposition than “use GPT-n to align GPT-(n + 1)”. It’s more like someone postulating a new physical law and trying to build a particle accelerator to demonstrate it, than like the kind of deus ex machina one sees in perpetual motion and most alignment schemes. The physical law might turn out not be there, or it might only manifest at energies way higher than you can ever achieve with a particle accelerator, but you’re at least trying to discover something nontrivially interesting—figure out if you can bell the cat while it sleeps.
We never articulated it this way, but this is a major reason we thought Apollo could be promising very soon after they launched. Thought that might be worth mentioning here for the (possible) positive example, if nothing else.
Credit where credit is due, incidentally: the biggest single inflection point for this phenomenon was clearly Situational Awareness. Almost zero reporting in the mainstream news; yet by the end of 2024, everyone in the relevant spaces had read & absorbed it.
Correct & timely. There do exist margins where honesty and effectiveness trade off against each other, but today—in 2025, that is—this is no longer one of them. Your SB 1047 friend is quite right to suggest that things were different in 2023, though. The amount of resistance we got behind the scenes in trying to get words like “extinction” and even “AGI” (!!) published in our report (which was mostly written in 2023) was something to behold: back then you could only push the envelope so far before it became counterproductive. No longer. The best metaphor I’ve seen for what’s happening right now in government circles is a kettle of boiling water: pockets of people who get it are coalescing and finding each other; entire offices are now AGI-pilled, where in 2023 you’d be lucky to find a single individual among a phalanx of skeptics.
“The time is ripe” indeed.
Yeah that could be doable. Dylan’s pretty natsec focused already so I would guess he’d take a broad view of the ROI from something like this. From what I hear he is already in touch with some of the folks who are in the mix, which helps, but the core goal is to get random leaf node action officers this access with minimum friction. I think an unconditional discount to all federal employees probably does pass muster with the regs, though of course folks would still be paying something out of pocket. I’ll bring this up to SA next time we talk to them though, it might move the needle. For all I know, they might even be doing it already.
Because of another stupid thing, which is that U.S. depts & agencies have strong internal regs against employees soliciting and/or accepting gifts other than in carefully carved out exceptional cases. For more on this, see, e.g., 5 CFR § 2635.204, but this isn’t the only such reg. In practice U.S. government employees at all levels are broadly prohibited from accepting any gift with a market value above 20 USD for example. (As you’d expect this leads to a lot of weird outcomes, including occasional hilarious minor diplomatic incidents with inexperienced foreign counterparties who have different gift giving norms.)
Yep, can confirm this is true. And this often leads to shockingly stupid outcomes, such as key action officers at the Office of [redacted] in the Department of [redacted] not reading SemiAnalysis because they’d have to pay for their subscriptions out of pocket.
This is a great & timely post.
Thanks very much for writing this. We appreciate all the feedback across the board, and I think this a well done and in-depth write up.
On the specific numerical thresholds in the report (i.e., your Key Proposal section), I do need to make one correction that also applies to most of Brooks’s commentary. All the numerical thresholds mentioned in the report, and particularly in that subsection, are solely examples and not actual recommendations. They are there only to show how one can calculate self-consistent licensing thresholds under the principles we recommend. They are not themselves recommendations. We had to do it this way for the same reason we propose granting fairly broad rule-setting flexibility to the regulatory entity. The field is changing so quickly that any concrete threshold risks being out of date (for one reason or the other) in very short order. We would have liked to do otherwise, but that is not a realistic expectation for a report that we expect to be digested over the course of several months.
To avoid precisely this misunderstanding, the report states in several places that those very numbers are, in fact, only examples for illustration. A few screencaps of those disclaimers are below, but there are several others. Of course we could have included even more, but beyond a certain point one is simply adding more length to what you correctly point out is already quite a sizeable document. Note that the Time article, in the excerpt you quoted, does correctly note and acknowledge that the Tier 3 AIMD threshold is there as an example (emphasis added):
the report suggests, as an example, that the agency could set it just above the levels of computing power used to train current cutting-edge models like OpenAI’s GPT-4 and Google’s Gemini.
Apart from this, I do think overall you’ve done a good and accurate job of summarizing the document and offering sensible and welcome views, emphasis, and pushback. It’s certainly a long report, so this is a service to anyone who’s looking to go one or two levels deeper than the Executive Summary. We do appreciate you giving it a look and writing it up.
Gotcha, that makes sense!
Looks awesome! Minor correction on the cost of the GPT-4 training run: the website says $40 million, but sama confirmed publicly that it was over $100M (and several news outlets have reported the latter number as well).
Done, a few days ago. Sorry thought I’d responded to this comment.
Excellent context here, thank you. I hadn’t been aware of this caveat.
Inverse scaling can become U-shaped
POWERplay: An open-source toolchain to study AI power-seeking
Great question. This is another place where our model is weak, in the sense that it has little to say about the imperfect information case. Recall that in our scenario, the human agent learns its policy in the absence of the AI agent; and the AI agent then learns its optimal policy conditional on the human policy being fixed.
It turns out that this setup dodges the imperfect information question from the AI side, because the AI has perfect information on all the relevant parts of the human policy during its training. And it dodges the imperfect information question from the human side, because the human never considers even the existence of the AI during its training.
This setup has the advantage that it’s more tractable and easier to reason about. But it has the disadvantage that it unfortunately fails to give a fully satisfying answer to your question. It would be interesting to see if we can remove some of the assumptions in our setup to approximate the imperfect information case.
Agreed. We think our human-AI setting is a useful model of alignment in the limit case, but not really so in the transient case. (For the reason you point out.)
I think you might have reversed the definitions of and in your comment,[1] but otherwise I think you’re exactly right.
To compute (the correlation coefficient between terminal values), naively you’d have reward functions and , that respectively assign human and AI rewards over every possible arrangement of matter . Then you’d look at every such reward function pair over your joint distribution , and ask how correlated they are over arrangements of matter. If you like, you can imagine that the human has some uncertainty around both his own reward function over houses, and also over how well aligned the AI is with his own reward function.
And to compute (the correlation coefficient between instrumental values), you’re correct that some of the arrangements of matter will be intermediate states in some construction plans. So if the human and AI both want a house with a swimming pool, they will both have high POWER for arrangements of matter that include a big hole dug in the backyard. Plot out their respective POWERs at each , and you can read the correlation right off the alignment plot!
- ^
Looking again at the write-up, it would have made more sense for us to define as the terminal goal correlation coefficient, since we introduce that one first. Alas, this didn’t occur to us. Sorry for the confusion.
- ^
Good question. Unfortunately, one weakness of our definition of multi-agent POWER is that it doesn’t have much useful to say in a case like this one.
We assume AI learning timescales vastly outstrip human learning timescales as a way of keeping our definition tractable. So the only way to structure this problem in our framework would be to imagine a human is playing chess against a superintelligent AI — a highly distorted situation compared to the case of two roughly equal opponents.
On the other hand, from other results I’ve seen anecdotally, I suspect that if you gave one of the agents a purely random policy (i.e., take a random legal action at each state) and assigned the other agent some reasonable reward function distribution over material, you’d stand a decent chance of correctly identifying high-POWER states with high-mobility board positions.
You might also be interested in this comment by David Xu, where he discusses mobility as a measure of instrumental value in chess-playing.
Thanks for you comment. These are great questions. I’ll do the best I can to answer here, feel free to ask follow-ups:
On pre-committing as a negotiating tactic: If I’ve understood correctly, this is a special case of the class of strategies where you sacrifice some of your own options (bad) to constrain those of your opponent (good). And your question is something like: which of these effects is strongest, or do they cancel each other out?
It won’t surprise you that I think the answer is highly context-dependent, and that I’m not sure which way it would actually shake out in your example with Fred and Bob and the $5000. But interestingly, we did in fact discover an instance of this class of “sacrificial” strategies in our experiments!
You can check out the example in Part 3 if you’re interested. But briefly, what happens is that when the agents get far-sighted enough, one of them realizes that there is instrumental value in having the option to bottle up the other agent in a dead-end corridor (i.e., constraining that other agent’s options). But it can only actually do this by positioning itself at the mouth of the corridor (i.e., sacrificing its own options). Here is a full-size image of both agents’ POWERs in this situation. You can see from the diagram that Agent A prefers to preserve its own options over constraining Agent H’s options in this case. But crucially, Agent A values the option of being able to constrain Agent H’s options.
In the language of your negotiating example, there is instrumental value in preserving one’s option to pre-commit. But whether actually pre-committing is instrumentally valuable or not depends on the context.
On babies being more powerful than adults: Yes, I think your reasoning is right. And it would be relatively easy to do this experiment! All you’d need would be to define a “death” state, and set your transition dynamics so that the agent gets sent to the “death” state after N turns and can never escape from it afterwards. I think this would be a very interesting experiment to run, in fact.
On paperclip maximizers: This is a very deep and interesting question. One way to think about this schematically might be: a superintelligent paperclip maximizer will go through a Phase One, in which it accumulates its POWER; and then a Phase Two in which it spends the POWER it’s accumulated. During the accumulation phase, the system might drive towards a state where (without loss of generality) the Planet Earth is converted into a big pile of computronium. This computronium-Earth state is high-POWER, because it’s a common “way station” state for paperclip maximizers, thumbtack maximizers, safety pin maximizers, No. 2 pencil maximizers, and so on. (Indeed, this is what high POWER means.)
Once the system has the POWER it needs to reach its final objective, it will begin to spend that POWER in ways that maximize its objective. This is the point at which the paperclip, thumbtack, safety pin, and No. 2 pencil maximizers start to diverge from one another. They will each push the universe towards sharply different terminal states, and the more progress each maximizer makes towards its particular terminal state, the fewer remaining options it leaves for itself if its goal were to suddenly change. Like a male praying mantis, a maximizer ultimately sacrifices its whole existence for the pursuit of its terminal goal. In other words: zero POWER should be the end state of a pure X-maximizer![1]
My story here is hypothetical, but this is absolutely an experiment on can do (at small scale, naturally). The way to do it would be to run several rollouts of an agent, and plot the POWER of the agent at each state it visits during the rollout. Then we can see whether most agent trajectories have the property where their POWER first goes up (as they, e.g., move to topological junction points) and then goes down (as they move from the junction points to their actual objectives).
Thanks again for your great questions. Incidentally, a big reason we’re open-sourcing our research codebase is to radically lower the cost of converting thought experiments like the above into real experiments with concrete outcomes that can support or falsify our intuitions. The ideas you’ve suggested are not only interesting and creative, they’re also cheaply testable on our existing infrastructure. That’s one reason we’re excited to release it!
- ^
Note that this assumes the maximizer is inner aligned to pursue its terminal goal, the terminal goal is stable on reflection, and all the usual similar incantations.
I’m not sure if they discussed this approach at length anywhere in public, and your tabular Q-learner is indeed a limit case counterexample. My main point wasn’t that this was definitely or even likely to work, just that it seemed qualitatively more promising than the “AIs will do it for us” approaches that were prevalent at the time and are predictably starting to break down now. For what it’s worth, they also proactively brought up the need to track computations offloaded by the system (e.g., use of a calculator or third party tools) since they saw this as one possible vector of amortization/dispersion/externalization of deceptive cognition. That was something else we found encouraging.
I guess I tend to think of early ideas like these less as being about “could this work” and more as being about “is this in the vicinity of something that could work in the future”. For example, it might be impossible to catch every instance of deception (or even to fully define the concept usefully), but might be possible instead to define a class of models such that deceptive thoughts originating from those models fall into a usefully defined and detectable class. This would still be fundamental, just fundamental to that particular class of systems.