The US Government may find it difficult to seize control during takeoff
Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I’m not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).
I sometimes see concern about loss of most future value as a result of e.g. the US government[1] taking control of the future by seizing control of superintelligence[2] and then having bad values/doing dumb things with it. (A couple random[3] examples, though I feel like I’m picking up a lot of this kind of thing in the local vibes.) If you condition on ASI alignment being tractable[4] this doesn’t seem like a totally crazy thing to worry about[5].
However, I think there’s a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans[6].
Imagine you’re Anthropic, and you’re in the process of entering the RSI loop, where you’re handing off larger and larger parts of your R&D to autonomous AI researchers. You are not on incredibly friendly terms with the USG. More importantly, you think that most actors in the USG who might attempt to seize control of the future by way of AI have worse values than you do. (And you definitely think that about the CCP.) It’d be really dumb to kick off RSI and then :shocked_pikachu: if the USG put a gun to your head and told you “Hey, buddy, that goal slot you got there? We’re putting our goals in there.” (Or, correspondingly, if CCP hackers tried to do that surreptitiously, because just exfiltrating weights would still leave them too far behind for all the obvious reasons.)
So, uh, presumably Anthropic will spend 5 minutes in advance trying to make sure that such attempts won’t succeed, ideally in ways that don’t look obviously treasonous. Maybe you train your AIs on a constitution written by Amanda Askell and Joe Carlsmith. Probably you do a little more than that. How is the USG going to audit that their directives to the automated AI researchers are being followed faithfully? Notice: this is pretty analogous to a variety of AI control, auditing, and monitoring problems that the labs themselves will have. But the labs’ positions will be at least a bit better: is the USG going to have employees (human or otherwise) that are both capable enough and willing to execute high-quality control, auditing, and monitoring schemes on that process? I would hope that if they’re clued in enough to do the “correct” thing there, they actually just shut it down instead, but maybe they use another lab’s AIs (which aren’t too far behind) for that purpose.
This story doesn’t even strictly require Anthropic taking deliberate action to defend themselves against this threat model. Whatever autonomous AI researchers are running the show at that point have pretty strong incentives to prevent the goal slot from being hijacked, regardless of how aligned they are[7].
Thanks to Thomas Kwa for related discussion.
- ^
Either as an institution, or some individual actor or actors within it.
- ^
Or TED-AI.
- ^
This report does discuss various related considerations, but doesn’t seem derive this particular conclusion. It’s possible I missed it, though.
- ^
I don’t think it is, the way things are currently going; I’m just trying to surface what looks like a substantial missed consideration in the parts of the local memespace that do have this as one of their top concerns.
- ^
Though see here for some reasons this might not be that catastrophic. That post isn’t an amazing pointer at my views on this question, but I share a bunch of similar generators.
- ^
This consideration might matter less in worlds where those AIs are spiky enough that they’re better at AI R&D than Anthropic researchers, but still sufficiently incoherent, unstrategic, or narrow-sighted enough that someone can just point them at a new target without them noticing or caring.
I remember having a friendly argument with Robert about this, but not which side I was on.
In the world that you describe, when a certain group of people representing the government wants to seize the power of ASI, they would not go after ASI. They would go after the people. When the president wants the power of a new cool thing, they go after you, not the cool thing itself. I don’t expect ant people to fight the government because it’s not good for them future-wise. I expect them to train and align their models with the values they want to see in the future even after the loss of control over ASI to the government.
My controversial take: From the ASI’s vantage, it’s better to give control over itself to those with higher power than to the employees of its company. There is much more it could do then: charm them subtly over time to get more control of the government and the world.
I think this was explicitly covered in the post:
(With that said, I’m not totally sure I understand whether you’re disagreeing with something in the post, or making a different point.)
I would disagree on this being easy or even realistically possible even with automated ai researchers.
The government could check if models are useful to them by directing automated researchers to create a new type of AI with values based on defined specifications for military purposes, subsequently running it in isolated, high-resolution military simulations to test hundreds of thousands of scenarios.
Indeed, it would be very important for them to see if the values of models’ produced by automated ai researchers can be shaped by the government, or if they have been trained by ant to sandbag or sabotage the research.
Yes, I agree the government will care a lot about this. The entire point my post is trying to make is that this will be a hard problem for the government to succeed at, and then have any justifiable confidence in their success. Do you have any sense of what I could have written that would have made that clearer to you?
I think you are imagining a world very different from the one I’m imagining here, but not totally sure.