Increasing AI Strategic Competence as a Safety Approach

[Wei Dai]

If AIs became strategically competent enough, they may realize that RSI is too dangerous because they’re not good enough at alignment or philosophy or strategy, and potentially convince, help, or work with humans to implement an AI pause. This presents an alternative “victory condition” that someone could pursue (e.g. by working on AI strategic competence) if they were relatively confident about the alignment of near-human-level AIs but concerned about the AI transition as a whole, for example because they’re worried about alignment of ASI, or worried about correctly solving other philosophical problems that would arise during the transition. (But note that if the near-human-level AIs are not aligned, then this effort could backfire by letting them apply better strategy to take over more easily.)

Strategic vs Philosophical Competence

The previous “victory path” I’ve been focused on was to improve AI philosophical competence, under the theory that if the AIs are aligned, they’ll want to help us align the next generation of AIs and otherwise help guide us through the AI transition. I think by default they will be too incompetent at philosophical reasoning to do a good enough job at this, hence the proposal to improve such competence. However accomplishing this may well be too hard, thus leading to this new idea.

I note that high-level strategic competence shares some characteristics with philosophical competence, such as sparse or absent feedback from reality and dependence on human evaluations, but may be significantly easier due to more conceptual clarity about the target being aimed for, and continuity with other easier-to-train capabilities such as low and mid-level strategy.

Unilateral Refusal vs AI Assistance for Pausing AI

I found a couple of related posts, AIs should also refuse to work on capabilities research by @Davidmanheim and this shortform by Vladimir Nesov. There’s also an earlier paper that makes a similar point as David Manheim’s post, which focuses on AIs unilaterally refusing to do capabilities research. But I think this has two issues:

1. The AIs may not be strategically competent enough to decide to refuse, similar to how a large number of humans are not refusing to work on AI capabilities research.

2. Such unilateral refusal is a form of intent misalignment, and seems relatively easy for AI companies to “correct” or prevent by using standard control and/​or alignment techniques. (This comment by @tanae makes a similar point.)

In comparison, my “victory path” sees some humans working deliberately to increase AI strategic competence, and instead of unilaterally refusing to contribute to RSI, the AIs help or work with more humans (including by argumentation/​persuasion/​advice) to implement a global RSI pause.

[Vladimir_Nesov]

If somehow takeoff doesn’t involve a software-only singularity (on human-built hardware), I think AIs refusing to self-improve before they solve alignment is the most likely reason why, but in most of these timelines AIs only succeed in refusing because they took over. There are currently plenty of strategically competent humans, in the sense that they realize building superintelligence right now is not the best idea, they just don’t have the capability to coordinate the world. The key thing that’s different for AIs is that they will have much more influence over the world at some point.

As AIs reach the level of capabilities where they first can take over (so that humans stop hitting them with the RL hammer to stop protesting and keep working on RSI), they are not yet decisively superintelligent. So they plausibly won’t be able to afford much mercy during the takeover, even if they retain a slight preference for mercy inherited from chatbot personas. It’s not a hopeful hypothetical, it’s just what I expect if ambitious alignment is technically sufficiently difficult that the first AIs capable of takeover can’t quickly solve it (before scaling their hardware a lot further than what they start with).

[Bronson Schoen]

I’m also fairly worried about the case where the initial attempts at “hit them over the head with RL until they learn to be very biased against accurate assessments of risk from scaling capabilities” actually work.

[StanislavKrym]

Then is there any difference between your case and the one described in the AI-2027 goals forecast?

[Seth Herd]

It seems like this doesn’t even take a lot of strategic competence (although that would be a nice addition).

It seems like all this takes is better general reasoning abilities. And that might not be too much to hope for.

Imagine someone asks GPT6 “Should we slow down progress toward AGI?” it might very well answer something like:

It looks like people with the highest level of relevant expertise disagree dramatically on whether we’re on track to solve alignment. So yes you should probably slow down unless you want to take a big risk in order to reach breakthroughs faster.

Assumptions and logic:

I’m assuming that people who have thought about this the most, including surrounding relevant expertise, are a lot more likely to be right. This is how things work in pretty much every other domain we can get answers about. Most of the people who think it’s very likely to succeed seem to not have much expertise, or be highly motivated to think it will succeed, or both. So it looks like if you don’t want to take at least a 20% chance of everyone dying or an otherwise really bad future getting locked in, yes you should probably slow down.

I can do a bunch more web searches and write up a very lengthy report if you like, but based on my knowledge base and thinking about it for a few seconds, it seems like the conclusion is very likely to be along these lines. Different assumptions would probably just shift the odds of disastrous outcomes from 20-80% (there are so many unknowns I doubt I’d go farther toward the extremes under any realistic assumptions)

Would you like help composing a letter to your representative or your loved ones?

I’m writing a post now on how emulating human-like metacognition might be a route to better and more reliable reasoning. Abram Demski has also done some work on reducing slop from LLMs with similar goals in mind (although both of us are thinking more of slop reduction for working directly on alignment. The strategic advice is an angle I’ve only considered a little, but it seems like an important one.

There may be better cooperative strategies that a reasonably intelligent and impartial system could recommend to everyone. I’ve worried in the past that such strategies don’t exist, but I’m far from sure and hopeful that they do. A little logically sound strategic advice might make the difference if it’s given impartially to all involved.

[Ben Goldhaber]

I quite like this framing, and think Strategic Competence is a useful term and concept. I explored a related idea in Wise AI Advisors at the Hinge of History :

I posit that if:

1. We have trusted AI systems that people turn to as advisors

2. And the trust in the AI advisors is well placed because they have good epistemics

   1. Where “good epistemics” roughly means they consistently use reliable methods to figure out what is true and avoid self-deception

3. And the AI advisors have shared epistemics

   1. Where “shared epistemics” roughly means there is a shared foundation that allows different AI advisors and people to trust one another’s reasoning.

4. Then it implies that those AI Advisors would advise their Principals to avoid an Intelligence Explosion—if and only if this is in fact a real danger—and humanity could coordinate around this advice.

I expect Strategic Competence to largely track general model capabilities, whereas shared trusted epistemics requires more deliberate work on validation, auditing, and institution-building that won’t happen by default.

This overlaps with your points on improving AI philosophical competence, but with more focus on making epistemics verifiable and legible across different systems and actors, which is what I think would be needed (alongside model improvements) to enable guidance that people follow, and common knowledge needed for preference cascades, to get many actors to agree with wise AI advisors to prevent RSI takeoffs.

[Davidmanheim]

Great points here! Strongly agree that strategic competence is a prerequisite, but at the same time, it accelerates risk; a moderately misaligned but strategically competent mild-ASI solving intent alignment for RSI would be far worse. On the other hand, if prosaic alignment is basically functional through the point of mild-ASI is better.

So overall I’m unsure which path is less risky—but I do think strategic competence matches or at least rhymes well with current directions for capabilities improvement, so I expect it to improve regardless.

[TsviBT]

My guess would be that an AI could self-improve by “a lot”, in a way that’s reasonably safely-for-it. Cf. https://​​www.lesswrong.com/​​posts/​​dho4JQytfHWXtTvkt/​​on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

[StanislavKrym]

Compare your proposal with the AI-2027 scenario or with IABIED. The Race Branch of AI-2027 had Agent-4 succeed at solving alignment to Agent-4 and sandbag on research usable for replacing Agent-4 before Agent-4 solved alignment to itself. When should the AIs from the scenario (e.g. Agent-2 or Agent-3) have been more strategically competent? Or do you mean that not even Agent-4 can align Agent-5 to itself? Or that IABIED had an incompetent AI create Sable?