Employee Incentives Make AGI Lab Pauses More Costly

Disclaimer: I know approximately nothing about internal governance at AGI labs. My current understanding is that the main AGI labs have basically no plans for how to navigate a conditional pause, and this post is predicated on this assumption.

TL;DR: If an AGI lab pauses, it’s essential that their employees don’t defect and advance capabilities in other ways, e.g. by breaking the pause commitment within the company, or by leaving the company and accelerating other companies’ capabilities. If entering a pause would make many employees angry, this sets up strong incentives for labs to never pause. To fix this, I recommend that companies take measures to keep their capabilities-focused employees happy during a conditional pause, such as giving them plenty of early warning, other projects to work on, or multiple years of paid time off.

It’s currently against employee incentives to enter a conditional pause

If a major AGI lab entered a voluntary conditional pause tomorrow, this would probably be a pretty stressful event for the employees that were working on training frontier models. It would probably also cause a significant shift in company priorities, such that many high-ranking people would have to be switched from their current roles into other roles. There would probably be people that are left without much to do at all, and potentially fired from the company. I expect that entering a conditional pause would cause a significant fraction of a company’s employees to have a stressful few weeks/​months.

If such a restructuring were to happen overnight, I wouldn’t be surprised if there were an employee backlash of similar magnitude to the one that happened when Sam Altman was fired. The employees of OpenAI, and partly by extension the employees of other AGI labs, know that they can fight for what they want and threaten to join competitor labs en masse if they disagree with leadership decisions.

AGI labs have strong incentives to avoid a conditional pause

AGI labs want to avoid their employees leaving. In the event of a conditional pause, the most affected employees would be the ones that work on their frontier model capabilities. There are a few reasons to avoid the loss of these employees in particular:

  • The employees that are pushing capabilities are often those with the highest status (compared to safety or product) and relatively long track records (as product employees are mostly recent hires). Through status and long-standing connections, they will have substantial leverage to mobilize the rest of the workforce to further their interests.

  • These people also have significant insider information and special skills that, if they moved to other AGI labs (which are not paused), would probably significantly speed up their AGI efforts and lead to their original lab being at a disadvantage.

If an AGI lab expected a large fraction of their top capabilities talent to leave their lab if they paused, this would:

  1. strongly incentivize AGI labs not to pause, and

  2. make AGI labs who pause systematically lose employees to labs that don’t.

There are significant negative externalities to capabilities-focused employees joining other labs:

  • We ideally want to limit the number of AGI labs that even enter the dangerous regions of AI development that trigger a conditional pause period, as there is a risk for any company to fail to trigger the conditional pause, and enter increasingly dangerous regions of AI development.

  • This means that, if an AGI lab enters a pause, their employees would ideally stay at their original lab, as going to other labs would increase their capabilities into more dangerous regions.

There are ways to improve employee incentives for a conditional pause

When an AGI lab starts to take drastic measures to reduce catastrophic risks, it’s important that the employees are willing to make the necessary sacrifices, and that the decision makers can appropriately raise morale and support when making drastic decisions

There are many ways to improve employee incentives:

  1. Give employees notice – possibly maintain internal prediction markets for the probability of a conditional pause in some time period, and put out an internal notice if a conditional pause is likely within the next year.

  2. Have a continuously updated restructuring plan such that fewer people are left without a role if a pause happens.

  3. Offer paid time off during the period of the pause for unhappy employees.

  4. Create a safety-focused cultue – emphasize how virtuous it is to make small personal sacrifices in case the company needs to do large restructuring.

  5. Bigger picture – establish pause commitments between companies to avoid races to the bottom.