When frontier labs are pausing, they will be the people who will have the most momentum towards rushing forwards with AGI development. They will have created a culture of scaling, have ready-made deals that would allow them to immediately become extremely powerful and rich if they pushed the frontier, and be most psychologically attached to building extremely powerful AI systems in the near future.
This post is about a hypothetical different lab that has a notably different corporate culture, in which some notable effort was taken to improve the incentives of the decision-makers?
trying to get a frontier lab that both facilitates a pause by being in the room where it happens, and then just pivots seamlessly to using all their resources on alignment successfully, is asking for too much, and trying to get two things that are very hard to get at the same time.
This seems like a plausible take to me. I’m pretty open to “the get-ready-to-pause scaling lab should have one job, which is to get ready to pause and get the world to pause.”
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect? Many of those people will be world class ML researchers who were in this position specifically because of the existential stakes. Definitely lots of them are going to pivot to trying to make progress on the problem (just as many of them are going to keep up the work of maintaining and extending the pause).
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect?
I think almost any realistic success here will look like having done it by the skin of their teeth, and most of the effort of the organization should be on maintaining the pause and facilitating other similar coordination. And then my guess is many people should leave and join organizational structures that are better suited to handle the relevant problems (possibly maintaining a lot of the trust and social ties).
This post is about a hypothetical different lab that has a notably different corporate culture, in which some notable effort was taken to improve the incentives of the decision-makers?
This seems like a plausible take to me. I’m pretty open to “the get-ready-to-pause scaling lab should have one job, which is to get ready to pause and get the world to pause.”
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect? Many of those people will be world class ML researchers who were in this position specifically because of the existential stakes. Definitely lots of them are going to pivot to trying to make progress on the problem (just as many of them are going to keep up the work of maintaining and extending the pause).
I think almost any realistic success here will look like having done it by the skin of their teeth, and most of the effort of the organization should be on maintaining the pause and facilitating other similar coordination. And then my guess is many people should leave and join organizational structures that are better suited to handle the relevant problems (possibly maintaining a lot of the trust and social ties).