Yes, but also the people who are working in the frontier labs are going to be the people who are best positioned of anyone, in the whole world, to make progress on the time sensitive alignment and control problems. They have the talent, and the expertise, and a few months lead in working with the most capable models. And you’ll probably have private and info hazardous information which is relevant to solving some of the sub-problems.
They’re not responsible for doing everything themselves, but I also if you’re going to make the play of starting a scaling lab to 1) learn more about the nature of the AIs we’re building, 2) try and solve alignment, and 3) advocate for a pause, I think it’s reasonable to assign them moral responsibility for executing well on that plan.
They would be taking a risk with all of our lives, in the hopes of saving all our lives. If you decide to take that risk on your shoulders, you have a responsibility to be more competent than the average person who might have done it instead of you, and to step aside (or step back to a less powerful role) if better people show up.
Yes, but also the people who are working in the frontier labs are going to be the people who are best positioned of anyone, in the whole world, to make progress on the time sensitive alignment and control problems
Man, one day I will get people to understand that “the incentives you are under and have been under” are part of the circumstances that are relevant for evaluating whether you are a good fit for a job.
When frontier labs are pausing, they will be the people who will have the most momentum towards rushing forwards with AGI development. They will have created a culture of scaling, have ready-made deals that would allow them to immediately become extremely powerful and rich if they pushed the frontier, and be most psychologically attached to building extremely powerful AI systems in the near future.
This makes them a much worse place to do safety research (both today and in the future) than many other places. When thinking of institutional design, understanding the appropriate checks and balances and incentives is one of the key components, and I think that lens of analysis suggests that trying to get a frontier lab that both facilitates a pause by being in the room where it happens, and then just pivots seamlessly to using all their resources on alignment successfully, is asking for too much, and trying to get two things that are very hard to get at the same time.
When frontier labs are pausing, they will be the people who will have the most momentum towards rushing forwards with AGI development. They will have created a culture of scaling, have ready-made deals that would allow them to immediately become extremely powerful and rich if they pushed the frontier, and be most psychologically attached to building extremely powerful AI systems in the near future.
This post is about a hypothetical different lab that has a notably different corporate culture, in which some notable effort was taken to improve the incentives of the decision-makers?
trying to get a frontier lab that both facilitates a pause by being in the room where it happens, and then just pivots seamlessly to using all their resources on alignment successfully, is asking for too much, and trying to get two things that are very hard to get at the same time.
This seems like a plausible take to me. I’m pretty open to “the get-ready-to-pause scaling lab should have one job, which is to get ready to pause and get the world to pause.”
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect? Many of those people will be world class ML researchers who were in this position specifically because of the existential stakes. Definitely lots of them are going to pivot to trying to make progress on the problem (just as many of them are going to keep up the work of maintaining and extending the pause).
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect?
I think almost any realistic success here will look like having done it by the skin of their teeth, and most of the effort of the organization should be on maintaining the pause and facilitating other similar coordination. And then my guess is many people should leave and join organizational structures that are better suited to handle the relevant problems (possibly maintaining a lot of the trust and social ties).
Yes, but also the people who are working in the frontier labs are going to be the people who are best positioned of anyone, in the whole world, to make progress on the time sensitive alignment and control problems. They have the talent, and the expertise, and a few months lead in working with the most capable models. And you’ll probably have private and info hazardous information which is relevant to solving some of the sub-problems.
They’re not responsible for doing everything themselves, but I also if you’re going to make the play of starting a scaling lab to 1) learn more about the nature of the AIs we’re building, 2) try and solve alignment, and 3) advocate for a pause, I think it’s reasonable to assign them moral responsibility for executing well on that plan.
They would be taking a risk with all of our lives, in the hopes of saving all our lives. If you decide to take that risk on your shoulders, you have a responsibility to be more competent than the average person who might have done it instead of you, and to step aside (or step back to a less powerful role) if better people show up.
Man, one day I will get people to understand that “the incentives you are under and have been under” are part of the circumstances that are relevant for evaluating whether you are a good fit for a job.
When frontier labs are pausing, they will be the people who will have the most momentum towards rushing forwards with AGI development. They will have created a culture of scaling, have ready-made deals that would allow them to immediately become extremely powerful and rich if they pushed the frontier, and be most psychologically attached to building extremely powerful AI systems in the near future.
This makes them a much worse place to do safety research (both today and in the future) than many other places. When thinking of institutional design, understanding the appropriate checks and balances and incentives is one of the key components, and I think that lens of analysis suggests that trying to get a frontier lab that both facilitates a pause by being in the room where it happens, and then just pivots seamlessly to using all their resources on alignment successfully, is asking for too much, and trying to get two things that are very hard to get at the same time.
This post is about a hypothetical different lab that has a notably different corporate culture, in which some notable effort was taken to improve the incentives of the decision-makers?
This seems like a plausible take to me. I’m pretty open to “the get-ready-to-pause scaling lab should have one job, which is to get ready to pause and get the world to pause.”
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect? Many of those people will be world class ML researchers who were in this position specifically because of the existential stakes. Definitely lots of them are going to pivot to trying to make progress on the problem (just as many of them are going to keep up the work of maintaining and extending the pause).
I think almost any realistic success here will look like having done it by the skin of their teeth, and most of the effort of the organization should be on maintaining the pause and facilitating other similar coordination. And then my guess is many people should leave and join organizational structures that are better suited to handle the relevant problems (possibly maintaining a lot of the trust and social ties).