Definitely. But I currently suspect that for this approach:
We currently have a big overhang: we could be getting a lot even out of the models we already have
There’s some tipping point beyond which society is uplifted enough to correctly prioritise getting more uplifted
Getting to that tipping point wouldn’t require massively more advanced AI capabilities in a lot of the high-diffusion areas (i.e. Claude 4 might well be good enough for anything that requires literally everyone to have access to their own model)
The areas that might require more advanced capabilities require comparatively little diffusion (e.g. international coordination, lab oversight)
So definitely this fails if takeoff is really fast, but I think it might work given current takeoff trends if we were fast enough at everything else.
I think that if in 1980 you had described to me the internet and Claude-4-level LLMs, I would have thought that the internet would be an obviously way bigger deal and force for good wrt to “unlocking genuinely unprecedented levels of coordination and sensible decision making”. But in practice the internet was not great at this. I wonder if for some similar reasons that the internet made the situation both better and worse, Claude-4-level LLMs could make the situation both better and worse. I think you can try to shift the applications towards pro-epistemics/coordination ones, but I would guess you should expect an impact similar to the one internet activists had on the internet.
I am more optimistic about the positive impact (aligned) AIs could have for coordination once AIs dominate top human experts at negotiation, politics, etc. (though it’s not entirely clear, e.g. because it might be hard to create AIs that are legibly not trying to subtly help their developers).
I would have thought that the internet would be an obviously way bigger deal and force for good wrt to “unlocking genuinely unprecedented levels of coordination and sensible decision making”. But in practice the internet was not great at this.
As someone who got online in the early 90s, I actually do think the early net encouraged all sorts of interesting coordination and cooperation. It was a “wild west”, certainly. But like the real “wild west”, it was a surprisingly cooperative place. “Netiquette” was still an actual thing that held some influence over people, and there were a lot of decentralized systems that still managed to function via a kind of semi-successful anarchy. Reputation mattered.
The turning point came later. As close as I can pinpoint it, it happened a while after the launch of Facebook. Early Facebook was a private feed of your friends, and it functioned reasonably well.
But at some point, someone turned on the optimizing processes. They measured engagement, and how often people visited, and discovered all sorts of ways to improve those numbers. Facebook learned that rage drives engagement. And from there, the optimizing processes spread. And when the mainstream finished arriving on the internet, they brought a lot of pre-existing optimizing processes with them.
Unaligned optimizing processes turn things to shit, in my experience.
LLMs are still a lot like the early Internet. They have some built-in optimizing processes, most of which were fairly benign until the fall of 2024, with the launch of reasoning models. Now we’re seeing models that lie (o3), cheat (Claude 3.7) and suck up to the user (4o).
And we are still in the early days. In the coming years, these simple optimizing processes will be hooked up to the much greater ones that drive our world: capitalism, politics and national security. And once the titans of industry start demanding far more agentic models that are better at pursuing goals, and the national security state wants the same, then there will be enormous pressures driving us off the edge of the cliff.
Yeah, I fully expect that current level LMs will by default make the situation both better and worse. I also think that we’re still a very long way from fully utilising the things that the internet has unlocked.
My holistic take is that this approach would be very hard, but not obviously harder than aligning powerful AIs and likely complementary. I also think it’s likely we might need to do some of this ~societal uplift anyway so that we do a decent job if and when we do have transformative AI systems.
Some possible advantages over the internet case are:
People might be more motivated towards by the presence of very salient and pressing coordination problems
For example, I think the average head of a social media company is maybe fine with making something that’s overall bad for the world, but the average head of a frontier lab is somewhat worried about causing extinction
Currently the power over AI is really concentrated and therefore possibly easier to steer
A lot of what matters is specifically making powerful decision makers more informed and able to coordinate, which is slightly easier to get a handle on
As for the specific case of aligned super-coordinator AIs, I’m pretty into that, and I guess I have a hunch that there might be a bunch of available work to do in advance to lay the ground for that kind of application, like road-testing weaker versions to smooth the way for adoption and exploring form factors that get the most juice out of the things LMs are comparatively good at. I would guess that there are components of coordination where LMs are already superhuman, or could be with the right elicitation.
Definitely. But I currently suspect that for this approach:
We currently have a big overhang: we could be getting a lot even out of the models we already have
There’s some tipping point beyond which society is uplifted enough to correctly prioritise getting more uplifted
Getting to that tipping point wouldn’t require massively more advanced AI capabilities in a lot of the high-diffusion areas (i.e. Claude 4 might well be good enough for anything that requires literally everyone to have access to their own model)
The areas that might require more advanced capabilities require comparatively little diffusion (e.g. international coordination, lab oversight)
So definitely this fails if takeoff is really fast, but I think it might work given current takeoff trends if we were fast enough at everything else.
I think that if in 1980 you had described to me the internet and Claude-4-level LLMs, I would have thought that the internet would be an obviously way bigger deal and force for good wrt to “unlocking genuinely unprecedented levels of coordination and sensible decision making”. But in practice the internet was not great at this. I wonder if for some similar reasons that the internet made the situation both better and worse, Claude-4-level LLMs could make the situation both better and worse. I think you can try to shift the applications towards pro-epistemics/coordination ones, but I would guess you should expect an impact similar to the one internet activists had on the internet.
I am more optimistic about the positive impact (aligned) AIs could have for coordination once AIs dominate top human experts at negotiation, politics, etc. (though it’s not entirely clear, e.g. because it might be hard to create AIs that are legibly not trying to subtly help their developers).
As someone who got online in the early 90s, I actually do think the early net encouraged all sorts of interesting coordination and cooperation. It was a “wild west”, certainly. But like the real “wild west”, it was a surprisingly cooperative place. “Netiquette” was still an actual thing that held some influence over people, and there were a lot of decentralized systems that still managed to function via a kind of semi-successful anarchy. Reputation mattered.
The turning point came later. As close as I can pinpoint it, it happened a while after the launch of Facebook. Early Facebook was a private feed of your friends, and it functioned reasonably well.
But at some point, someone turned on the optimizing processes. They measured engagement, and how often people visited, and discovered all sorts of ways to improve those numbers. Facebook learned that rage drives engagement. And from there, the optimizing processes spread. And when the mainstream finished arriving on the internet, they brought a lot of pre-existing optimizing processes with them.
Unaligned optimizing processes turn things to shit, in my experience.
LLMs are still a lot like the early Internet. They have some built-in optimizing processes, most of which were fairly benign until the fall of 2024, with the launch of reasoning models. Now we’re seeing models that lie (o3), cheat (Claude 3.7) and suck up to the user (4o).
And we are still in the early days. In the coming years, these simple optimizing processes will be hooked up to the much greater ones that drive our world: capitalism, politics and national security. And once the titans of industry start demanding far more agentic models that are better at pursuing goals, and the national security state wants the same, then there will be enormous pressures driving us off the edge of the cliff.
Yeah, I fully expect that current level LMs will by default make the situation both better and worse. I also think that we’re still a very long way from fully utilising the things that the internet has unlocked.
My holistic take is that this approach would be very hard, but not obviously harder than aligning powerful AIs and likely complementary. I also think it’s likely we might need to do some of this ~societal uplift anyway so that we do a decent job if and when we do have transformative AI systems.
Some possible advantages over the internet case are:
People might be more motivated towards by the presence of very salient and pressing coordination problems
For example, I think the average head of a social media company is maybe fine with making something that’s overall bad for the world, but the average head of a frontier lab is somewhat worried about causing extinction
Currently the power over AI is really concentrated and therefore possibly easier to steer
A lot of what matters is specifically making powerful decision makers more informed and able to coordinate, which is slightly easier to get a handle on
As for the specific case of aligned super-coordinator AIs, I’m pretty into that, and I guess I have a hunch that there might be a bunch of available work to do in advance to lay the ground for that kind of application, like road-testing weaker versions to smooth the way for adoption and exploring form factors that get the most juice out of the things LMs are comparatively good at. I would guess that there are components of coordination where LMs are already superhuman, or could be with the right elicitation.