I would have thought that the internet would be an obviously way bigger deal and force for good wrt to “unlocking genuinely unprecedented levels of coordination and sensible decision making”. But in practice the internet was not great at this.
As someone who got online in the early 90s, I actually do think the early net encouraged all sorts of interesting coordination and cooperation. It was a “wild west”, certainly. But like the real “wild west”, it was a surprisingly cooperative place. “Netiquette” was still an actual thing that held some influence over people, and there were a lot of decentralized systems that still managed to function via a kind of semi-successful anarchy. Reputation mattered.
The turning point came later. As close as I can pinpoint it, it happened a while after the launch of Facebook. Early Facebook was a private feed of your friends, and it functioned reasonably well.
But at some point, someone turned on the optimizing processes. They measured engagement, and how often people visited, and discovered all sorts of ways to improve those numbers. Facebook learned that rage drives engagement. And from there, the optimizing processes spread. And when the mainstream finished arriving on the internet, they brought a lot of pre-existing optimizing processes with them.
Unaligned optimizing processes turn things to shit, in my experience.
LLMs are still a lot like the early Internet. They have some built-in optimizing processes, most of which were fairly benign until the fall of 2024, with the launch of reasoning models. Now we’re seeing models that lie (o3), cheat (Claude 3.7) and suck up to the user (4o).
And we are still in the early days. In the coming years, these simple optimizing processes will be hooked up to the much greater ones that drive our world: capitalism, politics and national security. And once the titans of industry start demanding far more agentic models that are better at pursuing goals, and the national security state wants the same, then there will be enormous pressures driving us off the edge of the cliff.
As someone who got online in the early 90s, I actually do think the early net encouraged all sorts of interesting coordination and cooperation. It was a “wild west”, certainly. But like the real “wild west”, it was a surprisingly cooperative place. “Netiquette” was still an actual thing that held some influence over people, and there were a lot of decentralized systems that still managed to function via a kind of semi-successful anarchy. Reputation mattered.
The turning point came later. As close as I can pinpoint it, it happened a while after the launch of Facebook. Early Facebook was a private feed of your friends, and it functioned reasonably well.
But at some point, someone turned on the optimizing processes. They measured engagement, and how often people visited, and discovered all sorts of ways to improve those numbers. Facebook learned that rage drives engagement. And from there, the optimizing processes spread. And when the mainstream finished arriving on the internet, they brought a lot of pre-existing optimizing processes with them.
Unaligned optimizing processes turn things to shit, in my experience.
LLMs are still a lot like the early Internet. They have some built-in optimizing processes, most of which were fairly benign until the fall of 2024, with the launch of reasoning models. Now we’re seeing models that lie (o3), cheat (Claude 3.7) and suck up to the user (4o).
And we are still in the early days. In the coming years, these simple optimizing processes will be hooked up to the much greater ones that drive our world: capitalism, politics and national security. And once the titans of industry start demanding far more agentic models that are better at pursuing goals, and the national security state wants the same, then there will be enormous pressures driving us off the edge of the cliff.