After years of tinkering and incremental progress, AIs can now play Diplomacy as well as human experts.[6]
CICERO was a custom-trained diplomacy model that couldn’t win against human experts if they knew it was an AI. Now, in 2025, we have https://every.to/diplomacy which is just off-the-shelf LLM chatbots applied to Diplomacy. I’m curious to know how they would stack up against human experts who knew they were AIs. I expect they’d probably lose, but that if somehow they could do lots of RL on games against humans, they’d start winning, just as I originally forecast.
CICERO was a custom-trained diplomacy model that couldn’t win against human experts if they knew it was an AI. Now, in 2025, we have https://every.to/diplomacy which is just off-the-shelf LLM chatbots applied to Diplomacy. I’m curious to know how they would stack up against human experts who knew they were AIs. I expect they’d probably lose, but that if somehow they could do lots of RL on games against humans, they’d start winning, just as I originally forecast.
I really appreciate that you come back to update this.
I should do a blog post retrospective, now that it’s almost 2026, going over the whole post & comparing to reality!
…that’s a lot of work though, if anyone else wants to do it I’d be very grateful! As a bonus they might be less biased than me.
I’d be happy to link to it here in the OP.