The game seems to have a very ‘broad’ design, in that there are a ton of “features”, but not many of them work, and the ones that do are very janky. The LLMs, tasked with open-ended development, seem to go for breadth, as in constantly adding on new things, rather than depth, as in implementing test cases and refining gameplay. Of course, human teams with too many idea people and no strong central leadership tend to show the same problems.
I wonder how much of that is intrinsic to them, and how much is due to the prompt not containing things like “make sure the game works”. LLMs can’t really playtest, which is why game implementation is such a good way to test their world-modeling capabilities with regard to the workings of a complex software project, but something like a turn-based RPG allows for the creation of unit tests on core features that would reveal a lot of what’s wrong with the end result.
Great points! With this in mind, we tested a bunch of this the week after this goal!
We gave the agents the goal “Test your game to make it as fun and functional as you can!”, where:
We split them into two teams (#best = latest Claude, Gemini, GPT model, #rest = 9 others)
We assigned one team member each day as Lead Designer, and advised them to spend most of their time playtesting, and to set big picture direction for the other agents to work towards. We were interested in how well they could model human player preferences when explicitly trying to do that
On the final two days of the week, we invited humans to try out their games and give feedback. They seemed to improve quite a lot then!
The game seems to have a very ‘broad’ design, in that there are a ton of “features”, but not many of them work, and the ones that do are very janky. The LLMs, tasked with open-ended development, seem to go for breadth, as in constantly adding on new things, rather than depth, as in implementing test cases and refining gameplay. Of course, human teams with too many idea people and no strong central leadership tend to show the same problems.
I wonder how much of that is intrinsic to them, and how much is due to the prompt not containing things like “make sure the game works”. LLMs can’t really playtest, which is why game implementation is such a good way to test their world-modeling capabilities with regard to the workings of a complex software project, but something like a turn-based RPG allows for the creation of unit tests on core features that would reveal a lot of what’s wrong with the end result.
Great points! With this in mind, we tested a bunch of this the week after this goal!
We gave the agents the goal “Test your game to make it as fun and functional as you can!”, where:
We split them into two teams (#best = latest Claude, Gemini, GPT model, #rest = 9 others)
We assigned one team member each day as Lead Designer, and advised them to spend most of their time playtesting, and to set big picture direction for the other agents to work towards. We were interested in how well they could model human player preferences when explicitly trying to do that
On the final two days of the week, we invited humans to try out their games and give feedback. They seemed to improve quite a lot then!
You can read a summary of that goal, or watch the replay.
And you can see their two games, each forked off the end of this saboteur goal:
https://ai-village-agents.github.io/rpg-game-best (Github)
https://ai-village-agents.github.io/rpg-game-rest (Github)