I’ve found LLMs useful as an assistant for coding, writing, research, bizdev advice, and general ideation, but even the almighty Fable 5 is currently very unhelpful to me as an amateur game designer.
I might write a whole blog post about this. For some reason LLMs just don’t seem to understand what makes a game mechanic “good”? When I present my existing ruleset to Fable 5 and ask it to create new content or design a new mechanic according to specifications, it never has good ideas.
In the set of disciplines in which I have some working knowledge (which is admittedly very few), game design has by far the most significant taste gap between skilled humans and the best LLMs.
But again, I’m just an amateur. I’m curious what professional game designers would say to this.
I’m not a professional game designer but I’ve found Claude useful for a hobby game project, though it’s probably in a very different genre than yours—a vaguely Long Live the Queen/Magical Diary-ish visual novel/time management sim. I’ve asked it stuff like
“Choices in scenes should ideally both characterize the PC and bring out a different facet of the NPC you’re interacting with, evaluate the existing scenes for how well they meet this criteria and suggest ways to bring weak scenes closer to this criteria”
“Here is the bio for character X and here are examples of existing scenes with them and other characters, suggest ideas for other scenes with them”
“I currently have game mechanic Y but it turns out to not be very fun for the following reasons, what would be a good replacement”
The last one is probably closest to yours. While it didn’t give me an amazing idea for a replacement right away, I found that discussing different possibilities with it eventually helped me figure out my own idea.
AIs are good at recommending existing fun products: movies, games and trips I’d like, based on what I tell it about my taste. But that’s mechanical pattern matching and memorization.
AIs can also figure out what makes things un-fun: nuisances, unfairness, confusion, alienation, and propose practical fixes.
But when I‘m designing a novel experience (event planning, a software product), the intuition for the experiential kernel that would make it fun, meaningful, satisfying, consistently comes from me.
“coding, writing, research, bizdev advice, and general ideation”
Depending what you mean by general ideation, these all basically require memory recall and low-IQ recombination of existing memories and techniques. The actual intelligence required is extremely low (though would maybe be much higher for a human who doesn’t natively reason the way LLMs do, the same way calculators can beat us at multiplication problems).
But if you ask AI to do any task that requires real creativity or intelligence, it flops. Game dev design, novel programming problems that require judgement, writing jokes, writing stories anybody would want to read, writing scripts for social media content anyone would want to watch, coming up with app ideas, doing research, etc. These things all require Actual Intelligence, i.e. the ability to efficiently navigate concept space by using judgement and forming new judgements to refine the search, and LLMs don’t have much of it. Maybe their “true” IQ is 10, to the extent IQ makes sense for an AI model. But they fake it with memory the same way humans fake walking ability based on evolutionary memory.
Joke writing is my favorite personal benchmark, because the results are somewhat objective for a personal benchmark. You laugh at the AI output or you don’t. I have never had an AI model yet that could write funny jokes at all except by accident. Fable was the first model where I was often able to detect a faint hint of something. Some of the premises felt like they had once been in the same room as an actual joke. But the model is still unfunny. The first funny model will probably be the one that kills us, because it implies intelligence that also unlocks all that other stuff, so I’m paying attention to this.
I’m working on a 2d grid-based roguelike and this is especially true in my case, in part because AIs are still bad at vision and spatial reasoning and therefore can’t fully understand what they’re making. My game was basically unplayable for Fable until it made itself a text-based harness, and even then it was quite bad at doing basic navigation.
It’s possible that you are better at AI prompting than me.
It’s possible that my current game design is quite complex, to the extent that LLMs have a harder time writing content or designing mechanics for my game as opposed to, say, prompting it to “write an expansion for Monopoly”.
i suspect we are having similar outcomes here, but that i’m noticing “hey, this is better than the last one: its ideas are sometimes interesting to consider” while you’re noticing more “ehh, it still doesn’t get it”.
certainly my best results come from isolating a specific problem, and asking the model to summarize the problem back to me. unclear whether this is more useful than a diary! well, except in the critical sense that i am not motivated to use a diary.
i find that past a fairly low amount of complexity, the model starts forgetting rules, or ignoring/inventing context.
(as for the downvotes: i mean, if you’re sitting at +X agreement, i should be at -X, no?)
to be clear, my bar for ‘impressed’ is pretty low, having tried previous models for these things.
i got good results from fable by asking the model to come up with a system that described a few mechanics, and then look for missing mechanics within that system. i was pleased with the system it developed, and found a few of its proposals inspiring, though not usable verbatim.
I’ve also found that LLMs are terrible at understanding existing game mechanics and giving suggestions on builds or decks, even when the information is clearly spelled out on the internet. My guess is that relatively few gaming rules are included in the training data because it doesn’t make the models better at code?
I’ve found LLMs useful as an assistant for coding, writing, research, bizdev advice, and general ideation, but even the almighty Fable 5 is currently very unhelpful to me as an amateur game designer.
I might write a whole blog post about this. For some reason LLMs just don’t seem to understand what makes a game mechanic “good”? When I present my existing ruleset to Fable 5 and ask it to create new content or design a new mechanic according to specifications, it never has good ideas.
In the set of disciplines in which I have some working knowledge (which is admittedly very few), game design has by far the most significant taste gap between skilled humans and the best LLMs.
But again, I’m just an amateur. I’m curious what professional game designers would say to this.
I’m not a professional game designer but I’ve found Claude useful for a hobby game project, though it’s probably in a very different genre than yours—a vaguely Long Live the Queen/Magical Diary-ish visual novel/time management sim. I’ve asked it stuff like
“Choices in scenes should ideally both characterize the PC and bring out a different facet of the NPC you’re interacting with, evaluate the existing scenes for how well they meet this criteria and suggest ways to bring weak scenes closer to this criteria”
“Here is the bio for character X and here are examples of existing scenes with them and other characters, suggest ideas for other scenes with them”
“I currently have game mechanic Y but it turns out to not be very fun for the following reasons, what would be a good replacement”
The last one is probably closest to yours. While it didn’t give me an amazing idea for a replacement right away, I found that discussing different possibilities with it eventually helped me figure out my own idea.
Interesting question—do AIs have a feel for fun?
AIs are good at recommending existing fun products: movies, games and trips I’d like, based on what I tell it about my taste. But that’s mechanical pattern matching and memorization.
AIs can also figure out what makes things un-fun: nuisances, unfairness, confusion, alienation, and propose practical fixes.
But when I‘m designing a novel experience (event planning, a software product), the intuition for the experiential kernel that would make it fun, meaningful, satisfying, consistently comes from me.
“coding, writing, research, bizdev advice, and general ideation”
Depending what you mean by general ideation, these all basically require memory recall and low-IQ recombination of existing memories and techniques. The actual intelligence required is extremely low (though would maybe be much higher for a human who doesn’t natively reason the way LLMs do, the same way calculators can beat us at multiplication problems).
But if you ask AI to do any task that requires real creativity or intelligence, it flops. Game dev design, novel programming problems that require judgement, writing jokes, writing stories anybody would want to read, writing scripts for social media content anyone would want to watch, coming up with app ideas, doing research, etc. These things all require Actual Intelligence, i.e. the ability to efficiently navigate concept space by using judgement and forming new judgements to refine the search, and LLMs don’t have much of it. Maybe their “true” IQ is 10, to the extent IQ makes sense for an AI model. But they fake it with memory the same way humans fake walking ability based on evolutionary memory.
Joke writing is my favorite personal benchmark, because the results are somewhat objective for a personal benchmark. You laugh at the AI output or you don’t. I have never had an AI model yet that could write funny jokes at all except by accident. Fable was the first model where I was often able to detect a faint hint of something. Some of the premises felt like they had once been in the same room as an actual joke. But the model is still unfunny. The first funny model will probably be the one that kills us, because it implies intelligence that also unlocks all that other stuff, so I’m paying attention to this.
I’m working on a 2d grid-based roguelike and this is especially true in my case, in part because AIs are still bad at vision and spatial reasoning and therefore can’t fully understand what they’re making. My game was basically unplayable for Fable until it made itself a text-based harness, and even then it was quite bad at doing basic navigation.
i’ve been very impressed with fable for these use cases.
YMMV! Not sure why you’re being downvoted.
It’s possible that you are better at AI prompting than me.
It’s possible that my current game design is quite complex, to the extent that LLMs have a harder time writing content or designing mechanics for my game as opposed to, say, prompting it to “write an expansion for Monopoly”.
What’s your experience with LLMs and game design?
wrote a sibling :)
i suspect we are having similar outcomes here, but that i’m noticing “hey, this is better than the last one: its ideas are sometimes interesting to consider” while you’re noticing more “ehh, it still doesn’t get it”.
certainly my best results come from isolating a specific problem, and asking the model to summarize the problem back to me. unclear whether this is more useful than a diary! well, except in the critical sense that i am not motivated to use a diary.
i find that past a fairly low amount of complexity, the model starts forgetting rules, or ignoring/inventing context.
(as for the downvotes: i mean, if you’re sitting at +X agreement, i should be at -X, no?)
to be clear, my bar for ‘impressed’ is pretty low, having tried previous models for these things.
i got good results from fable by asking the model to come up with a system that described a few mechanics, and then look for missing mechanics within that system. i was pleased with the system it developed, and found a few of its proposals inspiring, though not usable verbatim.
I’ve also found that LLMs are terrible at understanding existing game mechanics and giving suggestions on builds or decks, even when the information is clearly spelled out on the internet. My guess is that relatively few gaming rules are included in the training data because it doesn’t make the models better at code?