Forgive me if you’ve answered this elsewhere. Out of curiosity, why not have them play existing games like Zork, and A Mind Forever Voyaging or if the reason is the existence of online answer keys, what about text games with no solutions like MUDs and MUSHs, (Gemstone, DragonRealms, Multi-User Middle Earth, etc)? In your first article I see you’re familiar with and enjoy the genre, but then you jump directly into creating your own game. I didn’t see the rationale for not using the ones that exist. I can think of my own reasons (answer keys etc) but am curious about your reasoning.
Actually, there are already benchmarks for some of these. There’s a Zork benchmark, for example: https://github.com/mnky9800n/zork-bench. The latest activity is around April of this year, as far as I can tell. And interestingly, the results on that benchmark were pretty bad up to that point.
As you suggest, I was worried about contamination. The Zork benchmark indicates that wasn’t that big a worry, because there are plenty of full walkthroughs on the internet and the agents still perform badly. A couple of hypotheses around this are that such data is sparse in training, or that converting third-person description into first-person agentic action is very difficult.
So I wanted a domain I was sure was not in the training data. And I wanted a small, constrained test that I had full control over. Plus, it was fun to design.
The small.and constrained I hadn’t considered. I was assuming that would be a hindrance, but given what you said about the results on Zork, it makes sense. Thanks again.
I have been tempted to try and get Claude to play a MUD, but don’t have the time to invest. There’s a ton of scripts people use to farm resources repetitively, but not for open-ended playing. Also, it has a chat function and pvp. I’m really curious how it would react to the other players and how they would react to it.
Yeah I would be super interested in experiments with MUDs and other open-ended domains. Evaluation is the hard part. Something like farming would be easy to measure. Determining player reaction would be highly subjective. There’s definitely a lot of interest now in harder-to-verify and open-ended tasks, so you should definitely find the time. ;)
Forgive me if you’ve answered this elsewhere. Out of curiosity, why not have them play existing games like Zork, and A Mind Forever Voyaging or if the reason is the existence of online answer keys, what about text games with no solutions like MUDs and MUSHs, (Gemstone, DragonRealms, Multi-User Middle Earth, etc)? In your first article I see you’re familiar with and enjoy the genre, but then you jump directly into creating your own game. I didn’t see the rationale for not using the ones that exist. I can think of my own reasons (answer keys etc) but am curious about your reasoning.
No problem.
Actually, there are already benchmarks for some of these. There’s a Zork benchmark, for example: https://github.com/mnky9800n/zork-bench. The latest activity is around April of this year, as far as I can tell. And interestingly, the results on that benchmark were pretty bad up to that point.
As you suggest, I was worried about contamination. The Zork benchmark indicates that wasn’t that big a worry, because there are plenty of full walkthroughs on the internet and the agents still perform badly. A couple of hypotheses around this are that such data is sparse in training, or that converting third-person description into first-person agentic action is very difficult.
So I wanted a domain I was sure was not in the training data. And I wanted a small, constrained test that I had full control over. Plus, it was fun to design.
I hadn’t seen the Zork benchmark, thank you!
The small.and constrained I hadn’t considered. I was assuming that would be a hindrance, but given what you said about the results on Zork, it makes sense. Thanks again.
I have been tempted to try and get Claude to play a MUD, but don’t have the time to invest. There’s a ton of scripts people use to farm resources repetitively, but not for open-ended playing. Also, it has a chat function and pvp. I’m really curious how it would react to the other players and how they would react to it.
No problem.
Yeah I would be super interested in experiments with MUDs and other open-ended domains. Evaluation is the hard part. Something like farming would be easy to measure. Determining player reaction would be highly subjective. There’s definitely a lot of interest now in harder-to-verify and open-ended tasks, so you should definitely find the time. ;)