Really cool stuff! Is this in a place where you can easily run it on new models as they get released? It’s hard to find benchmarks where the LLMs don’t saturate, and some form of “playing DF with a particular goal” seems like it’d be a good benchmark
Thanks! I haven’t tried this in a while, but it should work in any agent harness that supports either bash or MCP tools. Frontier models are so good at tmux I’m not sure if there’s much value in the MCP tools at this point.
Really cool stuff! Is this in a place where you can easily run it on new models as they get released? It’s hard to find benchmarks where the LLMs don’t saturate, and some form of “playing DF with a particular goal” seems like it’d be a good benchmark
Thanks! I haven’t tried this in a while, but it should work in any agent harness that supports either bash or MCP tools. Frontier models are so good at tmux I’m not sure if there’s much value in the MCP tools at this point.
If you goal is a reasoning benchmark and not a vision benchmark, you might want to check out Ryan Miller’s DFHack-based approach.