I actually would bet money that a Sonnet-level model, given previous turn inputs, outputs, and tool calls (but NOT previous turn CoT), and the CoT of this turn could reconstruct
the game state prior to this turn
each of the proposed lines in the CoT, with visualizations of what Mythos was thinking about
This seems like the sort of thing where objective scoring is possible, too. The literal simplest thing of “the monitor must predict the next tool call based on prior turns + this turn CoT” wouldn’t work but I expect there’s something adjacent to that in eval space which would.
Yeah, I really like that idea.
I actually would bet money that a Sonnet-level model, given previous turn inputs, outputs, and tool calls (but NOT previous turn CoT), and the CoT of this turn could reconstruct
the game state prior to this turn
each of the proposed lines in the CoT, with visualizations of what Mythos was thinking about
This seems like the sort of thing where objective scoring is possible, too. The literal simplest thing of “the monitor must predict the next tool call based on prior turns + this turn CoT” wouldn’t work but I expect there’s something adjacent to that in eval space which would.