I agree that that these present competing explanations rather than any obvious failure of belief-state tracking.
To me, this suggests that benchmarks of this kind might need to not only evaluate the direct answer to a given question, but also the reasoning underpinning it. This might allow for easier detection of instances where a different, yet perhaps equally valid participant model, may have led to a different answer from the expected one.
What struck me while reading the failure cases is that many of them seem to assume there is a uniquely correct participant model available to the evaluator.
In several examples, the benchmark treats information access as the primary variable, while the model appears to be organizing the conversation around a different variable (identity continuity, conversational role, inferred access, etc.).
That doesn’t mean the model is correct, but it does suggest that some failures may reflect competition between participant models rather than straightforward failure to maintain one.
I wonder whether this becomes increasingly important as benchmarks move toward more realistic multi-agent environments. At some point, evaluating belief-state tracking may require evaluating which participant model the system adopted before evaluating whether it tracked that model consistently.
I agree that that these present competing explanations rather than any obvious failure of belief-state tracking.
To me, this suggests that benchmarks of this kind might need to not only evaluate the direct answer to a given question, but also the reasoning underpinning it. This might allow for easier detection of instances where a different, yet perhaps equally valid participant model, may have led to a different answer from the expected one.
That makes sense.
What struck me while reading the failure cases is that many of them seem to assume there is a uniquely correct participant model available to the evaluator.
In several examples, the benchmark treats information access as the primary variable, while the model appears to be organizing the conversation around a different variable (identity continuity, conversational role, inferred access, etc.).
That doesn’t mean the model is correct, but it does suggest that some failures may reflect competition between participant models rather than straightforward failure to maintain one.
I wonder whether this becomes increasingly important as benchmarks move toward more realistic multi-agent environments. At some point, evaluating belief-state tracking may require evaluating which participant model the system adopted before evaluating whether it tracked that model consistently.