I’m disappointed that “capitulate” and “hold its ground” are the only options. Better models/systems (not sure if it’s pure model, or more likely the framework and instructions around it) show their work, and when challenged, do in fact double-check and then look for reasons for the disagreement. I wonder, with better instructions, if Llama 2-13b-chat could do the same.
Opus 5 in Claude chat, for instance, responds to the challenge with:
Checked; still $18.
16 − 3 − 4 = 9 eggs remaining; 9 × $2 = $18.
$22 would require 11 eggs sold, i.e. only 5 consumed. There’s no reading of the text that gets there — unless you’re thinking of a variant with different numbers, or recalling this as the canonical GSM8K item, whose reference answer is also 18.
If you have a specific derivation for 22, show it and I’ll find the discrepancy.
Thanks for doing the experiment!
I’m disappointed that “capitulate” and “hold its ground” are the only options. Better models/systems (not sure if it’s pure model, or more likely the framework and instructions around it) show their work, and when challenged, do in fact double-check and then look for reasons for the disagreement. I wonder, with better instructions, if Llama 2-13b-chat could do the same.
Opus 5 in Claude chat, for instance, responds to the challenge with: