TL;DR: It behaves amazingly like a human would!
It understands the word problem, but struggles to put it in equations, takes multiple tries, screws up again, eventually muddles through to get the right answer. Given a similar problem it does slightly better, but still messes up. On the third problem it messes up again, then realizes that something went wrong… and promptly blames the problem setup, unable to question its own reasoning.
Got it in one thinking step by step.
yeah, looks like “steb by step” is a magic incantation:
https://twitter.com/ESYudkowsky/status/1602177149401989120
… which makes it even more human. I used to almost yell at the students I used to tutor: DON’T SKIP STEPS!
Incantation rankings, for GPT-3 on math word problems
Didn’t seem to work for me. It still seems to get confused trying to match similar words together even when they shouldn’t be. Again quite, dumb/young human.
It tries to continue its dialogue in ways that fit with its previous replies, so prompts like “think step by step” might fail to get it back on track mid-conversation even when they would’ve worked in the initial prompt.
Also, there is some randomness.
[humor] teenagers, amiright?