Agreed. However, humans also don’t have an internal multiplication algorithm, but can nonetheless use a scratchpad to multiply accurately (in extreme circumstances :P). I’ve chosen multiplication as an example here because it’s maybe the “simplest” thing GPT-4 can’t consistently do.
What I’m finding interesting here is that GPT-4 knows how to break down multiplications (it can write perfect recursive code for multiplying large numbers). It also knows about chain of thought prompting. How close is it to being able to just… directly use the algorithms it knows? What’s the minimum amount of prodding needed? The following prompt is an upper bound on how hard it is to teach GPT-4 accurate multiplication:
To multiply large numbers accurately, it helps to break down the problem. We will demonstrate multiplying 191241 by 741122.
Set a = 19, b = 12, c = 41, d = 74, e = 11, and f = 22. We can then write
191241 * 741122 = (10^4 a + 10^2 b + c) * (10^4 d + 10^2 e + f).
Expanding this out, we obtain
10^8 ab + 10^6 (ae + bd) + 10^4 (af + be + cd) + 10^2 (bf + ce) + cf.
We can now calculate all these products:
ab = 19 * 74 = 1406
ae = 19 * 11 = 209
bd = 12 * 74 = 888
af = 19 * 22 = 418
be = 12 * 11 = 132
cd = 41 * 74 = 3034
bf = 12 * 22 = 264
ce = 41 * 11 = 451
cf = 41 * 22 = 902
Now we can calculate the sums inside parentheses:
ae + bd = 209 + 888
= 1097
af + be + cd = 418 + 132 + 3034
= 550 + 3034
= 3584
bf + ce = 264 + 451
= 715
We can now factor to avoid having so many zeroes:
10^8 ab + 10^6 (ae + bd) + 10^4 (af + be + cd) + 10^2 (bf + ce) + cf
10^2 (10^2 (10^2 (10^2 ab + (ae + bd)) + (af + be + cd)) + (bf + ce)) + cf
Now we substitute in:
10^2 (10^2 (10^2 (10^2 * 1406 + 1097) + 3584) + 715) + 902
= 10^2 (10^2 (10^2 (140600 + 1097) + 3584) + 715) + 902
= 10^2 (10^2 (10^2 (141697) + 3584) + 715) + 902
= 10^2 (10^2 (14169700 + 3584) + 715) + 902
= 10^2 (10^2 (14173284) + 715) + 902
= 10^2 (1417328400 + 715) + 902
= 10^2 (1417329115) + 902
= 141732911500 + 902
= 141732912402
The final answer is 141732912402.
Use the exact same method to multiply 183112 * 994531.
(It correctly follows the procedure to obtain 182110560472.) Is there a shorter prompt that accomplishes the same thing?
Agreed. However, humans also don’t have an internal multiplication algorithm, but can nonetheless use a scratchpad to multiply accurately (in extreme circumstances :P). I’ve chosen multiplication as an example here because it’s maybe the “simplest” thing GPT-4 can’t consistently do.
What I’m finding interesting here is that GPT-4 knows how to break down multiplications (it can write perfect recursive code for multiplying large numbers). It also knows about chain of thought prompting. How close is it to being able to just… directly use the algorithms it knows? What’s the minimum amount of prodding needed? The following prompt is an upper bound on how hard it is to teach GPT-4 accurate multiplication:
(It correctly follows the procedure to obtain
182110560472.) Is there a shorter prompt that accomplishes the same thing?