Having GPT3/4 multiply numbers is a bit like eating soup with a fork. You can do it, and the larger you make the fork, the more soup you’ll get—but it’s not designed for it and it’s hugely impractical. GPT4 does not have an internal algorithm for multiplication because the training objective (text completion) does not incentivize developing that. No iteration of GPT (5, 6, 7) will ever be a 100% accurate calculator (unless they change the paradigm away from LLM+RLHF), it will just asymptotically approach 100%. Why don’t we just make a spoon?
Agreed. However, humans also don’t have an internal multiplication algorithm, but can nonetheless use a scratchpad to multiply accurately (in extreme circumstances :P). I’ve chosen multiplication as an example here because it’s maybe the “simplest” thing GPT-4 can’t consistently do.
What I’m finding interesting here is that GPT-4 knows how to break down multiplications (it can write perfect recursive code for multiplying large numbers). It also knows about chain of thought prompting. How close is it to being able to just… directly use the algorithms it knows? What’s the minimum amount of prodding needed? The following prompt is an upper bound on how hard it is to teach GPT-4 accurate multiplication:
To multiply large numbers accurately, it helps to break down the problem. We will demonstrate multiplying 191241 by 741122.
Set a = 19, b = 12, c = 41, d = 74, e = 11, and f = 22. We can then write
191241 * 741122 = (10^4 a + 10^2 b + c) * (10^4 d + 10^2 e + f).
Expanding this out, we obtain
10^8 ab + 10^6 (ae + bd) + 10^4 (af + be + cd) + 10^2 (bf + ce) + cf.
We can now calculate all these products:
ab = 19 * 74 = 1406
ae = 19 * 11 = 209
bd = 12 * 74 = 888
af = 19 * 22 = 418
be = 12 * 11 = 132
cd = 41 * 74 = 3034
bf = 12 * 22 = 264
ce = 41 * 11 = 451
cf = 41 * 22 = 902
Now we can calculate the sums inside parentheses:
ae + bd = 209 + 888
= 1097
af + be + cd = 418 + 132 + 3034
= 550 + 3034
= 3584
bf + ce = 264 + 451
= 715
We can now factor to avoid having so many zeroes:
10^8 ab + 10^6 (ae + bd) + 10^4 (af + be + cd) + 10^2 (bf + ce) + cf
10^2 (10^2 (10^2 (10^2 ab + (ae + bd)) + (af + be + cd)) + (bf + ce)) + cf
Now we substitute in:
10^2 (10^2 (10^2 (10^2 * 1406 + 1097) + 3584) + 715) + 902
= 10^2 (10^2 (10^2 (140600 + 1097) + 3584) + 715) + 902
= 10^2 (10^2 (10^2 (141697) + 3584) + 715) + 902
= 10^2 (10^2 (14169700 + 3584) + 715) + 902
= 10^2 (10^2 (14173284) + 715) + 902
= 10^2 (1417328400 + 715) + 902
= 10^2 (1417329115) + 902
= 141732911500 + 902
= 141732912402
The final answer is 141732912402.
Use the exact same method to multiply 183112 * 994531.
(It correctly follows the procedure to obtain 182110560472.) Is there a shorter prompt that accomplishes the same thing?
I couldn’t get yours to ever work with Bing Chat but I did eventually find something that did work for me (most of the time) and is about half the characters of yours without any real code golfing. My prompt was the following:
Having GPT3/4 multiply numbers is a bit like eating soup with a fork. You can do it, and the larger you make the fork, the more soup you’ll get—but it’s not designed for it and it’s hugely impractical. GPT4 does not have an internal algorithm for multiplication because the training objective (text completion) does not incentivize developing that. No iteration of GPT (5, 6, 7) will ever be a 100% accurate calculator (unless they change the paradigm away from LLM+RLHF), it will just asymptotically approach 100%. Why don’t we just make a spoon?
Agreed. However, humans also don’t have an internal multiplication algorithm, but can nonetheless use a scratchpad to multiply accurately (in extreme circumstances :P). I’ve chosen multiplication as an example here because it’s maybe the “simplest” thing GPT-4 can’t consistently do.
What I’m finding interesting here is that GPT-4 knows how to break down multiplications (it can write perfect recursive code for multiplying large numbers). It also knows about chain of thought prompting. How close is it to being able to just… directly use the algorithms it knows? What’s the minimum amount of prodding needed? The following prompt is an upper bound on how hard it is to teach GPT-4 accurate multiplication:
(It correctly follows the procedure to obtain
182110560472.) Is there a shorter prompt that accomplishes the same thing?I couldn’t get yours to ever work with Bing Chat but I did eventually find something that did work for me (most of the time) and is about half the characters of yours without any real code golfing. My prompt was the following: