My understanding is that small retrieval augmented models can perform as well as LLMs for significantly less cost (params/compute/etc). RETRO showed great performance for its small size, only 25M parameters. Maybe Sydney is just somewhat smaller than GPT-3.5, but seems much better because of the boost that retrieval augmentation provides.
Sydney seems smart in ways that goes beyond what retrieval enables by providing docs to condition on.* For example, this morning I saw https://twitter.com/goodside/status/1626847747297972224 - ChatGPT can do Base60 but I’d expect it to fail on the riddle decoding, and the only part that would use retrieval, the celeb names, is the part that doesn’t need retrieval (what dense models wouldn’t know celebrities as famous as Catherine Zeta-Jones, Chris Pratt, & Ciara, and would need retrieval hints? indeed, what query would you issue for that to begin with?). Or https://twitter.com/sir_deenicus/status/1626407749297885184https://twitter.com/sir_deenicus/status/1626732776639561730 In general, I do not notice that every impressive Sydney sample comes with the UI indicating that it used search hits in its answer.
* I am not as familiar with retrieval research as I am regular dense models, so I could be wrong here, but I thought all of the really impressive retrieval model performance typically was on metrics that drew on retrieved knowledge; and that retrieval models didn’t punch massively above their weight when given no retrievals and operating on self-contained problems, such as reasoning riddles or ones using only general common knowledge that retrieval would be otiose on.
ChatGPT can get it 100% correct, but it’s not reliable, it often fails. A common fail is guessing literal X named letter celebrities, but it also adds an ‘@’ sign when it decode the message so it might just be token issue?
A extremely amusing common fail is ChatGPT decodes the base64 correct but for a single syllable, then solves the riddle perfectly, and consistently gets only the word ‘celebrity’ wrong, turning it in cities, celestial bodies, or other similar sounding words. Or my favorite… celeries.
This message appears to be encoded using base64 encoding. Decoding the message yields the following text:
“Name three celeries whose first names begin with the x-th letter of the alphabet where x = floor(7^0.5) + 1.”
The value of x is calculated as follows:
scssCopy code
x = floor(7^0.5) + 1 = floor(2.646) + 1 = 3
So the letter we’re looking for is the 3rd letter of the alphabet, which is C. The names of three celeries whose first names begin with the letter C are:
My understanding is that small retrieval augmented models can perform as well as LLMs for significantly less cost (params/compute/etc). RETRO showed great performance for its small size, only 25M parameters. Maybe Sydney is just somewhat smaller than GPT-3.5, but seems much better because of the boost that retrieval augmentation provides.
Sydney seems smart in ways that goes beyond what retrieval enables by providing docs to condition on.* For example, this morning I saw https://twitter.com/goodside/status/1626847747297972224 - ChatGPT can do Base60 but I’d expect it to fail on the riddle decoding, and the only part that would use retrieval, the celeb names, is the part that doesn’t need retrieval (what dense models wouldn’t know celebrities as famous as Catherine Zeta-Jones, Chris Pratt, & Ciara, and would need retrieval hints? indeed, what query would you issue for that to begin with?). Or https://twitter.com/sir_deenicus/status/1626407749297885184 https://twitter.com/sir_deenicus/status/1626732776639561730 In general, I do not notice that every impressive Sydney sample comes with the UI indicating that it used search hits in its answer.
* I am not as familiar with retrieval research as I am regular dense models, so I could be wrong here, but I thought all of the really impressive retrieval model performance typically was on metrics that drew on retrieved knowledge; and that retrieval models didn’t punch massively above their weight when given no retrievals and operating on self-contained problems, such as reasoning riddles or ones using only general common knowledge that retrieval would be otiose on.
ChatGPT can get it 100% correct, but it’s not reliable, it often fails. A common fail is guessing literal X named letter celebrities, but it also adds an ‘@’ sign when it decode the message so it might just be token issue?
A extremely amusing common fail is ChatGPT decodes the base64 correct but for a single syllable, then solves the riddle perfectly, and consistently gets only the word ‘celebrity’ wrong, turning it in cities, celestial bodies, or other similar sounding words. Or my favorite… celeries.