What specific rhyme-related tasks are you saying ChatGPT can’t do? I tried it on some unusual words and it got a bunch of things right, made a few weird mistakes, but didn’t give me the impression that it was totally unable to rhyme unusual words.
No, you’re doing it wrong, as I already explained. You’re letting GPT fall back onto its policy by choosing any response. You need to force it out of its comfort zone—force it off-policy, off the safe conservative path. Ask it to explain a pun it did not write, or answer questions like whether a pair of words that you picked rhyme. Write pairs of new words that have never been seen before, etc. The task of ‘come up with a memorized rhyme for reasonably common words’ does not disprove extensive memorization or show that it has failed to understand the underlying phonetics.
What specific rhyme-related tasks are you saying ChatGPT can’t do? I tried it on some unusual words and it got a bunch of things right, made a few weird mistakes, but didn’t give me the impression that it was totally unable to rhyme unusual words.
No, you’re doing it wrong, as I already explained. You’re letting GPT fall back onto its policy by choosing any response. You need to force it out of its comfort zone—force it off-policy, off the safe conservative path. Ask it to explain a pun it did not write, or answer questions like whether a pair of words that you picked rhyme. Write pairs of new words that have never been seen before, etc. The task of ‘come up with a memorized rhyme for reasonably common words’ does not disprove extensive memorization or show that it has failed to understand the underlying phonetics.