If a model has never seen a token spelled out in training, it can’t spell it.
I wouldn’t be sure about this? I guess if you trained a model e.g. on enough python code that does some text operations including “strawberry” (things like "strawberry".split("w")[1] == "raspberry".split("p")[1]) it would be able to learn that. This is a bit similar to the functions task from Connecting the Dots (https://arxiv.org/abs/2406.14546).
Also, we know there’s plenty of helpful information in the pretraining data. For example, even pretty weak models are good at rewriting text in uppercase. ” STRAWBERRY” is 4 tokens, and thus the model must understand these are closely related. Similarly, “strawberry” (without starting space) is 3 tokens. Add some typos (eg. the models know that if you say “strawbery” you mean “strawberry”, so they must have learned that as well) and you can get plenty of information about what 101830 looks like to a human.
And ofc, somewhere there in the training data you need to see some letter-tokens. But I’m pretty sure it’s possible to learn how many R’s are in “strawberry” without ever seeing this information explicitly.
I wouldn’t be sure about this? I guess if you trained a model e.g. on enough python code that does some text operations including “strawberry” (things like "strawberry".split("w")[1] == "raspberry".split("p")[1]) it would be able to learn that. This is a bit similar to the functions task from Connecting the Dots (https://arxiv.org/abs/2406.14546).
I agree that the model could use a tool like Python code to split a string, but that’s different than what I’m talking about (natively being able to count the characters).See below.
Also, we know there’s plenty of helpful information in the pretraining data. For example, even pretty weak models are good at rewriting text in uppercase. ” STRAWBERRY” is 4 tokens, and thus the model must understand these are closely related. Similarly, “strawberry” (without starting space) is 3 tokens. Add some typos (eg. the models know that if you say “strawbery” you mean “strawberry”, so they must have learned that as well) and you can get plenty of information about what 101830 looks like to a human.
Yes, this is possible, but the LLM had to memorize these relationships from the training data. It can’t just look at the characters and count like them a human does.
I should update this to be more clear that the LLM can memorize a number of different things that would let them answer this question, but my point is just that whatever they do, it has to involve memorization because counting the characters in the input is impossible.
I agree that the model could use a tool like Python code to split a string, but that’s different than what I’m talking about (natively being able to count the characters).
Hmm, I don’t see how that’s related to what I wrote.
I meant that the model has seen a ton of python code. Some of that code had operations on text. Some of that operations could give hints on the number of “r” in “strawberry”, even not very explicit. The model could deduce from that.
I should update this to be more clear that the LLM can memorize a number of different things that would let them answer this question, but my point is just that whatever they do, it has to involve memorization because counting the characters in the input is impossible.
I agree this has to involve some memorization. My point is that I believe it could easily know the number of “r” in “strawberry” even if nothing similar to counting “r” in “strawberry” ever appeared in it’s training data.
Oh I see what you mean. Yes, if the model saw a bunch of examples implying things about the character structure of the token, it could memorize that and use it to spell the word. My point is just that it has to learn this info about each token from the training data since it can’t read the characters.
I wouldn’t be sure about this? I guess if you trained a model e.g. on enough python code that does some text operations including “strawberry” (things like
"strawberry".split("w")[1] == "raspberry".split("p")[1]) it would be able to learn that. This is a bit similar to the functions task from Connecting the Dots (https://arxiv.org/abs/2406.14546).Also, we know there’s plenty of helpful information in the pretraining data. For example, even pretty weak models are good at rewriting text in uppercase. ” STRAWBERRY” is 4 tokens, and thus the model must understand these are closely related. Similarly, “strawberry” (without starting space) is 3 tokens. Add some typos (eg. the models know that if you say “strawbery” you mean “strawberry”, so they must have learned that as well) and you can get plenty of information about what 101830 looks like to a human.
And ofc, somewhere there in the training data you need to see some letter-tokens. But I’m pretty sure it’s possible to learn how many R’s are in “strawberry” without ever seeing this information explicitly.
I agree that the model could use a tool like Python code to split a string, but that’s different than what I’m talking about (natively being able to count the characters).See below.Yes, this is possible, but the LLM had to memorize these relationships from the training data. It can’t just look at the characters and count like them a human does.
I should update this to be more clear that the LLM can memorize a number of different things that would let them answer this question, but my point is just that whatever they do, it has to involve memorization because counting the characters in the input is impossible.
Hmm, I don’t see how that’s related to what I wrote.
I meant that the model has seen a ton of python code. Some of that code had operations on text. Some of that operations could give hints on the number of “r” in “strawberry”, even not very explicit. The model could deduce from that.
I agree this has to involve some memorization. My point is that I believe it could easily know the number of “r” in “strawberry” even if nothing similar to counting “r” in “strawberry” ever appeared in it’s training data.
Oh I see what you mean. Yes, if the model saw a bunch of examples implying things about the character structure of the token, it could memorize that and use it to spell the word. My point is just that it has to learn this info about each token from the training data since it can’t read the characters.