In order to predict the inner workings of a language model well enough to understand the outputs, you not only need to know the structure of the model, but also the weights and how they interact. It is very hard to do that without a deep understanding of the training data, and so effectively predicting what the model will do requires understanding both the model and the world the model was trained on.
Here is a concrete example:
Let’s say I have two functions, defined as follows:
import random
words = []
def do_training(n):
for i in range(n):
word = input('Please enter a word: ')
words.append(word)
def do_inference(n):
output = []
for i in range(n):
word = random.choice(words)
output.append(word)
return output
If I call do_training(100) and then hand the computer to you for you to put 100 words into, and you then handed the computer back to me (and cleared the screen), I would be able to tell you that do_inference(100) would spit out 100 words pulled from some distribution, but I wouldn’t be able to tell you what distribution that is without seeing the training data.
See this post for a more in-depth exploration of this idea.
In order to predict the inner workings of a language model well enough to understand the outputs, you not only need to know the structure of the model, but also the weights and how they interact. It is very hard to do that without a deep understanding of the training data, and so effectively predicting what the model will do requires understanding both the model and the world the model was trained on.
Here is a concrete example:
Let’s say I have two functions, defined as follows:
If I call
do_training(100)
and then hand the computer to you for you to put 100 words into, and you then handed the computer back to me (and cleared the screen), I would be able to tell you thatdo_inference(100)
would spit out 100 words pulled from some distribution, but I wouldn’t be able to tell you what distribution that is without seeing the training data.See this post for a more in-depth exploration of this idea.