Prior work showed that when you train transformers on in-context linear regression tasks, they will learn a function that either specialises to the specific tasks you show them during training (task memorisation) or implements a general regression algorithm that can also handle unseen tasks like ridge regression (task generalisation); and this depends on the number of tasks you show it during training.
In this new paper, we tried changing the set of tasks you show the transformer during training to emulate non-stationary post-training. There are two new possibilities: the model can either continually update its memorised algorithm to work with the latest set of tasks (seems like your concept of ‘strategy churn’) or can learn the general regression algorithm that can handle past and future tasks (seems like ‘ecological generalist’). Indeed we see these two outcomes. The slower we change the tasks, the more likely we are to see strategy churn.
It’s pretty unclear to me exactly what is determining the decision. Hypothetically, the transformer has some kind of “recency window” which extends back some way over the history of changing tasks (but not all the way to the start), and it decides which tasks to remember based on what is in the recency window (if there are lots of tasks, it will fit them using the generalising algorithm, like in the stationary case). But this seems pretty mechanistically naive, since if you are memorising a set of tasks and one of the tasks changes by just a little, it seems likely that you’ll just update your internal memory of that tasks in a way that completely forgets the old version and replaces it with the new version. (IDK if this makes sense I think I need to write it out more carefully.)
Whatever the case in this setting, it seems to me a promising next step for testing these principles is to design a toy learning problem that better captures the space of low/high shared structure x low/high distinguishability than this in-context linear regression setting, and then run some experiments like the ones we ran but in the new setting.
Similar motivations led to this recent workshop paper.
Prior work showed that when you train transformers on in-context linear regression tasks, they will learn a function that either specialises to the specific tasks you show them during training (task memorisation) or implements a general regression algorithm that can also handle unseen tasks like ridge regression (task generalisation); and this depends on the number of tasks you show it during training.
In this new paper, we tried changing the set of tasks you show the transformer during training to emulate non-stationary post-training. There are two new possibilities: the model can either continually update its memorised algorithm to work with the latest set of tasks (seems like your concept of ‘strategy churn’) or can learn the general regression algorithm that can handle past and future tasks (seems like ‘ecological generalist’). Indeed we see these two outcomes. The slower we change the tasks, the more likely we are to see strategy churn.
It’s pretty unclear to me exactly what is determining the decision. Hypothetically, the transformer has some kind of “recency window” which extends back some way over the history of changing tasks (but not all the way to the start), and it decides which tasks to remember based on what is in the recency window (if there are lots of tasks, it will fit them using the generalising algorithm, like in the stationary case). But this seems pretty mechanistically naive, since if you are memorising a set of tasks and one of the tasks changes by just a little, it seems likely that you’ll just update your internal memory of that tasks in a way that completely forgets the old version and replaces it with the new version. (IDK if this makes sense I think I need to write it out more carefully.)
Whatever the case in this setting, it seems to me a promising next step for testing these principles is to design a toy learning problem that better captures the space of low/high shared structure x low/high distinguishability than this in-context linear regression setting, and then run some experiments like the ones we ran but in the new setting.