One simple example: pretraining an LLM[1] will converge to an approximation of Bayesian inference over the training distribution[2]. Any part of the output distribution clearly off the pareto frontier of approximation with respect to the architecture/training scheme/data would be akin to an anomalous and persistent area of high pressure within a gas; it’s an unstable high energy state with respect to optimization.
I can’t tell you with certainty how the weights will move during the training process, but I can tell you where it’s going at a higher level.
there are degrees of freedom outside of the training distribution that aren’t bound by the same dynamics. Adversarial intervention to build and maintain circuitry outside the training distribution could survive the pressure, for example.
The presence of induction heads would be a decent answer. If you show a model …[A][B]...[A] It will often complete it with [B]. One specific mechanism which can cause this is well-studied, but the general reason is that this pattern is really common in lots of different types of pretraining data.
What strong claims can we make about LLM behavior which are analogous to e.g. the ideal gas law?
One simple example: pretraining an LLM[1] will converge to an approximation of Bayesian inference over the training distribution[2]. Any part of the output distribution clearly off the pareto frontier of approximation with respect to the architecture/training scheme/data would be akin to an anomalous and persistent area of high pressure within a gas; it’s an unstable high energy state with respect to optimization.
I can’t tell you with certainty how the weights will move during the training process, but I can tell you where it’s going at a higher level.
assuming proper scoring rule etc.
there are degrees of freedom outside of the training distribution that aren’t bound by the same dynamics. Adversarial intervention to build and maintain circuitry outside the training distribution could survive the pressure, for example.
Relevant xkcd: link.
The presence of induction heads would be a decent answer. If you show a model …[A][B]...[A] It will often complete it with [B]. One specific mechanism which can cause this is well-studied, but the general reason is that this pattern is really common in lots of different types of pretraining data.