Hi. I am interested in much of the mathematics which underlies theories of physics, such as complex analysis, as well as most of mathematics, although I sadly do not have the capacity to learn about the majority of it. Your interests seem interesting to me, but I do not understand enough about AI to know exactly what you mean. What is the residual stream of a transformer?
I sadly do not have the capacity to learn about the majority of it.
Sadly, it’s a problem you share with me and most humans, I think, with possible rare exceptions like Paul Erdős.
I’ll try to build up a quick sketch of what the residual stream is, forgive me if I say things that are basic, obtuse, or slightly wrong for brevity.
All neural networks (NN) are built using linear transformations/maps which in NN jargon are called “weights” and non-linear maps called “activation functions”. The output the activation functions are called “activations”. There are also special kinds of maps and operations depending on the “architecture” of the NN (eg: convNet, resNet, LSTM, Transformer).
A vanilla NN is just a series of “layers” consisting of a linear map and then an activation function.
The activation functions are not complicated nonlinear maps, but quite simple to understand. One of the most common, ReLu, can be understood as “for all vectors, leave positive components alone, set negative components to 0” or “project all negative orthants onto the 0 hyperplane”. So, since most of the complex behaviour of NNs is coming from the interplay of the linear maps and these simple nonlinear maps, so linear algebra is a very foundational tool for understanding them.
The transformer architecture is the fanciest new architecture that forms the foundation of modern LLMs which act as the “general pretrained network” for products such as chat-GPT. The architecture is set up with a series of “transformer blocks” each of which has a stack of “attention heads” which is still matrix transformations but set up in a special way, and then a vanilla NN.
The output of each transformer block is summed with the input to use as the input for the next transformer block. The input is called a “residual” from the terminology of resNets. So the transformer block can be thought of as “reading from” and “writing to” a “stream” of residuals passed along from one transformer block to the next like widgets on a conveyor belt, each worker doing their one operation and then letting the widget pass to the next worker.
For a language model, the input to the first transformer block is a sequence of token embeddings representing some sequence of natural language text. The output of the last transformer block is a sequence of predictions for what the next token will be based on the previous ones. So I imagine the residual stream as a high dimensional semantic space, with each transformer block making linear transformations and limited nonlinear transformations to that space to take the semantics from “sequence of words” to “likely next word”.
I am interested in understanding those semantic spaces and think linear algebra, topology, and manifolds are probably good perspectives.
Thanks for your clear explanation, understanding the topology of the space seems fascinating. If it’s a vector space, I would assume its topology is simple, but I can see why you would be interested in the subspaces of it where meaningful information might actually be stored. I imagine that since topology is the most abstract form of geometry, the topological structure would represent some of the most abstract and general ideas the neural network thinks about.
Yeah! I think distance, direction, and position (not topology) are at least locally important in semantic spaces, if not globally important, but continuity and connectedness (yes topology) are probably important for understanding the different semantic regions, especially since so much of what neural nets seem to do is warping the spaces in a way that wouldn’t change anything about them from a topological perspective!
subspaces of it where meaningful information might actually be stored
At least for vanilla networks, the input can be embedded into higher dimensions or projected into lower dimensions, so you’re only ever really throwing away information, which I think is an interesting perspective for when thinking about the idea that meaningful information would be stored in different subspaces. It feels to me more like specific kinds of data points (inputs) which had specific locations in the input distribution would, if you projected their activation for some layer into some subspace, tell you something about that input. But whatever it tells you was in the semantic topology of the input distribution, it just needed to be transformed geometrically before you could do a simple projection to a subspace to see it.
Hi. I am interested in much of the mathematics which underlies theories of physics, such as complex analysis, as well as most of mathematics, although I sadly do not have the capacity to learn about the majority of it. Your interests seem interesting to me, but I do not understand enough about AI to know exactly what you mean. What is the residual stream of a transformer?
Sadly, it’s a problem you share with me and most humans, I think, with possible rare exceptions like Paul Erdős.
I’ll try to build up a quick sketch of what the residual stream is, forgive me if I say things that are basic, obtuse, or slightly wrong for brevity.
All neural networks (NN) are built using linear transformations/maps which in NN jargon are called “weights” and non-linear maps called “activation functions”. The output the activation functions are called “activations”. There are also special kinds of maps and operations depending on the “architecture” of the NN (eg: convNet, resNet, LSTM, Transformer).
A vanilla NN is just a series of “layers” consisting of a linear map and then an activation function.
The activation functions are not complicated nonlinear maps, but quite simple to understand. One of the most common, ReLu, can be understood as “for all vectors, leave positive components alone, set negative components to 0” or “project all negative orthants onto the 0 hyperplane”. So, since most of the complex behaviour of NNs is coming from the interplay of the linear maps and these simple nonlinear maps, so linear algebra is a very foundational tool for understanding them.
The transformer architecture is the fanciest new architecture that forms the foundation of modern LLMs which act as the “general pretrained network” for products such as chat-GPT. The architecture is set up with a series of “transformer blocks” each of which has a stack of “attention heads” which is still matrix transformations but set up in a special way, and then a vanilla NN.
The output of each transformer block is summed with the input to use as the input for the next transformer block. The input is called a “residual” from the terminology of resNets. So the transformer block can be thought of as “reading from” and “writing to” a “stream” of residuals passed along from one transformer block to the next like widgets on a conveyor belt, each worker doing their one operation and then letting the widget pass to the next worker.
For a language model, the input to the first transformer block is a sequence of token embeddings representing some sequence of natural language text. The output of the last transformer block is a sequence of predictions for what the next token will be based on the previous ones. So I imagine the residual stream as a high dimensional semantic space, with each transformer block making linear transformations and limited nonlinear transformations to that space to take the semantics from “sequence of words” to “likely next word”.
I am interested in understanding those semantic spaces and think linear algebra, topology, and manifolds are probably good perspectives.
Thanks for your clear explanation, understanding the topology of the space seems fascinating. If it’s a vector space, I would assume its topology is simple, but I can see why you would be interested in the subspaces of it where meaningful information might actually be stored. I imagine that since topology is the most abstract form of geometry, the topological structure would represent some of the most abstract and general ideas the neural network thinks about.
Yeah! I think distance, direction, and position (not topology) are at least locally important in semantic spaces, if not globally important, but continuity and connectedness (yes topology) are probably important for understanding the different semantic regions, especially since so much of what neural nets seem to do is warping the spaces in a way that wouldn’t change anything about them from a topological perspective!
At least for vanilla networks, the input can be embedded into higher dimensions or projected into lower dimensions, so you’re only ever really throwing away information, which I think is an interesting perspective for when thinking about the idea that meaningful information would be stored in different subspaces. It feels to me more like specific kinds of data points (inputs) which had specific locations in the input distribution would, if you projected their activation for some layer into some subspace, tell you something about that input. But whatever it tells you was in the semantic topology of the input distribution, it just needed to be transformed geometrically before you could do a simple projection to a subspace to see it.