I’m curious what happens if you relax the accumulating sum, allowing the model to “reposition” the tokens as it desires. Due to the causal mask the model already has access to the ordering (which is why NoPE is an effective positional encoding), but this might allow it to move related words near eachother, even if they are not adjacent in the sentence.
Moreover, you could provide multiple dimensions of “position” with which to do this, by say having a rotary encoding on the first half of the latent vectors and then a separate independently learned encoding on the second half.
I wonder how (or if) the model would use this to group tokens at various layers.
I’m not sure how interesting this is, but I relaxed the positive-increment constraint and removed the sigmoid, and the model only learned one negative increment, specifically the last byte of the Chinese possessive modifier (的), although ’s in English sometimes got slightly-negative increments.
There’s definitely aspects of this that make me think it’s constrainted by the inability to change its mind, since s’ in English (plural possessive) gets a larger increment, presumably because it’s too late to change “s” and the model likes to separate on punctuation. I’m guessing if we can figure out a way to let the model decide on spacing after seeing the whole sentence it would do something more interesting.
Interesting! Currently, you are deciding the increment to the next token… using which activations? Post transformer? Post MLP?
It seems like maybe what you’re looking for is, for each layer, determine the increment from the previous token after the previous layer’s MLP (for the current token). Since layer 0 you have no previous layer to reference, maybe it just uses standard RoPE, or maybe you do something similar to what you’re doing now where you determine the layer 0 increment based on the end representation of the previous token.
The DeltaMLP blocks consume the incoming residual stream (or embeddings) for the current token between the transformer blocks, so we always have one (although advancing the position before position 0′s is meaningless since only relative position matters). For layer 0, we use the token’s embedding, so the position is fully static per-token.
From the attention section’s perspective, we’re feeding in the embedding or residual the same way we normally would, and the only difference is that Q and K are also rotated based on the sum of all calculated position increments instead of the position count.
I think the requirement that increments be positive probably isn’t necessary, although the problem where the model can’t look forward prevents a lot of uses for this (it might work decently with BPE though).
For retroactively changing positions, I think training would be unstable and finnicky, but it would be interesting if you could get it to work.
Something related that I’m interested in is BLT/H-Net style models that do dynamic chunking instead of tokenization. H-Net does this at multiple scales, so it could theoretically group words in the first layer and concepts in the second. I haven’t actually tried to train or inspect on yet though.
This is really interesting!
I’m curious what happens if you relax the accumulating sum, allowing the model to “reposition” the tokens as it desires. Due to the causal mask the model already has access to the ordering (which is why NoPE is an effective positional encoding), but this might allow it to move related words near eachother, even if they are not adjacent in the sentence.
Moreover, you could provide multiple dimensions of “position” with which to do this, by say having a rotary encoding on the first half of the latent vectors and then a separate independently learned encoding on the second half.
I wonder how (or if) the model would use this to group tokens at various layers.
I’m not sure how interesting this is, but I relaxed the positive-increment constraint and removed the sigmoid, and the model only learned one negative increment, specifically the last byte of the Chinese possessive modifier (的), although ’s in English sometimes got slightly-negative increments.
There’s definitely aspects of this that make me think it’s constrainted by the inability to change its mind, since s’ in English (plural possessive) gets a larger increment, presumably because it’s too late to change “s” and the model likes to separate on punctuation. I’m guessing if we can figure out a way to let the model decide on spacing after seeing the whole sentence it would do something more interesting.
Interesting! Currently, you are deciding the increment to the next token… using which activations? Post transformer? Post MLP?
It seems like maybe what you’re looking for is, for each layer, determine the increment from the previous token after the previous layer’s MLP (for the current token). Since layer 0 you have no previous layer to reference, maybe it just uses standard RoPE, or maybe you do something similar to what you’re doing now where you determine the layer 0 increment based on the end representation of the previous token.
The DeltaMLP blocks consume the incoming residual stream (or embeddings) for the current token between the transformer blocks, so we always have one (although advancing the position before position 0′s is meaningless since only relative position matters). For layer 0, we use the token’s embedding, so the position is fully static per-token.
From the attention section’s perspective, we’re feeding in the embedding or residual the same way we normally would, and the only difference is that Q and K are also rotated based on the sum of all calculated position increments instead of the position count.
I think the requirement that increments be positive probably isn’t necessary, although the problem where the model can’t look forward prevents a lot of uses for this (it might work decently with BPE though).
For retroactively changing positions, I think training would be unstable and finnicky, but it would be interesting if you could get it to work.
Something related that I’m interested in is BLT/H-Net style models that do dynamic chunking instead of tokenization. H-Net does this at multiple scales, so it could theoretically group words in the first layer and concepts in the second. I haven’t actually tried to train or inspect on yet though.